Seeing the Wind from a Falling Leaf
Authors: Zhiyuan Gao
Jiageng Mao
Hong-Xing Yu
Haozhe Lou
Emily Yue-ting Jia
Jernej Barbic
Jiajun Wu
Yue Wang
Venue: NeurIPS 2025 poster ()
Keywords: Physics-based Vision
Primary area: applications

Abstract: A longstanding goal in computer vision is to model motions from videos, while the representations behind motions, i.e. the invisible physical interactions that cause objects to deform and move, remain largely unexplored. In this paper, we study how to recover the invisible forces from visual observations, e.g., estimating the wind field by observing a leaf falling to the ground. Our key innovation is an end-to-end differentiable inverse graphics framework, which jointly models object geometry, physical properties, and interactions directly from videos. Through backpropagation, our approach enables the recovery of force representations from object motions. We validate our method on both synthetic and real-world scenarios, and the results demonstrate its ability to infer plausible force fields from videos. Furthermore, we show the potential applications of our approach, including physics-based video generation and editing. We hope our approach sheds light on understanding and modeling the physical process behind pixels, bridging the gap between vision and physics. Please check more video results in our project page https://chaoren2357.github.io/seeingthewind/ .

Seeing the Wind from a Falling Leaf
Zhiyuan Gao!" Jiageng Mao! Hong-Xing Yu?
Haozhe Lou' Yue Wang'
Emily Yue-Ting Jia
Jernej Barbic' Jiajun Wu²
University of Southern California Stanford University
{gaozhiyu, jiagengm, haozhelo, eyjia, jub, yue. w) (koven, jiajunwu)@cs wtt.e/esssstttoradeeeeeeeeeeeeee stanford.edu
Input Video
Recevered Force Field
Physirs-busod Video Generiation
Input
Force
Ply siesisedd Mution Editiing
We propose end-to-end differentiable framework capable of estimating invisible forces directly from video data, mimicking the human ability to perceive unseen physical effects through vision alone. This approach enables applications such as physics-based video generation, where new objects can be seamlessly introduced into a scene and simulated within the same force field. Force strength: from low high (best viewed in colors).
Figure I:
Abstract
A longstanding goal in computer vision is to model motions from videos, while the representations behind motions, i.c. the invisible physical interactions that cause objects to deform and move, remain largely unexplored. In this paper, we study how to recover the invisible forces from visual observations, e.g., estimating the wind field by observing leaf falling to the ground. Our key innovation is an end-to-end differentiable inverse graphics framework, which jointly models object geometry, physical properties, and interactions directly from videos. Through backpropagation, our approach enables the recovery of force representations from object motions. We validate our method on both synthetic and real-world sce- narios, and the results demonstrate its ability to infer plausible force fields from videos. Furthermore, we show the potential applications of our approach, including physics-based video generation and editing. hope our approach sheds light on understanding and modeling the physical process behind pixels, bridging the gap
Equual contribution.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

between vision and physics. Please check more video results in our project page https://chaoren2357. htt.:///oronnn..gggiooooooeeeeen
"Who has seen the wind? Neither I nor you: But when the leaves hang trembling. the -Chisstna Rossetti
wind is passing through."
Introduction
Watching leaves swirl and glide through the autumn breeze, we can almost sense the wind gently guiding them in natural choreography. Similarly, cherry blossom petals drift in spring, it feels as though the air cradles them, orchestrating their delicate descent. Although we cannot directly see the wind, humans can seamlessly infer these invisible physical interactions from visible cues in their surroundings, such as those captured in vidcos. While this intuitive physics capability has long existed in human vision, it remains underexplored in computer vision. In this paper, we bridge the gap by introducing differentiable framework to revealing invisible forces from visual data.
The key challenge of this problem lies in extracting insights about unseen target- dynamic forces-while relying exclusively on visible inputs. To address this, it is essential to understand how videos, visible cues, connect to the underlying invisible dynamics. Consider video of leaf falling: external forces like wind, apply to a leaf with known shape, appearance, and physical properties, producing a motion that aligns with physical laws and is captured visually. By developing an end-to-end differentiable model of this physical process, we can learn and predict these invisible forces, such as wind, based on video evidence alone.
To this end, we propose a differentiable inverse graphics framework, which models objects' in- herent properties (geometry, appearance, and physical properties), invisible force representations, and physical processes from video inputs. For object modeling, we leverage 3D Gaussians [1] as representations for shape and appearance, which can be easily obtained from videos. To model objects' physical properties, we propose a novel approach that leverages commonsense about physical properties in vision-language models and attaches the knowledge to 3D Gaussians. For force represen- tations, we adopt the Eulerian perspective and introduce a novel causal tri-plane representation, which models the spatio-temporal continuity and intrinsic causality of forces with high fidelity. For physical processes, we implement a differentiable physics simulator for deformable objects to animate object motions based on object properties and forces. We note that our object representation (Gaussians as Lagrangian elements) and the force representation (causal tri- tri-plane as grids) perfectly fit into the formulations of the material point method [2], allowing us accurately model the physical process. Together, these components form differentiable framework that bridges perception and physics, so that we can estimate forces from video object motions via backpropagation.
While our framework accurately models the physical process, recovering force representations from object motions in videos remains highly challenging. Unlike system identification approaches which estimate only a few physical parameters, forces are omni-directional and can present throughout the 3D space. Estimating such dense and complex force representations poses great challenges to optimization. Moreover, time integration in the physics simulator leads to unstable backpropagation, with gradients often exploding as they accumulate over time. To address the challenges, we propose a novel 4D sparse tracking objective, where we represent object motions as the movements of sparse keypoints in the ppatio-tempora space, and the movements of the Lagrangian elements, i.e., the 3D Gaussians, are further controlled by their neighboring keypoints via barycentric interpolation. With this objective, we greatly reduce the complexity of the prediction space and facilitate the estimating of force representations.
We evaluated our estimated force representations on both synthetic and real-world scenarios. The results demonstrate our method's 's ability to recover invisible forces from videos. Moreover, we show that with the estimated force representation, we can generate novel and physically plausible object motions by changing object types, physical properties, or boundary conditions, which enables realistic passed video generation and editing.
To conclude, we summarize our contributions as follows:
We identify an important problem physies understanding from videos: recovering invisible forces from object motions. To tackle this problem, propose novel inverse-graphin framework

---

that jointly models object properties, forces, and physical processes, enabling the estimation of underlying forces directly from video observations.
We introduce novel sparse tracking objective, which effectively handles the optimization chal- lenges in differentiable physics and enables robust estimation of forces from visual inputs.
We demonstrate our method's ability to recover forces from motion, and showcase its potential for generating physically plausible motions and enabling physics-based video generation and editing.
2 Related Works
Understanding the physical world is fundamental aspect of human intelligence. Researchers have long sought to bring this intuitive physics understanding ability to machine intelli- gence. Galileo [3] and the following works [4, 5] integrated deep learning with physics simulation to estimate physical object properties from visual observations. More recent approaches performed system identification by leveraging differentiable physies [6-9], neural fields [10-12], 3D Gaussian splatting [13, 14]. vision-language models [15, 16]. and video generation [17-20]. enabling more accurate estimation of physical object properties. However, these methods primarily focus on single physical parameter, such as mass, friction, and Young's 's modulus. In contrast, estimating forces is significantly more challenging. they are vectors that can exist throughout the 3D space. In this paper, we propose novel framework that successfully recovers force fields from visual inputs.
Intuitive Physics.
Differentiable Physies. Differentiable physics simulators [21-29] have been widely used to bridge perception and physics by enabling the backpropagation of particle motion gradients to physical parameters. However, using gradients from physics simulators to optimize physical properties can be notoriously difficult, as the inherent discontinuous behavior and the time integration of physics simulation often lead to vanishing or exploding gradients. To handle this challenge. we propose an optimization scheme with novel sparse tracking objective, which greatly stabilizes the estimation process and enables robust recovery of high-dimensiooal forces.
Force Estimation. Researchers explored modeling contact forces for robotic manipulation [30- 34] and human-object interactions [35-40]. However, most approaches rely controlled robotic environments with tactile sensors or require strong priors on hand and object shapes, as well as physical properties, estimate forces. In contrast, our method operates on natural videos with minimal assumptions about object properties, enabling force estimation in unconstrained scenarios.
Physics-basse Generation. Researchers explored reconstructing physically interactive scenes [41, 42] and generating physically plausible vidcos [43-46]. Most approaches rely on physics simulators or physics-infooe sicc--nnfrmmed neural networks [47] to animate motion, but they typically require manually specified forces and environmental conditions. Beyond these, interactive editing methods [48, 49] drive visual changes by optimizing displacement fields in the image or feature space under generative priors; such formulations specify apparent motion without estimating underlying physical forces. An alternative approach learns 3D velocity fields directly from videos [50, 51], producing smooth trajectories yet lacking explicit force representations, which makes parameter-aware edits (e.g., changing mass) less principled. In contrast, our approach automatically recovers forces and physical conditions from natural videos and applies them to novel objects, enabling physics-driven video generation without manual parameter tuning.
3 Method
We study recovering invisible forces from videos. Our inverse graphics framework first models object properties (Section 3.1), force representations (Section 3.2), and physical processes (Section 3.3) from videos. To optimize force representations, we introduce a sparse tracking objective (Section 3.4). An overview of method is in Figure 2.
3.1 Object Modeling
Capturing the essence of dynamic objects requires modeling both their shape for accurate physical interactions and their appearance for visual fidelity. This necessitates a representation that seamlessly integrates precise Lagrangian shape modeling with photorealistic rendering. To this end, we adopt 3D Gaussians [1] the representation for shape and appearance. Specifically, an object in a video is
3

---

Object Modeling (Sec. 3.1]
hysical Process es. 3.3)
Loss Optnizztion (Sec. 3.4)
Tracke Gaussian Motions
biferrndihh Phyclcs
Image
GraectGa
Gaussians fies
Force Field Representation (Sec., 3.2)
Spone Tracking
Materbal
Object maas
Denclly in
N.P #0
Image fext
Put
Young's modulus Poisson rarlo
Language Foundatio
Image
Figure 2: We propose differentiable inverse graphies framework recover invisible forces from videos by integrating object modeling, physics simulation, and optimization, Objects are represented with 3D Gaussians and assigned physical properties via Vision-Language Models. Forces are modeled as a causal tri- tri-plane, and object motions are simulated using differentiable physics simulator. A sparse tracking objective enables robust differentiable force recovery from videos.
represented by set of Gaussian kernels G' in the 3D space. Each Gaussian kernel G is parameterized by
G = (x. v. E,,,
E,
(1)
where x E R and R2 are the spatial location and velocity of a Gaussian kernel respectively. The covariance matrix E represents the shape, and opacity a and spherical harmonics S represent the appearance of Gaussian. Moreover, attach each Gaussian with its physical properties: deformation gradient D, the mass m, Young's s modulus E, and the Poisson ratio v, which we will discuss later.
Since objects a video undergo motion and deformation due to external forces, their corresponding 3D Gaussians also evolve over time. Let t denote a timestep in the video. The Gaussians at time t is then defined as
G (x', v E' o, a, SH, D', m. E,v),
where the spatial position x', velocity v covariance E', and deformation gradient De change over time t.
We initialize the Gaussians {G"} at ( only using the first frame of the video. Specifically, we use pixel-aligned point clouds that are extracted from the first image 7 via a pretrained metric-depth model [52] to initialize the Gaussian positions {x"]. and optimize {ΣO, oo, SH} via Gaussian splatting on 1º, Notably, although point clouds from a single image can be incomplete, we can still obtain robust force estimates thanks to the proposed sparse tracking objective, which will be discussed later. We have also explored multiview object reconstruction in our experiments. For o and D initialized them 0 and I respectively.
To model the physical interactions between objects and forces, we also need know the objects' physical properties from videos. To this end, we introduce simple but effective approach that leverages commonsense knowledge from vision-language models to assign the physical properties (m, E, } to each 3D Gaussian. Specifically, given the first image 70, we first query a vision-languaae model [53] to infer the object types and provide an estimate of the physical propertics (m, E, v/ from commonsense knowledge. Then, query grounded segmentation model [54] to generate object segmentation masks based on the object types. Finally, the pixcl-aligned Gaussians (G") that are inside the object masks are assigned with the corresponding estimated physical properties {m, E, v). For common objects, the estimated physical properties from the vision- asiio-llanguage model quite robust. Hence, even without accurate system identification, our framework could provide robust force estimate with the commonsense physical properties(see Experiments 4.3).
Leveraging foundation models [52-54], our framework automatically recovers objects' geometry, appearance, and physical properties from videos without manual effort. The recovered object Gaussians serve unified representation for modeling physical interactions in dynamic videos.

---

3.2 Force Representations
Properly modeling foree representations is essential to our framework. For point contact forces, we can directly define force vectors on Gaussian particles. However. for forces that are distributed throughout 3D space, e.g., wind, we adopt the Eulerian perspective and introduce causal tri tri-plane to represent forces in 3D space. This representation is based on the observations that forces are spatially continuous and causally dependent over time. Specifically, we define the force f the position x and the time f. as
f(x, t) (x)( + (((:+((t-1)))
where D(-) is a feature decoder and 7()) represents the tri-plane feature map from [55]. (p()) is small MLP that encodes the time t, initialized using the learned weights from the previous timestep t 1, i.e. sp(t - 1). Compared other 4D representations [56-58], this representation disentangles space and time, leading to superior computational efficiency. Additionally, the recursive dependency of (t) on spt - 1) enables accurate modeling of evolving force dynamics over time.
3.3 Physical Process
With the object Gaussians and force representations, we are ready to simulate object motion following physical laws. To this end, we implemented differentiable physics simulator for deformable objects using the Material Point Method (MPM) [2]. Empirically, we found that our object representation, where Gaussians act as Lagrangian elements, and our force representation, modeled as causal tri-plane on a grid, naturally align with the formulations of [2], enabling accurate modeling of the physical process.
In detail, the forward physical process Fphysica takes the object Gaussians {G*} at time t and the force field f(x, t), and outputs [G++1] at the next timestep +
(G = Ephysiis(( ((x,,
The physical process relies on multiple sub-steps ôt update motions from to t + 1 incrementally. In the following section, we introduce the computational flow in a sub-step öt. For simplicity, we consider single Gaussian G and omit the particle-to-gen to-grid, grid computation, and grid-to-particle process in MPM, focusing solely on the core physics principles and update formulas.
For Gaussian G [x', v", Σ' o. SH,, D',m, m, E, v) at the time I, we first characterize the object deformations by updating the deformation gradient
gradient D':
D' (I V Vt
(5)
where Vv'-ót is the velocity gradient and I is the identity matrix. Then, we update the Gaussian velocity v' by incorporating both external and internal forces:
6t + 51 f(x',1) + 81 fi(x S,,,,,E,DDD
(6)
where f(x!, f) is the external force by querying the casual tri-plane f(x, i) the Gaussian position x", fi(x,, 1, E, D') represents internal forces f, determined by the constitutive model. m is the mass matrix derived from the mass of the Gaussian. Note that external forces computed particles before the particle-to- grid step, while internal forces are calculated on the corresponding grid during the prid-to-particn step. External forces are applied directly to particles because the scene volume is much larger than the occupied regions. Acting on particles instead of grid nodes avoids empty cells and yields finer, less noisy fields aligned with the moving mass.
Next, we update the position x" of the Gaussian following standard time integration:
1-8t 8tv*.
Finally, the covariance matrix Σ is calculated based the deformation gradient:
n = D''"
(8)
Through multiple sub-step updates, we can evolve the Gaussian state from G to G Notably, the computational process is fully differentiable. Hence, given the per-Gaussian motion & g(+]
5

---

extracted from adjacent video frames, we leverage X + xt as the motion tracking target to optimize the force field f(x. t) via backpropagation.
min [ x f(x.()
s.t. (x¹, 11) E (G', G G = Plyysios (G', f()*,1)).
(9)
In the following sections, we will discuss how establish per-Gaussian motion x - x(+1 from videos, and how to optimize the force field f(x, t) robustly.
3.4 Recovering Forces from Videos
To optimize the force field f(x, !), it is essential track per- per-Gaussian motions X - x from videos as the optimization target. A straightforward approach is to use photometric loss, iee. comparing the pixel differences in adjacent frames I | by projecting the Gaussians onto the image plane: I' = x{{G"), }), where π is the projection function. Nevertheless, we found that photometric loss alone fails provide sufficient motion constraints, often resulting in vanishing gradients during optimization. Alternatively, we can extract dense 3D scene flows from videos using off-the-shelf depth and optical flow prediction or 4D reconstruction 55961]. Nevertheless, the aceuracy of these pre-trained models is limited, resulting in noisy dense 3D flows that significantly hinder the optimization process. We found the key to robust optimization adopt more reliable motion estimates. To this end, for objects with bending-only deformations (e.g., paper folding) or small deformations, we introduce novel 4D sparse-tracking objective. Specifically, we adopt a more reliable point-tracking algorithm [62] that provides sparse, pixel-level estimates of object keypoint motions p + p'+1, where E IR Nxx X2 is N keypoint pixel coordinates. Next, we want establish the keypoint correspondences in 3D: P - p'+1, where use P E R&xx and p E Nx to denote the associated keypoints in the 3D and pixel space respectively. To obtain p + p(+1 from p' - p p²+1, we first estimate the 3D keypoint locations po in the first frame by unprojecting p" into 3D with depth estimates d:
to reduce the target space and
pD = π (p, ,d):
(10)
where d comes from metric-depth model [52] and π is an inverse-projection function. Then, for each adjacent frames, we obtain p + p(+1 from P and p'+l by optimizing the following objective:
min P pt-pr+1
| (P(+1) - p ++| + ACaraps
(11)
where the seggid-as-possible loss Larap is represented as
Larap =
- -PJ+)) (P: -P))--
(12)
By minimizing the re-projection errors while keeping the object skeleton as rigid as possible, obtain a robust estimate of the 3D keypoint motions Pe - p Notably, without reliance on per-frame depth estimation, method circumvents the inconsistent video depth estimation problem and enables more robust 3D motion estimates.
Next, we leverage the sparse keypoint motions P - p4+1 to control the per-Gaussian motions x¹
(+1 via
X a a 1oj+oo agPk.
(13)
where the Equation 13 is the barycentric interpolation, Pt.j.k are the 3-nearest neighbors of & and the coefficients Oi.j,& are computed in the first frame and fixed in the following frames. The sparse keypoints P characterize the object skeletons and control the fine-grained Gaussian positions & Com- pared to the 3D scene flow approach that directly tracks each Gaussian's motion, estimating sparse a
keypoint motions P + p*+1 reduces the prediction space and demonstrates superior robustness and accuracy, allowing us to obtain high-quality & + * for optimizing the force field.
Finally, we optimize the force f(x, E) using the motion-tracking loss Lmotion in Equation 9, with the estimated Gaussian motions & + x Moreover, we add two regularization terms Lapace and Lisme for spatial and temporal smoothness, respectively:
L Lmotion + X1 Espace + XgLiimer
(14)

---

where Lspace follows [56] and penalize the total variation in space, and Ltime encourages temporal smoothness by penalizing the parameter differences of the time encoder (p()):
Ltime
1 -phl.
(15)
where v is the parameters of the time encoder at the time / in Equation 3. With the losses in Equation 14, we are able to recover the force f(x, t) by tracking the Gaussian motions - x extracted from videos.
Experiments
In this section, we conduct comprehensive experiments to investigate the following key questions:
Can our method successfully recover forces from both synthetic and real-world videos? (Sec- tion 4.2)
How do the proposed components, i.e, force representation and loss function affect the final performance, and how robust is the proposed VLM framework to variations in object physical properties?(Section 4.3)
How can our method be applied to physics-basee based video generation and editing? (Section 4.4)
4.1 Experimental Setup
We conduct experiments both real-world and synthetic data. For real-world data, leverage Internet videos to verify the physical plausibility of recovered forces by visualizing the force field. In addition, conduct real physical experiments with force gauge to measure the actual forces, and we evaluate our recovered forces via re-simulation. For synthetic data, we use synthetic objects in [63, 18] evaluate the numerical accuracy of recovered forces. We leverage objects from distinct material types, i.e., elastic, elastoplastic, viscoplastic, 6-8 unique force fields, and 2 different camera viewpoints to build the synthetic scenarios in the physics simulator [24]. We rendered videos as inputs to our system to estimate the forces and compare them with the ground truth forces in the simulator. For numerical comparisons in synthetic scenarios, adopt image reconstruction metrics, Le., PSNR, SSIM [64], and LPIPS [65], to compare the re-simulated videos with the recovered forces and the original input videos, demonstrate the accuracy of recovered forces match the ground truth object motions in simulation. Moreover, we compare the recovered forces with the ground truth forces using two metries: average magnitude error (reported as percentages) and direction error (measured in degrees).
4.2 Force Recovery
In-the Wild Videos. We evaluate method real Internet videos demonstrate its ability to recover plausible force fields from natural object motions. Figure 4 presents qualitative results on var- ious scenes. Our method successfully infers the underlying forces by observing object deformations and trajectories over time. The visualized forces dynamically adapt to object motion, demonstrating a physically plausible force field that varies over time.
Controlled Real- World Physics Experiments. Since the ground truth forces in real Internet videos are unknown, we conduct controlled real-world experiments to further validate our method. Using force gauge, we apply known forces to an object while capturing its motion on video. We then use method to recover the force field and reapply it the object in simulation. As shown in Figure 3, our method successfully reconstructs the object's motion and deformation, closely aligning with real-world observations. The experimental results demonstrate the accuracy of our recovered forces in real-world scenarios.
Synthetic Scenarios. To evaluate the numerical accuracy of recovered forces, we build synthetic scenarios in the physies simulator to obtain the "ground truth" forces. As shown in Table I, our method successfully recovers the original force fields with low numerical enrors. These quantitative results provide strong evidence that our approach accurately estimates the underlying force dynamics and generalizes well across various objects and physical properties.
7

---

Material Type
Methed
PENR.
550M4
LIPPS
Map: Ereoe (5))
Dir. Emer (?)
Blasto
Poser
20.57
0.36
0114
70.08
Elsoppbeic
Point
31.40
0.96
0.01
98.36
26,8
planes
30.14
0.9%
0001
47.51
01.42
Oun
39.93
0.59
0.01
75.06
45.50
Visoopantic
Oan
39.00
0.99
0,01
21.44
7.50
Table 2: Quantitative comparison force repre- sentations.
Observed
Re simulation
Observed
I Re-siisslation
Loss functions
PSNR
SSIM
LPIPS
Mag. Emree(3)
Dir. Ervor (*)
Figure 3: Comparison observed data (left in each frame) and re-simulated results (right in each frame) for two different frames in the real-world experiment.
Image Loss
37.24
0.59
0.01
86.74
50,23
Doot+Depth Loss
41.54
0.99
0.01
27.90
16.07
Qurs
39.79
0.99
11.01
5.14
4.38
Table 3: Quantitative comparison of loss func- tions.
Maseialiype
Onpect
Mog.
Material Type
ASamples
Type
pMAAE(()
E Eloo-MAPEE(S)
Eladic
Lago
33.70
0.95
0.01
19.53
7.62
Elastic
2.38
5.31
Suudower
Pooss
34.08
25.00
1154
0.99
0.01
0.05
23.1
14.38
11.54
7.85
Viscoplastic
Elastoplactic
3.36 10.00
13.98
Hasoollatic
Overall
Misoppaotic
Hordog
30.63
0.96
0.02
1509
11.63
Table 4: VLM performance on the daily-item dataset.
Table I: Force recovery in synthetic scenarios.
Recevered Force Field
Recovered Farce Field
Figure 4: Our method estimates invisible force fields from real-world videos, producing physically plausible motion interpretations.
4.3 Empirical Study
Force Representations. To evaluate the effectiveness of our force representation in Section 3.2, compare our causal tri-plane with other 4D representations such as K-planes [56] and point contact forces. The results in Table demonstrate that causal tri-plane force representation performs better than other 4D representations. This is mainly because our force representation accurately models the spatial continuity and temporal dependence of forces.
To evaluate the effectiveness of our loss functions in Section 3.4, we compare our sparse tracking loss with the image reconstruction loss and the dense 3D scene flow loss derived from depth and flow estimation (Flow+Depth loss). The results in Table demonstrate that our sparse tracking loss shows better performance than others, especially in force accuracy. This is because sparse tracking provides more robust motion estimates that can be leveraged as more accurate signals to optimize the forces.
Loss Functions.
VLM Material- Property Estimation. evaluate robustness of leveraging a vision-language model for physical parameter estimation, we measure the estimation errors by utilizing the GPT- 4o- Vision infer material type and material propertiessdensity and Young's modulus F) from single image and comparing with the ground truth values. A small benchmark of 17 everyday objects

---

Our method recovers force fields from input videos and enables the insertion of novel objects while maintaining physically plausible motion. We demonstrate new object insertion, and modifications of physical conditions (e.g. mass and external force), showcasing the model's ability to generate physically plausible videos.
Figure 5:
Iaput
Swi Dail Byer
Eai Dod
Ded
Best Dad
Fis'
Bast
Plysicz-basee Mation Edilingg Set
Eeet and Fut
Sel pat cur
Bei Dad cak
Oud Dar
Iaput video
Vistin
Figure 6: Our method allows modifying object motion by adjusting external constraints while preserving physical realism. We demonstrate how altering boundary conditions (e.g.. fixing parts of an object) influences motion under the same estimated force field. These results highlight the flexibility of our approach for controllable, physics-based video editing.
collected, each made of a single, well-documented engineering material. Ground-truth p and E values taken from MatWeb database[66). Results are summarised in Table 4. The model achieves an overall F1 of 1.0 for material classification, an 4.65% mean-abss mean-absolute- saan-absolute-percetaaee percentage error (MAPE) on density, and log- MAPE of 7.17%, which indicates that the VLM can have robust initial estimate of the physical parameters.
For non-stationary cameras we adopt 4D reconstruction pipelines to supply camera poses and trajectories, which we consume without retraining. Qualitative videos for dynamic-camera sequences are best viewed on the project page.
Moving Cameras
4.4 Physics- based Generation
With the recovered force field, we demonstrate the potential of our approach for physics-based video generation and editing. Figure 5 shows the results of physics-based video generation. Our framework enables the replacement of novel objects within the same force field, generating physically plausible motions for novel objects via physics simulation. This flexibility also allows to modify the object's

---

physical properties and force strengths to create distinct object motions that adhere to physical laws. Compared to other video generation methods, our approach produces more controllable and mmyically-accurral videos. Figure 6 shows the results of physics-based video editing, where we can modify the boundary conditions in the scenes, e.g., fixing point, to generate different physically plausible object motions in the same video. These results highlight the versatility of our framework in generating and editing videos while maintaining physical consistency.
We qualitatively compare against interactive editing/ / motion-driven methods [48, 49] and velocity- field learning baselines [50, 51] on matched inputs. These approaches optimize displacements or velocities under strong priors and thus yield kinematically plausible results, but they do not estimate identifiable physical forces and do not enforce Newtonian consistency when physical parameters are edited (e.g., doubling mass). Consequently, the resulting motion may continue to "match" an appearance prior yet diverge from the dynamics implied by the edited parameters. By explicitly recovering time-varying forces inside differentiable simulator, our method preserves dynamical consistency under parameter edits and non-uniform fields. Qualitative videos are best viewed on the project page.
5 Conclusion
In this work, we introduced a differentiable inverse graphics framework to recover invisible forces from video object motions, bridging vision and physics. By modeling object properties, forces, and physical processes, our method enables robust force estimation via backpropagation. Experiments in both real-world and synthetic data have demonstrated accurate force recovery and controllable physics-based video generation and editing of our approach.
Limitations. Our framework is primarily applied objects with small deformation bending-onl deformation. Fluids or other object types that require different, differentiable physical processes are out of the scope of this paper, which is left for future works.
As in our demo, we model foreground physics and composite over static background, as off off-thhe-selff per-frame inpainting can introduce temporal flicker that obscures our contribution.
Acknowledgments and Disclosure of Funding
This work is in part supported by NSF RI #2211258 and #2338203, ONR MURI 100001422-1127444 and the Okawa Research Grant. The USC Geometry, Vision, and Learning Lab acknowledges generous supports from Toyota Research Institute, Dolby, Google DeepMind, Capital One, Nvidia, and Qualcomm. Yue Wang is also supported by a Powell Research Award.
References
[1] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph., 139-11 2023.
[2] Chenfanfu Jiang, Craig Schroeder, Joseph Teran, Alexey Stomakhin, and Andrew Selle. The material point method for simulating continuum materials, In ACM SIGGRAPH 2016 courses, pages 1-52, 2016.
[3] Jiajun Wu, I1ker Yildiri,, Joseph J Lim, Bill Freeman, and Josh Tenenbaum. Galileo: Perceiving physical object properties by integrating a physics engine with deep learning. Advances in neural information processing systems, 28, 2015.
[4] Jiajun Wu, Joseph Lim, Hongyi Zhang, Joshua B Tenenbaum, and William Freeman. Physics 101: Learning physical object properties from unlabeled videos. In BMVC, volume 2, page 7, 2016.
[5] Jiajun Wu, Erika Lu, Pushmeet Kohli, Bill Freeman, and Josh Tenenbaum. Learning to see physies via visual de-animation. Advances in neural information processing systems, 30, 2017.
[6] Jennifer Cardona, Michael Howland, and John Dabiri. Seeing the wind: Visual wind speed prediction with a coupled convolutional and recurrent neural network, Advances in Neural Information Processing Systems, 32, 2019.
10

---

[7] David Hahn, Pol Banzet, James M. Bern, and Stelian Coros. Real2sim: visco-elastic parameter estimation from dynamic motion. ACM Trans. Graph.. 38(6), November 2019.
[8] Krishna Murthy Jatavallabhula, Miles Macklin, Florian Golemo, Vikram Voleti, Linda Petrini, Martin Weiss, Breandan Considine, Jéröme Parent-Lévesque, Kevin Xie, Kenny Erleben, et al. gradsim: Differentiable simulation for system identification and visuomotor control. arXiv preprint arXiv:2104.02646 2021.
[9] Mingyu Ding, Zhenfang Chen, Tao Du, Ping Luo, Josh Tenenbaum, and Chuang Gan. Dynamic visual reasoning by learning differentiable physics models from video and language. Advances In Neural Information Processing Systems, 34:887-899, 2021.
[10] Xuan Li, Yi-Ling Qiao, Peter Yichen Chen, Krishna Murthy Jatavallabhula, Ming Lin, Chen- fanfu Jiang, and Chuang Gan. Pac-nerf: Physics augmented continuum neural radiance fields for scometry-agnostic system identification. arXiv preprint arXiv:2303.05512 2023.
[11] Ri-Zhao Qiu, Ge Yang, Weijia Zeng, and Xiaolong Wang. Feature splatting: Language-driven physics-based scene synthesis and editing, 2024.
[12] Albert J Zhai, Yuan Shen, Emily Y Chen, Gloria X Wang, Xinlei Wang, Sheng Wang, Kaiyu Guan, and Shenlong Wang. Physical property understanding from language-embedded feature fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 28296- 28305, 2024.
[13] Junhao Cai, Yuji Yang. Weihao Yuan, Yisheng He, Zilong Dong, Liefeng Bo, Hui Cheng, and Qifeng Chen. Gaussian-informed continuum for physical property identification and simulation. arXiv preprint arXiv:2406. /4927, 2024.
[14] Licheng Zhong, Hong- Xing Yu, Jiajun Wu, and Yunzhu Li. Reconstruction and simulation of elastic objects with spring mass 3d gaussians. In European Conference on Computer Vision, pages 407-423. Springer, 2025.
[15] Xingrui Wang, Wufei Ma, Angtian Wang, Shuo Chen, Adam Kortylewski, and Alan Yuille. Compositional 4d dynamic scenes understanding with physics priors for video question answer- ing, arXiv preprint arXiv:2406.. 200.02400000 2024.
[16] Wei Chow, Jiageng Mao, Boyi Li, Daniel Seita, Vitor Guizilini, and Yue Wang. Physbench: Benchmarking and enhancing vision-language models for physical world understanding. arXiv preprint arXiv:2501. /64/1, 2025.
[17] Tianyuan Zhang, Hong-Xing Yu, Rundi Wu, Brandon Y Feng, Changxi Zheng, Noah Snavely, Jiajun Wu, and William T Freeman. Physdreamer: Physics-based interaction with 3d objects via video generation. In European Conference on Computer Vision, pages 388-406. Springer, 2025.
[18] Fangfu Liu, Hanyang Wang, Shunyu Yao, Shengjun Zhang, Jie Zhou, and Yueqi Duan. Physics3d: Learning physical properties of 3d gaussians via video diffusion. arXiv preprint arXiv:2406.00 04338, 2024.
[19] Tianyu Huang. Yihan Zeng, Hui Li, Wangmeng Zuo, and Rynson WH Lau. Dreamphysics: Learning physical properties of dynamic 3d gaussians with video diffusion priors, arXiv preprint arXiv:2406.01477 01446, 2024.
[20] Quentin Garrido, Nicolas Ballas, Mahmoud Assran, Adrien Bardes, Laurent Najman, Michael Rabbat, Emmanuel Dupoux, and Yann LeCun. Intuitive physics understanding emerges from sulf-supervised pretraining on natural videos. arXiv preprint arXiv:2502. /881, 2025.
[21] Tao Du, Kui Wu, Pingchuan Ma, Sebastien Wah, Andrew Spielberg. Daniela Rus, and Wojeiech Matusik. Diffpd: Differentiable projective dynamics. ACM Trans. Graph., 41(2), nov 2021.
[22] William Gilpin. Generative learning for nonlinear dynamics. Nature Reviews Physics, 6(3):194-
206, 2024.
[23] Eric Heiden, David Millard, Erwin Coumans, Yizhou Sheng, and Gaurav S Sukhatme. Neural- sim: Augmenting differentiable simulators with neural networks. In 202/ IEEE International Conference Robotics and Automation (ICRA), pages 9474-9481. IEEE, 2021.
11

---

[24] Yuanming Hu, Yu Fang, Ziheng Ge, Ziyin Qu, Yixin Zhu, Andre Pradhana. and Chenfanfu Jiang. moving least squares material point method with displacement discontinuity and two-way rigid body coupling. ACM Transactions on Graphics (TOG), 17(4))1-14, 2018.
[25] Yuanming Hu, Tzu-Mao Li, Luke Anderson, Jonathan Ragan-Kelley. and Frédo Durand. Taichi: computation on spatially sparse data structures. ACM Transac-
a language for lions on Graphics (TOG), 38(6):201, 2019.
[26] Yuanming Hu, Luke Anderson, Tzu-Mao Li, Qi Sun, Nathan Carr, Jonathan Ragan- and Frédo Durand. Difftaichi: Differentiable programming for physical simulation. ICLR, 2020.
[27] Yuanming Hu, Jiafeng Liu, Xuanda Yang. Mingkuan Xu, Ye Kuang, Weiwei Xu, Qiang Dai, William T. Freeman, and Frédo Durand. Quantaichi: A compiler for quantized simulations. ACM Transactions on Graphics (TOG), 40(4), 2021.
[28]
Junior Rojas, Eftychios Sifakis, and Ladislav Kavan. Differentiable implicit soft-body physics. arXiv preprint arXiv:2102.00579) 2021.
[29] Zizhou Huang. Davi Colli Tozoni, Arvi Gjoka, Zachary Ferguson, Teseo Schneider, Daniele Panozzo, and Denis Zorin. Differentiable solver for time-dependent deformation problems with contact, ACM Transactions on Graphics, 43(3):1-30, May 2024.
[30] Barza Nisar, Philipp Foehn, Davide Falanga, and Davide Scaramuzza. Vimo: Simultaneous visual inertial model-based odometry and force estimation. IEEE Robotics and Automation Letters, 43):27885- 2019.
[31] Xingyu Chen, Jialei Shi, Helge Wurdemann, and Thomas George Thuruthel. Vision-based tip force estimation on a soft continuum robot. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 7621-7627. IEEE, 2024.
[32] Wenzhen Yuan, Rui Li, Mandayam A Srinivasan, and Edward H Adelson. Measurement of shear and slip with gelsight tactile sensor. In 2015 IEEE International Conference on Robotics and Automation (ICRA), pages 304-311. IEEE, 2015.
[33] Eric Heiden, Miles Macklin, Yashraj Narang. Dieter Fox, Animesh Garg, and Fabio Ramos. Disect: A differentiable simulation engine for autonomous robotic cutting. arXiv preprint arXiv:2105.1223 122444 2021.
[34] Haozhe Lou, Yurong Liu, Yike Pan, Yiran Geng, Jianteng Chen, Wenlong Ma, Chenglong Li, Lin Wang, Hengzhen Feng, Lu Shi, et al. Robo-gs: A physics consistent for robotic arm with hybrid representation. arXiv preprint wrXiv:2408.14877 2024.
model
[35] Hyun Soo Park, Jianbo Shi, et al. Force from motion: decoding physical sensation in a first person video. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3834-3842, 2016.
[36] Zongmian Li, Jiri Sedlar, Justin Carpentier, Ivan Laptev, Nicolas Mansard, and Josef Sivic. Estimating 3d motion and forces of person-object interactions from monocular video. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, pages (640-8649,10 2019.
[37] Davis Rempe, Leonidas J Guibas, Aaron Hertzmann, Bryan Russell, Ruben Villegas, and Jimei Yang. Contact and human dynamics from monocular video. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part V 16, pages 71 71-87. Springer, 2020.
[38] Kiana Ehsani, Shubham Tulsiani, Saurabh Gupta, Ali Farhadi, and Abhinav Gupta. Use the force, luke! learning to predict physical forces by simulating effects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 224-233, 2020.
[39] Yufei Zhang, Jeffrey O Kephart, Zijun Cui, and Qiang Ji. Physpt: Physics-aware pretrained transformer for estimating human dynamics from monocular videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2305-2317, 2024.
12

---

[40] Bin Wang, Longhua Wu, KangKang Yin, Uni M Ascher, Libin Liu, and Hui Huang. Deformation capture and modeling of soft objects. ACM Trans. Graph., 34(4):94-1, 2015.
[41] Gengshan Yang, Shuo Yang, John Z Zhang, Zachary Manchester, and Deva Ramanan. Ppr: Physically plausible reconstruction from monocular videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3914-3924, 2023.
[42] Yandan Yang, Baoxiong Jia, Peiyuan Zhi, and Siyuan Huang. Physcene: Physically interactable
3d scene synthesis for embodied ai. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages (6262-1627, 2024.
[43] Katerina Fragkiadaki, Pulkit Agrawal, Sergey Levine, and Jitendra Malik. Learning visual predictive models of physics for playing billiards. arXiv preprint arXiv: (rxXiv:/511.00 1511,0404 2015.
[44] Zhengqi Li. Richard Tucker, Noah Snavely, and Aleksander Holynski. Generative image dynamics, 2024.
[45] Shaowei Liu, Zhongzheng Ren, Saurabh Gupta, and Shenlong Wang. Physgen: Rigid-body physics-grounded image-to-video generation. In European Conference on Computer Vision, pages 360-378. Springer, 2024.
[46] Tianyi Xie, Zeshun Zong, Yuxing Qiu, Xuan Li, Yutao Feng, Yin Yang. and Chenfanfu Jiang. Physgaussian: Physics-integrated 3d gaussians for generative dynamics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4389 4398, 2024.
[47] Chayan Banerjee, Kien Nguyen, Clinton Fookes, and Karniadakis George. Physics-informed informed computer vision: review and perspectives. ACM Computing Surveys, 57(1):1-38, 2024.
[48] Xingang Pan, Ayush Tewari, Thomas Leimküühler Lingjie Liu, Abhimitra Meka, and Christian Theobalt. Drag your gan: Interactive point- based manipulation on the generative image manifold. In ACM SIGGRAPH 2023 conference proceedings, pages 1-11, 2023.
[49] Daniel Geng, Charles Herrmann, Junhwa Hur, Forrester Cole, Serena Zhang, Tobias Pfaff, Tatiana Lopez-Guevara, Carl Doersch, Yusuf Aytar, Michael Rubinstein, Chen Sun, Oliver Wang, Andrew Owens, and Deqing Sun. Motion prompting: Controlling video generation with motion trajectories. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025.
[50] Jinxi Li, Ziyang Song, and Bo Yang. Nvfi: Neural velocity fields for 3d physics learning from dynamic videos. Advances in Neural Information Processing Systems, 01:34723-347555 2023.
[51] Jinxi Li, Ziyang Song, Siyuan Zhou, and Bo Yang. Freegave: 3d physics learning from dynamic videos by gaussian velocity. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 12433 12443,2443 2025.
[52] Luigi Piccinelli, Yung-Hs Yang, Christos Sakaridis, Mattia Segu, Siyuan Li, Luc Van Gool, and Fisher Yu. Unidepth: Universal monocular metric depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10106-10116, 16, 2024.
[53] Aaron Hurst, Adam Lerer, Adam Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
[54] Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, et al. Grounded sam: Assembling open-world models for diverse visual tasks. arXiv preprint arXiv:2401. 14159, 2024.
[55] Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, et al. Efficient geometry- aware 3d generative adversarial networks. Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 16123-16133 2022.
13

---

[56] Sara Fridovich- Keil, Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12479-12488, 2023.
[57] Ang Cao and Justin Johnson. Hexplane: A fast representation for dynamic scenes. In Proceed- ings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 130-141, 2023.
[58] Haotong Lin, Qianqian Wang, Ruojin Cai, Sida Peng, Hadar Averbuch-Elor, Xiaowei Zhou, and Noah Snavely, Neural scene chronology. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20752-20761, 2023.
[59] Qianqian Wang, Vickie Ye, Hang Gao, Weijia Zeng, Jake Austin, Zhengqi Li, and Angjoo Kanazawa. Shape of motion: 4d reconstruction from single video. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9660-9672 2025.
[60] Quankai Gao, Qiangeng Xu, Zhe Cao, Ben Mildenhall, Wenchao Ma. Le Chen, Danhang Tang. and UIrich Neumann. Gaussianflow: Splatting gaussian dynamics for content creation, 2024.
[61] Zhengqi Li, Richard Tucker, Forrester Cole, Qianqian Wang, Linyi Jin, Vickie Ye, Angjoo Kanazawa, Aleksander Holynski, and Noah Snavely. Megasam: Accurate, fast and robust structure and motion from casual dynamic videos. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 10486-10496, 2025.
[62] Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Cotracker: It is better to track together. In European Conference on Computer Vision, pages 18-35. Springer, 2024.
[63] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoor- thi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1)999-106, 2021.
[64] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility structural similarity. IEEE transactions image processing, 13(4):600- 612, 2004.
[65] Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreason- able effectiveness of deep features perceptual metric, 2018.
[66] MatWeb, LLC. Matweb: Online materials information resource, 2025. Accessed: 2025-05-13.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Our papers main contribution has been clearly declared in the abstract and introduction
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We include the limitation part in conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

Justification:
We present a comprehensive formulation of our proposed framework, with detailed descriptions provided in each subsection.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: A detailed description of our method is provided, and a demonstration code will be released upon acceptance.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification: The code not included for now, demonstration code will released upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
www.ummmissiooooiiyy..........cp for more details. Sttpsi//guiids//oeeeeeeeeee
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / httppsscccppbccccceceeceeeeccclc w/w.ides////eeeeeee/o /ttggddeeee/eeuubmiiiiii.
/ctoee/ummmsssionooe.....ooo ssi asionPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new reproducible, they proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We include them in implementation details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We perform the experiments multiple times and compute the average. We also have 2-sigma error results in appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
17

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide sufficient information on the computer resources for experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make it into the paper).
9. Code ethics
Question:
Does the research conducted in the paper conform, in every respect, with the wwwp//bblcc/tciiccuuiiieoooeoede
NeurlPS Code of Ethics https:
Answer:
[Yes]
Justification:
We follow the NeurIPS Code of Ethics
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discussed the potential positive social impact of work, outlined some of its applications the main text. As for negative social impacts, since this work represents relatively new and early-stage direction, have not yet identified significant negative implications.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
18

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
The codebase we use has been properly cited in the paper.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi twwode..ooo..... com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
19

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[Yes]
Justification:
The demo dataset will be released with well documentation.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
20

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
Justification: We clearly state the usage of VLM in our method.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy (https: http:////nuuuupeeeeeee cc/ http:ppssccccooeeeeeene////////0 for what should or should not be described.
21


Reviews:
[{"review_id": "dwcm2JBaB0", "reviewer": "Reviewer_mwjJ", "summary": "In this paper the authors present a new model for inferring a force field of “wind” by analyzing videos. This is performed by tying together a differentiable physics engine with sparse point tracking (along side a VLM for property estimation). This method recovers known forces in natural and synthetic videos, produces qualitatively reasonable vector fields for natural videos of objects flapping in the wind, and allows for video editing by changing known properties (e.g., making an object twice as heavy or the wind twice as strong)", "questions": "1. As mentioned above, the only real-world application presented for this model is the ability to edit or generate videos. Are there other applications for this work that might be of use to the broader NeurIPS or computer vision community?\n2. All of the examples in this paper present videos with a single focus object (e.g. one leaf or a swing). Would this model work with multiple objects? With particulate matter (e.g. snow)? Or is the object modeling limited to a single thing?\n3. A VLM is used in the object identification phase to identify material properties to add latent physical properties, and against a small benchmark provides good property estimation. But how important is this to the larger model – e.g., if you assume only average information, or a misidentification (e.g. something that looks like cloth but acts like cardboard) how brittle is the full model?\n4. What types of forces are used in the synthetic video set? Are the uniform in magnitude and direction but time varying? Or can the wind be blowing in two directions at once? Looking at Figure 4, the force field appears to be pointing in the same direction for each frame, and the forces appear to have a smooth gradient across the frame (generally peaking around the object) so it’s not clear whether this is just because the examples focus on a single object with wind where local direction and magnitude estimates are most important, or whether this is a heuristic found by the model", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 1, "originality": 4, "strengths_and_weaknesses": "This paper is clear and well written, and presents an interesting and novel model for inferring physical forces from videos, especially working on soft bodies. However, its main weakness is that it is not clear what this algorithm could be used for beyond the video editing examples presented in this paper – which make up a small part of video editing and generation as a whole – and thus it is not clear how broad the audience for this work would be beyond those finding inspiration in the algorithm for other purposes", "was_revised": true, "final_justification": "The authors appropriately addressed my (and other reviewers') comments. I believe it would be nice to have an explicit test of the effect of the VLM on model performance, but this is a minor point and does not affect my overall rating. Thus my rating (Accept) stands", "rebuttal": "We would like to express our sincere gratitude for your thoughtful and detailed feedback. Your comments have provided valuable insights, and we are eager to address them in detail.\n\n> **Q1**: As mentioned above, the only real-world application presented for this model is the ability to edit or generate videos. Are there other applications for this work that might be of use to the broader NeurIPS or computer vision community?  \n\n**A1**: Beyond video generation and editing, one of the potential applications of our work is in **augmented and virtual reality (AR/VR)**, where our method can model physics-driven behaviour of virtual objects. This could enable more immersive AR/VR experiences by dynamically adjusting objects' movements based on simulated forces like gravity or wind, which could be particularly useful in interactive gaming environments.\n\nAnother potential application is in **robotics and simulation**. By estimating the forces involved in object interactions, our framework could enhance robotic manipulation in dynamic environments—for example, guiding pneumatic non-prehensile “blowing’’ systems that corral objects into a receptacle [1]. This could improve robots’ ability to handle deformable or flexible items by modelling the forces acting on them, leading to better motion planning and control.\n\n[1] Wu, Jimmy, et al. \"Learning pneumatic non-prehensile manipulation with a mobile blower.\" IEEE Robotics and Automation Letters 7.3 (2022): 8471-8478.\n\n\n> **Q2**: All of the examples in this paper present videos with a single focus object (e.g. one leaf or a swing). Would this model work with multiple objects? With particulate matter (e.g. snow)? Or is the object modeling limited to a single thing?\n\n**A2**: Our model **does support multiple objects**. For example, Figure 1 illustrates a scenario with four stickers, each modeled individually, and Figure 5 shows four sakura petals being modeled independently. The pipeline allows for separate modeling of each object’s geometry, physical properties, and interactions, making it compatible with multi-object scenarios by treating each object as an independent entity while still modeling the overall force field acting on them.\n\nRegarding particulate matter like snow, as we mentioned in our limitation section, we acknowledge that our model is primarily designed for objects with small or bending-only deformation. That said, our method can model particulate matter with a particle-based differentiable physical process; this is beyond the scope of this work and we leave it for future exploration. \n\n\n> **Q3**: A VLM is used in the object identification phase to identify material properties to add latent physical properties, and against a small benchmark provides good property estimation. But how important is this to the larger model – e.g., if you assume only average information, or a misidentification (e.g. something that looks like cloth but acts like cardboard) how brittle is the full model?\n\n**A3**: We found that if the physical parameters are in a reasonable range, our framework is quite robust to object identification. Since VLM contains commonsense knowledge about object physical properties, we found it is sufficient for our use. A misidentification in the case you referred to may lead to a failure in our model, but in our experiments, we didn’t notice such a severe hallucination of VLMs.  \n\n\n> **Q4**: What types of forces are used in the synthetic video set? Are the uniform in magnitude and direction but time varying? Or can the wind be blowing in two directions at once? Looking at Figure 4, the force field appears to be pointing in the same direction for each frame, and the forces appear to have a smooth gradient across the frame (generally peaking around the object) so it’s not clear whether this is just because the examples focus on a single object with wind where local direction and magnitude estimates are most important, or whether this is a heuristic found by the model\n\n**A4**: Thank you for your inquiry! Firstly, for a clear and fair quantitative evaluation of our method, we set the forces to be time-varying but spatially uniform in our synthetic data. \n\nSecondly, yes, the wind can indeed be blowing in two directions at once. This is because we are capturing the force field at each time step, and the force field is a result of the combined forces acting on the object. In scenarios where multiple forces are acting simultaneously, they are captured as a cumulative force field, which can have different directions and magnitudes at different spatial locations in the scene.\n\nRegarding the observations in Figure 4, indeed the force field appears to be pointing in the same direction for each frame with a smooth gradient. This apparent uniformity is specific to that example and not a systematic tendency of our model’s estimates. In other examples, such as those in Figure 1, the force field is more complex and varies in both direction and magnitude, highlighting that the method is capable of handling more diverse scenarios, not just uniform forces."}, {"review_id": "z9gweJTaxI", "reviewer": "Reviewer_Utuv", "summary": "This paper aims to infer unseen forces (mostly wind) from real video. To do so, they construct an inverse graphics pipeline that takes into account the geometry of objects in the scene as well as their motion through time. This inverse graphics pipeline is differentiable, so can be optimized with gradient descent. This pipeline first extracts details about the objects in the scene using multimodal LLMs and segmentation to get info about material types and object boundaries. Then, modeling outcomes of inferred force fields as gaussians, they can optimize for the force field which produced the observed video. They demonstrate applications such as object insertion and force field modification.", "questions": "Please look at weaknesses. In general, I'm inclined to vote for acceptance due to the creative setting/solutions. The reason my score is not higher is because the scale of the results/quality of the generations seems like they limit broader applicability. I also think a comparison to recent interactive simulation work is very important (e.g. DragGAN and \"Motion Prompting: Controlling Video Generation with Motion Trajectories\")", "limitations": "See above", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths\n+ Very creative idea. Inferring forces from video has been generally studied but I haven't not seen this specific approach that allows for deformable materials.\n+ Intuitive strategy. The paper is written in a way that the rather complex inverse rendering pipeline is fairly intuitive.\n+ Insertion results are interesting. They demonstrate that, given an inferred force field, they can insert new objects that obey those forces. This seems generally applicable in other fields for things like data augmentation.\n\n Weaknesses\n- Missing discussion on the substantially related interactive simulation line of work (e.g. DragGAN). This is very important. The authors should show when those methods fail relative to this one, as they can also infer motion through optical flow and allow for insertion/editing.\n- The \"in-the-wild\" qualitative results provided are entirely focused on wind, which limits the scope of applicability. There could be other unseen forces, e.g. gravity/object occluded actions. Also potentially interested in the elasticity results for in-the-wild data.\n- Qualitatively, the new object insertion results are strange. In particular, for the paper airplane, the leaves demonstrate a slight upward lift close to the start of the video which is not reflected in the airplane's trajectory. As such, I would appreciate some more uncurated examples for the rebuttal.\n- All results/demo videos are done with respect to short video and a static camera. With the existing pipeline, it seems difficult to extend many of the results to real video with dynamic cameras (which could be important for broader applicability).", "was_revised": true, "final_justification": "I am inclined towards acceptance, however the lack of more complex scenes/examples preclude a higher score. I understand that theoretically, this framework can be extended to more complex scenes. However the devil is in the details when dealing with video inference/generation, so I would need to see the execution. Unfortunately, the format of the rebuttal does not allow for this. Hopefully I can see these examples in the final revision.", "rebuttal": "Thank you for your comprehensive and constructive feedback. We truly appreciate the time you spent reviewing our paper and are grateful for your suggestions, which we will address point by point.\n\n> **Q1**: Missing discussion on the substantially related interactive simulation line of work (e.g. DragGAN). This is very important. The authors should show when those methods fail relative to this one, as they can also infer motion through optical flow and allow for insertion/editing.\n\n**A1**: DragGAN and Motion Prompting are relevant to our work. The key difference is that **these methods do not follow an explicit physical law**. These methods infer motion through optical flow, using the term \"force\" to represent a direction of motion, which is more akin to displacement than to actual physical forces. In contrast, our approach explicitly models the underlying physical interactions—such as wind, gravity, and interaction forces—that govern the motion and deformation of objects. This distinction is crucial because our method estimates real physical forces, enabling more accurate and physically plausible simulations, especially in complex scenarios where precise force fields need to be recovered. **In the revision, we will add a dedicated discussion of both papers and include a qualitative comparison to highlight these distinctions.**\n\n\n\n> **Q2**: The \"in-the-wild\" qualitative results provided are entirely focused on wind, which limits the scope of applicability. There could be other unseen forces, e.g. gravity/object occluded actions. Also potentially interested in the elasticity results for in-the-wild data.\n\n**A2**: First, it’s important to note that our framework does **indeed account for gravity** in the scenarios, as gravity is inherently included in the physical processes modeled in our system. This is particularly evident in how we capture and simulate object motion under natural conditions, including the effects of gravity on objects in the scene. \n\nRegarding **elasticity and handling object occluded actions**, the results shown in Figure 1, particularly with the sticker example, demonstrate that our method is capable of handling objects with significant deformation and occlusion. The sticker example illustrates the model’s ability to track and simulate forces on objects undergoing elastic deformations, even in the presence of partial occlusion.\n\nIn summary, while the qualitative results presented focus on wind, our approach is capable of handling a broader range of forces, as demonstrated by the sticker example and other scenarios.\n\n\n> **Q3**: Qualitatively, the new object insertion results are strange. In particular, for the paper airplane, the leaves demonstrate a slight upward lift close to the start of the video which is not reflected in the airplane's trajectory. As such, I would appreciate some more uncurated examples for the rebuttal.\n\n**A3**: Thank you for noting that the leaves rise slightly at the start, whereas the inserted paper airplane stays nearly level. This difference in behavior is due to the **different physical properties** of the two objects, such as mass and aerodynamic characteristics, which affect how they respond to the forces applied in the simulation. The leaves, being lighter and more affected by the wind, show a noticeable upward lift, while the heavier paper airplane reacts differently to the same forces.\n\n> **Q4**: All results/demo videos are done with respect to short video and a static camera. With the existing pipeline, it seems difficult to extend many of the results to real video with dynamic cameras (which could be important for broader applicability).\n\n**A4**: Our model **doesn’t assume short video or a static camera**. As long as we can obtain a robust estimation of object motions in 4D, our method can infer forces from the motions. With the recent advances in 4D reconstruction and camera pose estimation, such as Shape of Motion[1] and MegaSAM[2], our method can be naturally extended to moving cameras.  \n\n[1] Wang, Qianqian, et al. \"Shape of motion: 4d reconstruction from a single video.\" arXiv preprint arXiv:2407.13764 (2024).\n\n[2] Li, Zhengqi, et al. \"MegaSaM: Accurate, fast and robust structure and motion from casual dynamic videos.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025."}, {"review_id": "9vq0DDEYil", "reviewer": "Reviewer_PKhM", "summary": "This paper presents an end-to-end framework that combines differentiable simulation with differentiable rendering to extract an implicit force field from video observations. To stabilize the optimization of the force field, a sparse tracking objective is utilized. Experiments conducted on both synthetic and real-world videos demonstrate the effectiveness of the force estimation process. Additionally, the learned implicit force field can be generalized to new configurations, such as inserting new objects or increasing the magnitude of the force.", "questions": "1.\tSeveral existing studies [a-b] have demonstrated the ability to estimate a velocity field from video observations, which is analogous to the proposed method that estimates a force field (assuming constant mass). It is recommended that the authors discuss the differences between these studies and the proposed method. Additionally, they should present quantitative comparisons to determine which approach yields more effective results.\n\n2.\tIt is challenging to reconstruct the 3D scene given only the first frame of the entire video. How does the method handle scenes that include object occlusions? Could the rest frames provide some information for reconstructing the 3D scene?\n\n3.\tIn Section 3.1 of the manuscript, the authors mainly discuss the modeling of foreground objects. However, it is unclear whether the background is modeled in the same way as the foreground objects. If that is the case, how does the method obtain the physical parameters (e.g., mass and Young's modulus) for the background particles? Additionally, in the supplementary videos, while the original clips show noticeable background motion, the videos generated through physics-based motion editing do not display any background movement. Can the authors provide some clarification on this?\n\n4.\tIn Section 3.3 of the manuscript, what is the consideration for applying external forces on each particle before the particle-to-grid step, rather than applying them on each grid node, similar to internal forces? For Equation 13, have the authors considered alternative interpolation strategies (e.g., K nearest neighbors), and how do they perform? What is the ratio between the number of $\\mathbf{x}$ and $\\mathbf{P}$?\n\n5.\tThe examples presented in the paper primarily involve force fields that have similar directions and a fixed input view. Can the proposed method address the following scenarios: (a) cases where the force directions are distinct (for instance, two leaves float from the left and right edges of the image toward the center, influenced by opposing forces) and (b) input videos that capture both camera movements and object movements?\n\nReferences\n\n[a] NVFi: Neural Velocity Fields for 3D Physics Learning from Dynamic Videos. NeurIPS 2023.\n\n[b] FreeGave: 3D Physics Learning from Dynamic Videos by Gaussian Velocity. CVPR 2025.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "This paper introduces a unified framework for estimating implicit force fields using only video observations. It presents a novel force representation that disentangles space and time with recursive dependency. Experiments demonstrate the framework's effectiveness in extracting force fields and applying the learned fields to new configurations. Despite its strengths, it also has some noticeable weaknesses, e.g., considering only very simple motions as inputs and failing to make comparisons with existing works. Overall, this work explores an intriguing direction by combining differentiable physics with video understanding, and it may inspire future research in similar areas.", "was_revised": true, "final_justification": "Most of my initial concerns have been addressed by the authors' response. While the experiments primarily demonstrate simple motions, which might be seen as a limitation of the study, I believe they could inspire future research in this field. Therefore, I tend to uphold my original rating.", "rebuttal": "We sincerely appreciate your valuable comments and suggestions. Your feedback has helped us refine our work, and we are excited to clarify and discuss each of your points in the following.\n\n> **Q1**: Several existing studies [a-b] have demonstrated the ability to estimate a velocity field from video observations, which is analogous to the proposed method that estimates a force field (assuming constant mass). It is recommended that the authors discuss the differences between these studies and the proposed method. Additionally, they should present quantitative comparisons to determine which approach yields more effective results.       \n> References      \n> [a] NVFi: Neural Velocity Fields for 3D Physics Learning from Dynamic Videos. NeurIPS 2023.             \n> [b] FreeGave: 3D Physics Learning from Dynamic Videos by Gaussian Velocity. CVPR 2025.\n\n**A1**: The key difference between our method and the mentioned NVFi and FreeGave is that **these methods don’t rely on explicit and differential physical processes** to model physical interactions. Specifically, these methods rely on a neural network that mimics the physical interactions and predicts particle velocities, with a physics-based regularization term to constrain the outputs. In contrast, our method leverages an explicit physical process to model the physical interactions and learn the force field via a differential physics engine. **In the revision, we will add a dedicated discussion of both NVFi and FreeGave to clarify these distinctions.** Thank you!   \n\n\n> **Q2**:It is challenging to reconstruct the 3D scene given only the first frame of the entire video. How does the method handle scenes that include object occlusions? Could the rest frames provide some information for reconstructing the 3D scene?\n\n**A2**: Our method relies on the first frame to initialize the 3D reconstruction of the scene, as described in Section 3.1. The 3D Gaussians representing the objects are initialized using pixel-aligned point clouds extracted from the first image via a pre-trained metric-depth model. This initialization is performed once using the first frame and does not get updated by subsequent frames.\n\nWhile the first frame provides the essential geometric initialization, we also mention in Section 3.1 that we have explored **both single-view (line 122) and multi-view (line 127) reconstruction techniques**. In the case of multi-view input, additional perspectives can further refine the 3D model, but the geometry itself is still primarily initialized from the first frame.\n\nRegarding object occlusions, the method handles them through sparse tracking of keypoints, which allows for robust motion tracking even in the presence of partial occlusion. By focusing on key points that are less likely to be occluded, we can maintain the accuracy of the 3D reconstruction and force estimation despite object occlusions.\n\n\n> **Q3**: In Section 3.1 of the manuscript, the authors mainly discuss the modeling of foreground objects. However, it is unclear whether the background is modeled in the same way as the foreground objects. If that is the case, how does the method obtain the physical parameters (e.g., mass and Young's modulus) for the background particles? Additionally, in the supplementary videos, while the original clips show noticeable background motion, the videos generated through physics-based motion editing do not display any background movement. Can the authors provide some clarification on this?\n\n**A3**: In our framework, we specifically model **only the foreground objects** and not the background. The background is treated as a static element in our simulation. The physical properties such as mass and Young’s modulus are assigned only to the foreground objects, and background is not modeled with these physical parameters. \n\nRegarding the lack of background motion: for all demos we assume the background is static. We extract the background from the first frame, inpaint occluded regions, and then composite our simulated foreground motion on top. This choice keeps the focus on foreground dynamics and sidesteps the additional complexity of modeling background physics.\n\nWe have indeed experimented with segmenting and inpainting every frame, then compositing our simulated objects into those reconstructed backgrounds. Current off-the-shelf inpainting techniques, however, still exhibit minor temporal flicker across frames, which can distract from our main contribution. For clarity of presentation, we therefore adopt the simpler static-background pipeline in the demos, allowing the reader to focus on the foreground physics.\n\n\n> **Q4**: In Section 3.3 of the manuscript, what is the consideration for applying external forces on each particle before the particle-to-grid step, rather than applying them on each grid node, similar to internal forces? For Equation 13, have the authors considered alternative interpolation strategies (e.g., K nearest neighbors), and how do they perform? What is the ratio between the number of x and P?\n\n**A4**: We chose to apply external forces to each particle rather than to the grid nodes to achieve finer force sampling. Because the scene volume is large compared with the objects, only a small fraction of grid cells are occupied in any given frame; applying forces at grid nodes would therefore leave most cells with zero force, yielding a coarse and noisy field.\n\nBy applying forces to particles, we can achieve more accurate and detailed force estimations, as the forces are directly linked to the moving particles, allowing for better resolution and precision. \n\nRegarding Equation 13, we selected barycentric interpolation because it works well for structured data like skeleton-driven animation. It ensures smooth transitions between keyframes, which is essential for our setup involving small deformations or bending-only deformations. While KNN interpolation could be useful for more irregular or unstructured data, it may result in less smooth transitions and be computationally expensive when dealing with large datasets.\n\nIn our experiments, the number of particles x typically ranges around 10k to100k, while the number of P (keypoints) is on the order of 10, so the ratio of x to P is generally in the range of 1k to 10k.\n\n\n> **Q5**: The examples presented in the paper primarily involve force fields that have similar directions and a fixed input view. Can the proposed method address the following scenarios: (a) cases where the force directions are distinct (for instance, two leaves float from the left and right edges of the image toward the center, influenced by opposing forces) and (b) input videos that capture both camera movements and object movements?\n\n**A5**: For Scenario A (distinct force directions): Our method is **already capable of handling situations where forces act in different directions**. In our real-world experiments, such as those involving in-the-wild videos, we demonstrated that the framework can successfully model forces that vary across the scene, e.g., in Figures 1 and 4. The force field is estimated at each point in the scene, allowing for distinct directions of force acting on different objects simultaneously. This flexibility in force direction handling has been validated in our experiments with random forces applied to objects, as shown in our results.\n\nFor Scenario B (camera and object movements): Our model **doesn’t assume a static camera**. As long as we can obtain a robust estimation of object motions in 4D, our method can infer forces from the motions. With the recent advances in 4D reconstruction and camera pose estimation, such as Shape of Motion[1] and MegaSAM[2], our method can be naturally extended to moving cameras.\n  \n[1] Wang, Qianqian, et al. \"Shape of motion: 4d reconstruction from a single video.\" arXiv preprint arXiv:2407.13764 (2024).\n\n[2] Li, Zhengqi, et al. \"MegaSaM: Accurate, fast and robust structure and motion from casual dynamic videos.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025."}, {"review_id": "tRHYvpL1Me", "reviewer": "Reviewer_MYah", "summary": "This paper presents an innovative inverse graphics framework for recovering invisible forces (such as wind) from video data. The key idea is to estimate dense, time-varying force fields by modeling object geometry and physical properties using 3D Gaussians, and simulating dynamics via a differentiable physics engine. A novel causal tri-plane representation encodes spatio-temporal force fields, and a sparse 4D tracking objective enables effective gradient-based optimization. Experiments on both synthetic and real-world data demonstrate strong performance in recovering force fields and simulating physically plausible object behavior.", "questions": "- How sensitive is the method to inaccuracies in depth estimation or object segmentation from vision-language models (VLMs) and SAM?\n- What directions might improve the realism of the generated and edited videos? Could this method eventually support practical applications such as content creation or simulation?\n- Can the framework be adapted for real-time use in mobile or robotics settings, where interactive or reactive force estimation is required?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "## Strength：\n\n1. The idea of estimating invisible forces purely from object motion is novel, underexplored, and highly compelling.\n2. The integration of 3D Gaussians, causal tri-plane force modeling, and differentiable physics is both elegant and technically effective.\n3. The proposed optimization strategy is well-motivated and successfully addresses common issues with gradient instability in differentiable simulation. The experimental evaluation is thorough and comprehensive, effectively validating the method’s performance.\n\n## Weaknesses:\n\n- **Limited applicability to object types**: As acknowledged in the paper, the proposed framework currently supports only solid objects with small or bending-only deformations. More complex materials such as fluids or granular substances remain out of scope.\n- **Heavy reliance on pretrained models**: The accuracy of the force estimation pipeline depends heavily on initial object geometry and physical property estimations derived from pretrained vision-language and depth models. This reliance could become a bottleneck in more complex or noisy real-world scenarios.\n- **Limited realism in generation and editing**: Although the model demonstrates strong performance in force recovery and simulation, the visual quality of the generated and edited videos appears noticeably less realistic compared to recent state-of-the-art video generation models. This limits the perceived quality and potential practical deployment of the method in high-fidelity applications.", "was_revised": true, "final_justification": "The paper presents a well-rounded and complete study. The proposed method is novel, and the experimental evaluation is thorough and convincing. The author response addressed my initial concerns adequately. I have decided to keep my original rating (5: Accept).", "rebuttal": "We are grateful for your thorough and insightful review. Your feedback has highlighted several important aspects, and we will carefully address each of your concerns in our response.\n\n> **Q1**:  How sensitive is the method to inaccuracies in depth estimation or object segmentation from vision-language models (VLMs) and SAM?\n\n**A1**: Our method is generally robust to inaccuracies in depth estimation and object segmentation due to the following reasons. \nAs noted in Section 3.1, while object segmentation is important for assigning physical properties to the 3D Gaussians, SAM is generally reliable for this task, and the object-background separation it provides is usually sufficient for our method.\n\nInaccurate depth estimation can impact tracking, but our sparse tracking objective (described in Section 3.4) effectively handles this issue. By focusing on sparse keypoints rather than relying on dense depth data, we reduce the sensitivity to depth errors, making the method more robust to noise in depth estimation. You can also refer to Table 3 for related results.\n\nOur experiments on in-the-wild video in Section 4.2 show that the method performs well even with minor errors in depth or segmentation. While significant errors can still affect the results, the sparse tracking approach ensures that the force recovery process remains reliable.\n\nIn summary, our sparse tracking mechanism effectively mitigates the impact of depth estimation inaccuracies, as demonstrated in our experiments.\n\n\n> **Q2**: What directions might improve the realism of the generated and edited videos? Could this method eventually support practical applications such as content creation or simulation?\n\n**A2**: As demonstrated in Figures 4, 5, and 6, our method has shown potential in content creation and simulation for real-world applications. These examples highlight how realistic object interactions and force fields can be modeled in various scenarios.\n\nTo improve realism, future work could refine the physical simulation to handle complex materials, such as fluids or hyper-elastic deformations, and integrate more advanced models for collisions, friction, and lighting effects, enhancing both visual fidelity and physical accuracy.\n\nThis method could support practical applications like content creation and simulation. For example, in film production, it could simulate realistic environmental interactions, such as wind, while in virtual reality, it could enable physics-driven interactions. It could also be used in simulation for robotics and industrial design, where accurate object behavior under forces is critical.\n\n> **Q3**: Can the framework be adapted for real-time use in mobile or robotics settings, where interactive or reactive force estimation is required?\n\n**A3**: Our framework can be applied to offline robotic data to estimate physical interactions, and with this information we can train a robotic policy that can be deployed in the real world. For instance, accurate wind-force estimation is valuable for pneumatic non-prehensile “blowing’’ manipulation, where a mobile blower herds scattered objects into a receptacle[1]. We will investigate this direction in the future. Thank you for the advice!\n\n> **Q4**: Limited applicability to object types: As acknowledged in the paper, the proposed framework currently supports only solid objects with small or bending-only deformations. More complex materials such as fluids or granular substances remain out of scope.\n\n**A4**: Our current implementation is indeed limited to solid objects exhibiting only small or bending-dominant deformations; fluids and granular media remain beyond its scope. However, it can be addressed within our framework, albeit with a particle-based differentiable physical process. We will include more discussion in the limitation section of our paper and plan to address it in future iterations of the framework, where we hope to expand its applicability to a wider range of materials.\n\n> **Q5**: Heavy reliance on pretrained models: The accuracy of the force estimation pipeline depends heavily on initial object geometry and physical property estimations derived from pretrained vision-language and depth models. This reliance could become a bottleneck in more complex or noisy real-world scenarios.\n\n**A5**: As discussed in Q1 and Section 3.4, our framework is designed to be resilient to errors in these initial estimates. The sparse tracking objective focuses on sparse, reliable keypoints for motion tracking. This approach helps mitigate the potential bottleneck caused by errors in the pre-trained models, particularly in real-world scenarios where noise and complexity may arise.\n\nThat being said, we agree that further improvements in the robustness of pre-trained models, especially in noisy or complex environments, would benefit our approach. We continue to explore methods to refine the initial object geometry and physical property estimations in future iterations of the framework.\n\n\n> **Q6**:  Limited realism in generation and editing: Although the model demonstrates strong performance in force recovery and simulation, the visual quality of the generated and edited videos appears noticeably less realistic compared to recent state-of-the-art video generation models. This limits the perceived quality and potential practical deployment of the method in high-fidelity applications.\n\n**A6**: Currently, the visual quality is constrained by the limitations of neural rendering techniques, but this can be addressed by integrating diffusion models to rectify the artifacts from neural rendering or using them as motion guidance to control a video diffusion model, similar to approaches like Difix3D[2]. This will be explored in future work.\n\n[1] Wu, Jimmy, et al. \"Learning pneumatic non-prehensile manipulation with a mobile blower.\" IEEE Robotics and Automation Letters 7.3 (2022): 8471-8478.\n\n[2] Wu, Jay Zhangjie, et al. \"Difix3d+: Improving 3d reconstructions with single-step diffusion models.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025."}]

Author rebuttal:

Meta-review: This work is aimed at recovering the physical force field from input videos. This is achieved using a differentiable inverse graphics framework which can model multiple objects which can undergo time-varying small bending deformations. Metric depth prediction models, object segmentation, and multi-modal LLMs are used to initialize the object depth, boundary, and physical properties like mass and Young's modulus. A tracking objective based on sparse key point tracking (using co-tracker) is also used, which helps with occlusions. Applications show video generation with object insertion, and force field modification.

The reviewers appreciated the novelty and creativity of the approach, esp. in modelling deformable objects, and were had positive ratings for this work. There were a few concerns regarding comparisons to prior (learning based) works, and, limited application domain, e.g., wrt moving cameras. In view of the above, I'm inclined to recommend this work for acceptance. I encourage the authors to address the limitations highlighted by the reviewers and incorporate their valuable feedback.
Decision: Accept (poster)