Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes Physics Augmented Continuum Neural Radiance Fields (PAC-NeRF), to estimate both the unknown geometry and physical parameters of highly dynamic objects from multi-view videos. The core idea of this paper is a hybrid Eulerian-Lagrangian representation of the neural radiance field. The proposed method on a wide variety of complex systems outperforms state of the art methods by up to two orders of magnitude. 
strength_and_weaknesses: Strengths:
1. This motivation of this paper is very clear and the problem studied in this work is of great value to the field of machine vision.
2. The proposed method has strong theoretical support and easy to understand. 
3. Extensive experiments are conducted to validate the performance of the proposed method. 

Weaknesses:
Since there are no existing approaches that estimate both geometry and material properties from videos, the authors make a best-effort comparison by blending combinations of state-of-the-art approaches for each sub-task. Such comparison seems not sufficient. 
clarity,_quality,_novelty_and_reproducibility: This paper is written clearly and easy to understand. The novelties of this paper are significant. 
summary_of_the_review: In summary, this paper studies an interesting problem and propose an effective solution. Experiments demonstrate it effectiveness.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a new NeRF model, PAC-NeRF, that incorporate physical particle dynamic into the estimation to simultaneously estimate particle parameters and geometries. The particle dynamics are assumed to follow the conservation laws of continuum mechanics, and a hybrid Eulerian-Lagrangian representation is proposed to solve this dynamic system. The proposed method was trained and evaluated using synthetic data, and with the high quality inputs that satisfied the particle system assumption, it demonstrate better results than the previous dynamic-NeRF (D-NeRF) model.
strength_and_weaknesses: Strength: Including particle dynamics into NeRF model is novel and it is a very useful to extend NeRF reconstruction beyond static scene/rigid object movements.

Weakness: The proposed method was only evaluated using synthetic data. It would be great if the authors can include some real-world examples as the real-world observation might not strictly follow the particle dynamic system studied in this paper. Yet, I would expect to see some interesting reconstructions based on the proposed model.
clarity,_quality,_novelty_and_reproducibility: The paper is overall clear. I believe this work is interesting and should be able to expand the scope of NeRF reconstruction. 
summary_of_the_review: Please check my comments above. I like this paper and I think this work is of high quality to be accepted.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper presents a method to recover a scene's geometry and physical parameters from a collection of images. It combines a neural radiance field method with a differentiable physics engine to jointly recover a deforming object's geometry, color, and physical parameters using continuum mechanics modeling. 

The NERF model is dynamic to adapt the geometry over time and is based on the direct voxel grid optimization proposed by Sun et al. (2022) that greatly speeds up training a NERF. The differentiable physics engine is based on the MPM included in DIFFTAICHI. 

The paper describes the required steps to convert from the NERF representation of the geometry to particles that move with physical constraints. The NERF and physical parameters are recovered by optimizing a photometric cost function where the physical constraints are implicitly included in the dynamic part of the NERF.    

The experiments use synthetically generated scenes with various continuum materials, including Newtonian fluids, non-Newtonian fluids, granular media, deformable solids, and plasticine.


strength_and_weaknesses: Strengths

- The combination of NERF and differentiable continuum mechanics to identify a system's physical parameters and geometry is new. 
- The method is well constructed, and each step is properly justified. The change from Eulerian to Lagrangian is a very clever way to combine the physical constraints and the NERF model. 
- The results are impressive, even though it only uses synthetically generated scenes. The method is even outperforming the oracle baseline, where the geometry is known. 

Weaknesses
- The paper does not include results with real data, which leaves some questions unanswered. (see my comments in the next section)
- The dataset includes only gravity as an external force and a ground plane whose position is known. Other more complex scenarios are not dealt with in the paper. 
- The camera poses must be accurate and vary significantly to obtain good results with a NERF method. In practice, this introduces some constraints on the scene, such as a rigid background with a sufficient amount of texture to run structure-from-motion. These constraints are a bit ignored in the paper.  
clarity,_quality,_novelty_and_reproducibility: Clarity

The paper is well-written, but it's not always easy to follow, and some details are only superficially discussed, although that is understandable in a conference publication. Some details about the implementation are somehow hidden or difficult to find :

- Is the method able to automatically distinguish between Newtonian fluids, non-Newtonian fluids, granular media, deformable solids, and plasticine? Does each material class need to be fixed beforehand since it employs different constitutive equations (strain-stress)? The system would need a material property classifier in a more general application.  
- The material parameters seem to be uniform. What could be the adaptation for non-uniform materials? Does the MPM support more complex materials?
- Why is the Oracle baseline so poor in comparison with other methods? Is there any logical explanation for it?.

Quality

The paper is of high quality in general and solves a very difficult problem. The results are well-presented and motivated. The state-of-the-art is well-evaluated, and the experimental evaluation is convincing.

Novelty

This paper combines two existing methods (NERF and a Differentiable Physics Engine) to solve a very complex problem. These methods work with very different representations. The paper proposes a well-grounded strategy to combine them and shows that it works well in photorealistic scenes with large deformations.  It makes the paper novelty more than sufficient. 

Reproducibility

Some of the details regarding the MPM method are hidden. It would be very beneficial for the community to publish the code or a more detailed report about the work. 



summary_of_the_review: The paper tackles a very difficult problem by combining two recent methods, mixing NERF with continuum mechanics models to identify the physical parameters of a scene from monocular images. 

It is well-written and sufficiently clear.

The results are convincing
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This article presents a novel system that brings differentiable physics and neural radiance fields for dynamic scenes by augmenting a NeRF with a differentiable continuum dynamics model. A hybrid Eulerian-Lagrangian formulation is defined in this system to permit advecting geometry and appearance. This framework outperforms previous approaches validated by a wide variety of systems.
strength_and_weaknesses: Strength:
(1) It is interesting to combine NeRF with physical constraints.
(2) This paper is well written.
(3) In most cases, the results are encouraging.

Summary of Weakness:   
(1) Euler method and advection equation are both very fundamental and usually applied in highly restricted mathematical or physical models, which are not defined directly so that the theoretical level of this work satisfies a more general case.   
(2) There are some limitations as mentioned before, which are not listed in the article clearly.    
(3) Part of the experimental results is difficult to understand, disobeying universal physics phenomenon.   
(4)The authors haven’t listed the source of their ground truth data.

Main Questions:   
Though part of the results shown in this article reflects stable properties of the framework, the authors may not be familiar with the fluid dynamics field. Here are some following questions:   
1) This article is aimed to estimate both the unknown geometry and physical parameters of highly dynamic objects from multi-view videos. It is noticed that a droplet is regarded as Newtonian fluid, but this property would be destroyed when velocity exceeds a boundary value. Additionally, in the general case, the descent of a droplet drop to the ground may not satisfy the two assumptions.
2) As for fluid dynamics fields, what is the source of the listed ground truth data (In house codes, CFD software, or open-source codes)? It cannot be made sure whether this ground truth data is convincing or not.   
3) It is conspicuous that the method presented in this article requires the properties defined as continuum and conservation. Hence, listing the limitations clearly is necessary.

Things to improve the paper that did not impact the score:   
(1) Clarify the limitations of the framework;   
(2) If simulations of physics models are unavoidable, fundamental knowledge of other fields is necessary to present meaningful outcomes. For example, fluid dynamics is necessary for the authors to get familiar with.  
(3) Illustrating what kinds of physics models could be solved by the Euler scheme and governed by the advection scheme with low accuracy loss and time steps cost.   
clarity,_quality,_novelty_and_reproducibility: This paper is well-written, and most results are promising. The idea is novel, and I think it is reproducible.
summary_of_the_review: I have listed my questions in Weakness. It is necessary to explain the results of droplets and the limitation of the proposed method. Overall, it is a nice conference paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper aims at recovering both the geometric structure and the physical properties of objects from multi-view video sequences. The proposed approach combines a neural radiance field (NeRF) with a differentiable physics engine to jointly recover objects' geometry, appearance, and physical parameters. A hybrid Eulerian-Lagrangian formulation is introduced to combine NeRF and differentiable physics so that advecting geometry and appearance. Experiments are conducted on various synthetic videos including a wide range of objects with varying environment lighting conditions and ground textures. 
The paper initially received positive reviews, with three accept (8) recommendations, and one borderline accept (6) recommendation. The reviewers pointed out the relevance and novelty of the methods, especially the non-trivial combination of NeRF and differentiable physics with the hybrid Eulerian-Lagrangian formulation. The limitations pointed out by reviewers were related to the absence of evaluation on real data, and requested to specify the assumptions of the methods and its regime of validity. During the rebuttal period, the authors added a new sets of experiments on real data, and provided a discussion regarding the assumption if the approach and its limitations. The paper has been updated accordingly. 

The AC carefully reads the submission and discussions. The AC considers that the method is a solid combination between NeRF and differentiable physics, with a particularly neat solution based on the hybrid Eulerian-Lagrangian formulation. The approach pioneers the joint estimation of geometrical and physical properties from image sequences, and can thus have an important impact in the field. Since the experimental comparisons do not show any direct competitors, the added evaluations on real data is important. The discussion on limitations of the approach on the updated paper is also helpful for further developments. The AC thus recommends paper acceptance.
justification_for_why_not_higher_score: - Limiting assumptions: robustness with respect to camera pose, fixed material class

justification_for_why_not_lower_score: - Very nice combination between NeRF and physics simulators using the Eulerian-Lagrangian formulation
- Pioneer work with a important potential impact
Decision: Accept: notable-top-25%