MovingParts: Motion-based 3D Part Discovery in Dynamic Radiance Field
Authors: Kaizhi Yang
Xiaoshuai Zhang
Zhiao Huang
Xuejin Chen
Zexiang Xu
Hao Su
Venue: ICLR 2024 spotlight (2024)
Keywords: NeRF
Dynamic
Motion
Part discovery
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We present MovingParts, a NeRF-based method for dynamic scene reconstruction and part discovery based on motion.

Abstract: We present MovingParts, a NeRF-based method for dynamic scene reconstruction and part discovery. We consider motion as an important cue for identifying parts, that all particles on the same part share the common motion pattern. From the perspective of fluid simulation, existing deformation-based methods for dynamic NeRF can be seen as parameterizing the scene motion under the Eulerian view, i.e., focusing on specific locations in space through which the fluid flows as time passes. However, it is intractable to extract the motion of constituting objects or parts using the Eulerian view representation. In this work, we introduce the dual Lagrangian view and enforce representations under the Eulerian/Lagrangian views to be cycle-consistent. Under the Lagrangian view, we parameterize the scene motion by tracking the trajectory of particles on objects. The Lagrangian view makes it convenient to discover parts by factorizing the scene motion as a composition of part-level rigid motions. Experimentally, our method can achieve fast and high-quality dynamic scene reconstruction from even a single moving camera, and the induced part-based representation allows direct applications of part tracking, animation, 3D scene editing, etc.

Published as a conference paper at ICLR 2024
MOVINGPARTS: MOTION-BASED 3D PART DISCOV- ERY IN DYNAMIC RADIANCE FIELD
Kaizhi Yang'. Xiaoshuai Zhang', Zhiao Huang", Xuejin Chen', Zexiang Xu Hao Su
University of Science and Technology of China, University of California, San Diego,
3 Adobe Research
ykz09230mail uustceed..cm (xiz040, 22huang, saosu)@eng.ucsd.cc x xjshen99@uste. edw.ssstteeeeeeeeeeee.eeeeee.... en, zexu@adobe. com
ABSTRACT
We present MovingParts, a NeRE-based method for dynamic scene reconstruction and part discovery. We consider motion an important cue for identifying parts, that all particles on the same part share the common motion pattern. From the perspective of fluid simulation, existing deformation-base methods for dynamic NeRF can be seen parameterizing the scene motion under the Fulerian view, i.e., focusing on specific locations in space through which the fluid flows as time passes. However, it is intractable to extract the motion of constituting objects or parts using the Eulerian view representation. In this work, we introduce the dual Lagrangian view and enforce representations under the Eulerian/Lagrangian views to be cycle-consistent. Under the Lagrangian view, we parameterize the scene motion by tracking the trajectory of particles objects. The Lagrangian view makes it convenient to discover parts by factorizing the scene motion as a composition of part-level rigid motions. Experimentally, fast and high-quality dynamic scene reconstruction from even a single moving camera, and the induced part-based representation allows direct applications part tracking, animation, 3D scene editing, etc.
method can achieve
INTRODUCTION
3D scene reconstruction and understanding is one of the central problems in computer vi- sion and graphics, with a wide range of appli- cations in mixed reality, robotics, movic pro- duction, etc. While many works focus on static scenes, real-world physical scenes usually dynamic and entangled with illumina- tion changes, object motion, and shape defor- mation. The reconstruction of dynamic scenes known be highly challenging. Non- rigid structure from motion methods (Bregler al.) (2000): Gotardo & Martinez (2011):
Eulerian View
Flow Specific Locations
Lagrangian View
Tracking Specifie Particles
Canonical Space
Time 1 World Space
Figure Location-based Eulerian view vs. particle-based Lagrangian view. The Eulerian view observes the flow at specific location (akin the approach taken by deformation-based dy- namic NeRF) and the Lagrangian view observes the trajectory of specific particles. These two views constitute the conversion between the dy- namic world space of each temporal frame and static canonical space.
Kong & Lucey (2019); [Sidhu et al. 20200)
could recover nonrigid shapes but are limited to sparse feature tracking. To reduce the am- biguity between shape and motion, some other methods introduce multi-view capture (Zhang ct al. (2003); Ossald et al. (2014); Tung et al. (2009) ) category-specific priors (Egger et al.
2021 Habermann et af.) (2019); Kocabas et al.
2020 ). Recently, neural radiance representa-
tion NeRF (Mildenhall et al. (2020)) has been applied to this field and achieved promising dynamic capture performance on general scenes using only monocular input (Pumarola al.) (2020); Park et al. (2)) Li al. (2022)). However, most NeRF-based dynamic scene modeling methods only focus on scene reconstruction without considering scene understanding, thus lacking the ability to directly support downstream applications that need tracking, shape editing, re-animation, etc.
ddeottes equal advisory and denotes the corresponding author.

---

Published as a conference paper at ICLR 2024
Our goal is to enable practical dynamic scene capture with both high- reconstruction and meaningful scene understanding from monocular input. To this end, we propose MovingParts, a novel NeRF-based approach that can achieve not only fast dynamic scene reconstruction but also automatic rigid part discovery. Our key insight is that motion (while complicating the reconstruc- tion) is an effective eue for identifying object parts because the scene content belonging to one rigid part has to share the same rigid transformation. Therefore, we design novel modules to explain the motions in dynamic neural fields, enabling unsupervised object part discovery via motion grouping. Since the rigid motion patterns are used as the evidence of part discovery, we make explicit the assumption of our input here, which is the general scene with piece-wise rigid motion.
Our approach is inspired by the literature on fluid simulation. We note that family of previous dynamic NeRF methods model motion using a 3D field that encodes scene flow (Li et al. (202T) or deformation (Pumarola et al. (2020); Park et al. (202T))) Specifically, at time f, for each location x, the 3D field encodes which particle x in the canonical frame has been deformed to x, which actually backward deforms the particles from world space to static canonical space. As shown in Figure I this is essentially the Eulerian view in fluid simulation (Fedkiw et al. (200T)) - motion information is denoted as function L (xxx, t) at each specific location x in the world coordinate frame. It is known that, while the entire scene motion can be modeled under the Eulerian view, specific object/part motion different temporal moments is actually intractable and hard to analyze. On the other hand, the Lagrangian view (Macklin et al.) (2014)) as the duality of the Fulerian view, uses the particle- based representation to track the motion of each particle belonging to the object, which forward deforms the particles from canonical space to world space. The constructed particle trajectory from the Lagrangian view can be an important clue for scene analysis. A relevant Lagrangian-base work is Watch- Mo-e (Noguchi et al. (2022) which composes objects into several ellipsoid-like parts by rendering supervision. However, the multi-view requirement and the ellipsoidal geometric prior highly limit its application. In contrast, we mainly focus on monocular input.
To achieve meaningful scene understanding by motion analysis, we propose a hybrid approach that learns motion under both the Fulerian and the Lagrangian views. In particular, our neural dynamic scene model consists of three modules: (I) a canonical module that models the scene geometry and appearance as radiance field in a static canonical space, (2) an Eulerian module U (x, t) that records which particle x, in the canonical space passes through each specific location x in the world coordinate frame at every time step, and (3) a Lagrangian module V (x.. t) that records the trajectory of all particles x, in the canonical space. Note that the motions modeled by the Eulerian and Lagrangian modules are inherently reciprocal, we, therefore, apply cycle-consistency loss during reconstruction to enforce the consistency between the two modules, constraining them to model the same underlying motion in the scene.
The construction of the Lagrangian view makes it convenient to discover parts by factorizing & (x t). As the particles in a rigid part share a common rigid transformation pattern, we pro- pose a novel motion grouping module as part of our Lagrangian module. By projecting the particle motion features into a few groups, we divide the seene into meaningful parts. Once reconstructed, our Lagrangian module could offer part-level representation and allow for direct downstream ap- plications such as part tracking, object control, and scene editing. Since the number of rigid parts generally differs across scenes, we introduce an additional post-processing merging module that can adaptively merge the over-segmented groups into a reasonable number of rigid parts.
We jointly train all modules with only rendering supervision. We demonstrate that our approach achieves high-quality dynamic scene reconstruction and realistic rendering results on par with state- of-the-art art methods. More importantly, compared with previous monocular NeRF methods, ours is the only one that simultaneously achieves part discovery, allowing for many more downstream applications. Finally, inspired by recent fast NeRF reconstruction methods (Sun al. (2022); Chen let al. 2 Yu et al. (2022), we construct our system with feature volumes and weiht multi- layer perceptrons (MLPs), leading to fast reconstruction speed comparable other concurrent methods that are specifically focused on speeding up dynamic NeRF.
In summary, our key contributions are:
We propose a novel NeRF-based method for simultancous dynamic scene reconstruction and rigid part discovery from monocular image sequences;
The hybrid representation of feature volume and neural network allows us to achieve both high-quality reconstruction and reasonable part discovery within 30 minutes;
2

---

Published as a conference paper at ICLR 2024
The extracted part-level representation can be directly applied to downstream applications like part tracking, object control, scene editing, etc.
2 RELATED WORK
Dynamic Neural Radiance Fields. Recently, the emergence of Neural Radiance Fields (NeRF)
(Mildenhall et al. 2020)) has facilitated the tasks of scene reconstruction and image synthesis. Due to the dynamic properties of the physical world, an important branch of NeRF research is to extend it dynamic scenes (Pumarola et al. (2020): Li et al. (2021); Park et al.) (202Tb); Pridovich-Keil
eet al. (202)); Cao & Johnson| ). In particular, some methods directly extend the 5D radiance field function 6D by adding additional time-dependent input to the network (Li et al. (2022);
Xian et al. (222I). ).. Other works enhance the temporal consistency in the 6D dynamic radiance field
by explicitly modcling dynamic scene flows (Li et al. (202T); Du al. (202T);
) Gao et al.] (2021) ).
leading to promising results from only monocular input. Meanwhile, deformation modules have also been adopted in deRF-based methods (Pumarola et al. (2020); [Tretschk al., (202T): :: Yuan
et al. (202T): Park et al. 2021aa b): Liu et al. (2022)), offering strong regularization for temporal consistency. Note that these various NeRF-based methods all explain motions (modeled as flows or deformation fields) from the location-based Eulerian view and do not support part discovery. We instead propose a hybrid model that models motions with both location-bec based Eulerian and particle- based Lagrangian views, enabling high-quality dynamic scene reconstruction with automatic part discovery based on particle motion. In addition to these general methods, some NeRF methods have been devised for particular domains, such as humans (Jiakai et al.] (2021): NNguuhh al. (2021); Weng et al. (2022); Peng et al. (2023), and articulated objects within specific categories Wei et al. (2022). While capable of achieving dynamic scene rendering and part segmentation, these methods often mcorporate category priors into the pipeline and cannot be directly applied to general objects.
Part Discovery from Motion. At the image level, most motion-based object discovery methods (Keuper et al. (2015);; Pia Bideau| (016)); (ang et al. (2 et al. PPaaazolooo & Ferrari (2033) employ the clustering of 2D pixels based on features related to optical flow. We share a common underlying logic with these 2D methods that discover parts (or objects) by constructing and grouping motion trajectories. However, in contrast to these approaches, our method establishes motion group module on canonical 3D particles and relies on predicted 3D rigid motion, which ensures arbitrary viewpoints consistency and temporal consistency of the grouping results. In the 3D domain, some methods (Shi al.] (2021); Kawana et al. (2022) reason about object parts by constructing point-wise correspondence at different object states and clustering their trajectories. Without 3D input, (Agudo & Moreno-Noguer (2019h) adopts non-rigid structure from motion to reconstruct the 3D shape and applies spatio-temporal clustering to the 3D points to reason about segmentation. However, only the geometry of sparse feature points could be achieved. Recently, NeRF-based dynamic scene decoupling methods (Yuan et al. (2021): Tschernezki et al. (2021); Wu al. (2022) have been proposed. Although they achieve dynamic scene decomposítion with high- quality reconstruction, they can only divide the scene into static/dynamic parts and are unable to identify motion patterns. A relevant recent work is Watch-It- achieves high-quality part-level reconstruction from image sequences. However, it requires dense multi-view input and imposes ellipsoid-like priors to the part geometry, which may completely fail on challenging monocular data with complex scene geometry. In contrast, our NeRF-based method does not require any shape priors of dynamic objects in complex scenes and can achieve dynamic reconstruction and part discovery from monocular input.
(Noguchi et al. (2022). which
3 PRELIMINARIES NERF AND D-NERF
By incorporating implicit function and volume rendering. Neural Radiance Field (NeRF) (Milden- hall et al. (2020) allows for scene reconstruction and novel view synthesis via optimizing scene representation directly. In general, NeRF interprets static scenario as continuous implicit func- tion Fg. By querying spatial coordinates (x) and view direction (d). Fe outputs the corresponding density (a) and observed color (c) as (c, a) = Fe d). Through classical volume rendering in graphics, the 3D scene representation Fo can be rendered into 2D image. Specifically, given ray r emitted from the optical center to a specific pixel in the image, the rendered color of that pixel is an integral of all the colors the ray with near and far bounds h, and hyy
C(r) = 1/' hm
ths
(())((((()))(((()) d)dh, where T(h) = =xxp (- (-f"" o(r(s))ds). (1)
hn
3

---

Published as a conference paper at ICLR 2024
Eulerian Volume V/3
Eulerian Motion Modeling
Canonical Volume V
E
fmm
Dg
A Specifie Location
Rg. tp
Cycle Consistency
R7
Specific
g
Partiele
X
Dr
LINE
E
Lagrangian Motion Modeling
Lagrangian Volume V3
Figure 2: Overview of our method. Inspired by the Eulerian and Lagrangian viewpoints in fluid simulation, we designed three modules for motion-based part discovery in a scene. The Eulerian module and the Lagrangian module observe the motion of specific spatial locations and specific particles, respectively. They both comprise mutual mapping of a point between its position at an arbitrary time instance and its canonical configuration. The canonical module serves to reconstruct the geometry and appearance for volume rendering. Based the particle trajectories recorded by the Lagrangian module, we can analyze the motion patterns and discover rigid parts.
T(h) be interpreted the transparency accumulated from hn to h. Because of the inherent differentiability of Eq. [ it only requires set of images with camera poses to optimize Fg directly.
D-NeRF (Pumarola al. 2020)) extends NeRF capture dynamic scenes, assuming that there is static canonical space and includes all objects. It divides the dynamic scene reconstruction in world space into two sub-problems: the NeRF representation learning of canonical space and the learning of the mapping from the world space to the canonical space (scene flow prediction) as:
(c, .) = Fo (V d)
where (x, t) predict the canonical space position from at time into its canonical configuration.
4
OUR METHOD
From the perspective of fluid simulation, the scene motion is composed of particle motions. T (x, t) in D-NeRF can be interpreted as recording the motion of particles passing through a given coordinate at time 1, corresponding to the Eulerian perspective. Following D-NeRF, we also assume canon- ical space that is static and includes all objects. Besides the Eulerian perspective, we also describe the dynamic scene from the Lagrangian perspective. Accordingly, we construct three modules, as Pigure[]showe that include an Eulerian module V (x, t) which maps a position x at any time in the world space to the canonical space, Lagrangian module V L which tracks the trajectory of a particle corresponding to X, in the canonical space, and a canonical module which encodes the appearance and geometry in the canonical scene. Under the assumption of finite rigid bodies, we exploit the learned motion by the Lagrangian module and design a motion grouping module to discover moving parts. The particles in the same group share a common rigid transformation and should belong to the same part. Next, we will describe these modules and loss functions in detail.
4.1 CANONICAL MODULE
Same as NeRF, the canonical module is formulated as an implicit function Fx(x.; d) + (c, r) which encodes the geometry and appearance in a canonical space. To accelerate convergence, hybrid representation of feature volume and neural network is adopted. The queried canonical coordinate x. is first used to interpolate the corresponding features within 3D feature volume V. No,, XN.NKNN...CCC where the N₂ x N, x N₂ denotes the spatial resolution and C' is the feature dimension. To alleviate the local gradient artifact of grid representation, we adopt multi-distance interpolation and concatenate the features in different resolutions as (Fang et al. (2022)):
f. (x--Interp(xe: V) @ ... @ Tri-Interp(xx, V[: sM])-
After positional encoding, the queried feature with fed into ML.Ps to predict and c.
4

---

Published as a conference paper at ICLR 2024
4.2 EULERIAN MODULE
The Eulerian module V (x, t) records which particle X. in the canonical space goes through a specific location x at the query time t. Assuming that the scene is piece-wise rigid, we formulate this mapping as rigid transformation in (3) similar to Park et al. (021a), which ensures that all the points on the same rigid body can be transformed using the same set of parameters. Specifically, our Eulerian module contains three components. First, the 3D feature volume V stores the information about the particles that pass through each position during the entire observation period. Second, a motion extractor &g decodes the motion feature from the interpolated feature in VE E at query time t. Third, different from Park et al. (2021a) that uses a screw axis as an intermediate representation, rigid transformation decoder DE directly maps the motion feature to rotation and translation parameters. The overall process can be formulated as:
(Rg, tg) (f(fmm), where fmm Eg (Tri- (x, V Ve),t)
Here we employ the continuous 6D intermediate representation (Zhou et al. (2019) for 3D rotation Rg. The Eulerian mapping from the world space at each temporal frame to the canonical space can be calculated by:
X REx( -ta)
(5)
4.3 LAGRANGIAN MODULE
As the inverse of the Eulerian module, the La- grangian module V (x.:t) tracks the trajec- tories of specific object particles over time. We use the same manner to construct VL. & and Dr, Different from the Eulerian perspec- tive, the trajectories of each particle in the La- grangian perspective can be an important cue for rigid part discovery. All particles belonging to the same rigid part share the same rigid body transformation, which means that their motion can be represented by single feature vector. So we add an additional motion grouping net- work G (see Figure 3) after V t to restrict that particle trajectories are only subject to a finite number of rigid motion patterns.
h
average & assignment
straight through
(S')
The architecture of our motion grouping network G. We first calculate the similarity be- tween each fused feature fū and each learnable slot S', and then apply Gumbel-softmax with the straight-through trick to achieve hard grouping A. Finally, we assign fj to be the average of {ft } in its corresponding group by Equation g
Figure 3:
Similar to (Xu et al. (2022)), we use the atten- tion module with the straight-through estima- tor trick to achieve the hard grouping of La- grangian features. To encourage the spatial coherence of points in the same group, the coordinate of each point x is concatenated to the corresponding Lagrangian feature fi. Specifically, we first
compute the similarity map A between the feature {fix fī Gumbell-sftmas
@ x and learnable slots {S') by
exp(Wafix W + +
(6)
exp( Sffx
where Wg a and We are linear mappings and y is sample drawn from Gumbel(0, 1). Then the trriigh-through estimator trick is used convert the soft similarity map to one-hot formulation:
À
one-hot( Anngmaxx + A detach(A)
(7)
where the detach operation cuts off the corresponding gradient. Despite the hard conversion, Equa- tion 7 can still keep the gradient the same as A. The hard similarity matrix Å distributes all the Lagrangian features into several groups, where each group represents the particles with the same motion pattern. Instead of directly assigning the learned slot as the updated Lagrangian feature, we calculate the average of all the Lagrangian features in the same group to update the original Lagrangian features. In this way, each updated Lagrangian feature will be directly related to the Lagrangian grid Vz, L. allowing for more efficient optimization. This procedure can be formulated as:
Σ Au fi.
(8)
Σ
Aa
5

---

Published as a conference paper at ICLR 2024
Then the updated Lagrangian features fū, with query time t is fed into εn and Dr sequentially to decode the motion feature fim and the rigid transformation R2, tz. As mentioned in Section 4.4 to efficiently implement the cycle consistency between the Eulerian and Lagrangian modules, expect R = R and ty. tg- So the Lagrangian mapping from the canonical space to the world space each temporal frame is calculated by:
R (Xe tz.)
4.4 LOss FUNCTIONS
As our main optimization goal, we adopt the Mean Squared Error (MSE) between the rendered pixel color and the ground truth pixel color as our reconstruction loss:
Σ (((()) C(r) PER
(10)
R
We also use a total variation loss Liv to smooth the motion volumes and encourage motion similarity of spatial neighbors. Following [Sun et al.] (2022). the per-point color loss fper.pt and background entropy loss Lentropy are used to directly supervise the sampled point color and encourage the back- ground probability to concentrate around 0 or 1.
In addition, eyclic consistency loss is designed to encourage the reciprocity of the Lagrangian module and the Eulerian module. Instead of measuring the displacement of the transformations between these two views like Liu al. (2022). we found that accounting for the difference between low-level motion features fm and fEmm leads to more robust optimization and better part discovery. Our cycle loss is defined
Leycle
I
(11)
iPobjl
XEPaby
Please refer Appendix A. for cyclic consistency loss. Since the deformation of free space does not satisfy the assumption of finite rigid motions, we filter free space according density value and only calculate Leyele at sampled points objects {x
detailed discussion of these implementations of the
Ponj [xx e). In experiments, 10-1 The overall loss function is:
>
L Lpooro Weycle Ceyele + Wpprrppeppo.p Wentropy Ceatroy + Why Ltv-
(12)
4.5 GROUP MERGING MODULE
It is not reasonable to use the same number of groups for variety of scenarios. We generally set large number of groups as an upper bound on the number of rigid bodies in the scene, which may cause over-segmented results (Fig- ure 5). This is because we provide an exces- sive number of groups, and also the same rigid transformations could be casily represented by very different high-level motion features. To address this problem, we design an efficient heuristic algorithm for group merging based motion differences. This algorithm is used as protpesoonsing after training only and does not affect the parameters of the model. We summa- rize this group merging algorithm in Figure 4 and filter the free space points with density lower than the threshold c. 2) These remained points are fed into the Lagrangian module to get the updated feature fi,, which is the high-level represen- tation of each motion group. 3) We evaluate the rigid transformation similarity between each pair generated by decoding the updated slots into
Lagrangion
Sampled Times
Transformation Sequences
Feature [
Mergo Steps
1500 APV
Rigid
Transformation
Decoder
Figure The group merging procedure. We decode the average Lagrangian features of each group into rigid transformation sequences and de- termine the merge order as well as the stop step by evaluating the APE cost between the sequences.
1) We sample points uniformly in canonical space
of groups: The rigid transformation sequences tation and translation with uniformly sampled times between 0 and 1. 4) We use the Absolute Pose Eror (APE) measure the difference between each sequence pair:
AP = Σ (P -Pp;
P - 14x44|
(13)

---

Published as a conference paper at ICLR 2024
Beuncing Hall
Legn
Hell Warrfur
Matani
Roalevd
Gropp
Figure 5: Visualization of our results on the D-NeRF synthetic dataset. The first three rows show our rendered image, depth, and imitial grouping result at different timesteps. The last row shows our merged groups which demonstrates merging module's ability to adaptively aggregate groups with similar motions and achieve clean and accurate segmentation results.
Table 1: Comparison with NeRF-based methods. We indicate the best and second best with bold and underlined markings. We achieve high-quality rendering on par with the state-of-the-art method while reasonably discovering rigid parts, efficiently completing both tasks in about 26 minutes.
Method
Deformation
Part
Training Time
PSNR 1
SSIM f
LPIPS +
T-NeRF
X
hours
29.50
0.95
0.08
K-Planes
X
X
52 mins
31.61
0.97
HexPlane
X
10 mins
31.04
0.97
0.04
D-NeRF
Eulerian
X
20 hours
30.43
0.95
0.07
NDVG
Eulerian
23 mins
30.54
0.96
0.05
TiNeu Vox
Eulerian
X 28 mins
32.67
0.97
0.04
WIM
Lagrangian
~ hours
15.72
0.83
0.19
Ours
Eulerian&&Lagrangian
26 mins
32.18
0.97
0.04
where P is the transformation matrix of group i time 5) We recursively find the two groups with the smallest APE at the current step and record their merge APE cost until all the groups are merged into a single one. In the early stages, the groups with similar motion patterns are merged, which keeps the merging cost growth slow, Once groups representing different motions are merged, the cost will jump. indicating that the merging process should terminate. practice, we simply find the termination step with the largest cost increase to the subsequent step final result.
5 EXPERIMENTS AND RESULTS
Our method not only enables high-quality dynamic scene reconstruction but also allows for the discovery of reasonable rigid parts. In this section, we first evaluate the reconstruction and part discovery performance synthetic dataset with ground-truth motion masks to quantitatively evaluate our motion grouping results. Finally, we provide direct applications for structural scene modeling and editing.
our method on the D-NeRF 360 synthetic dataset. Then, we construct
5.1 IMPLEMENTATION
We use 50 x 50 x 50 voxels for the Eulerian and Lagrangian volume and a 160 x 160 x 160 voxel for the canonical volume. Following Fang al. (2022), we employ the progressive upsample the resolution for acceleration. We use two separated linear layers to predict the 6D rotation and 3D) translation with biases as O,, 0,0,0.000 0, 1, 0) and (0, 0), respectively, so that the initial deformation is an identity. We use the Adam optimizer for a total of 20k iterations, by sampling 4096 rays from randomly sampled image in each iteration. All the experiments were conducted on a single NVIDIA RTX3090 GPU. More details can be found in the appendix.
5.2 EVALUATION ON D-NERF DATASET
We adopt the 360 Synthetic dataset provided by D-NeRF (Pumarola et al. (2020)) to evaluate our method quantitatively and qualitatively. The dataset contains eight synthetic dynamic scenes with different motion patterns, and only one view is captured at each time step. We compare method with the state-of-the dynamic NeRF methods: fon-defformaiionbbssee methods T-NeRF (Pumarola al. 22020), K-Planes (Fridovich- Keil et al. (202))) HexPlane (Cao & Johnson 2023
7

---

Published as a conference paper at ICLR 2024
Inpet
GT
Groups
Trajecterivs
0.824
Figure 6: Motion grouping evaluation on our generated datasets. Our method is able to achieve high mIOU scores across various scene configurations and has demonstrated the ability to continuously track specific parts as well as handle complex geometry and topology.
Sulerian-based method D-NeRF (Pumarola et al. (2020)) TiNeu (Fang et al. 2022) ). NDVG (Guo et al. (2022) and Lagrangian-viewt method WIM (Noguchi et al. (2022). For TiNeu Vox, we use their base version with a canonical grid 160 resolution and hidden layers of 256 channels. Following these previous works, we train each scene with images at 400 x 400 resolution and use two metrics for evaluation: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM) (Zhou ct al. (2004)) and Learned Perceptual Image Patch Similarity (LPIPS) RRihaard et al. (2018)).
As shown in Table [ while keeping the training time within 30 minutes on one GPU, our method not only achieves high rendering quality but also supports part discovery. Compared to the previous methods, we achieved the best in SSIM and LPIPS, and second best in PSNR. Compared to TiNeu- Vox, method has slight PSNR drop. The main reason is that TiNeuVox employs temporal enhancement module in the canonical space to improve quality, which also leads to a time-varying canonical space. After removing this enhancement module in TixeuVox, its average PSNR drops to 31,47. In our paper, to achieve better disentanglement of geometry and motion, we expect the geometric evolution only comes from the scene motion. Therefore we did not adopt similar en- hancement strategy to form time-invarian canonical space. For WIM, due to the nonexistence of canonical space, the significant motion ambiguity under the single view setting causes the failure.
We show our visualization results in Figure 5 It can be seen that our method enables high-quality appearance and geometry reconstruction. We also assign each query point the corresponding group color and render it to 2D images. As discussed in Section 4.5, over-segmentation occurs because similar motion could be represented by different high-level features (see the third row Figure (5). Through our group merging algorithm, we only retain the highly distinguishable motion modes and obtain concise part segmentation. Thanks to the motion-bascd grouping mechanism, our method is capable of overlooking inreeeeannele characteristics in geometry and appearance and producing clean part discovery results on these realistic complex scenes.
5.3 MOTION GROUPING EVALUATION
In this section, we provide a quantitative evaluation of our motion grouping results. We created synthetic dataset with ground truth mmagessegmentttion pairs using Kubric toolkit (Greff et al. (0022) ). Each created scene contains 1 to 5 realistic real-world objects from the GSO dataset (Downs al. (022)) with different initial velocities and motion directions. We followed the same sampling and rendering process as D-NeRF (Pumarola et al. (2020) to generate 120-frame monocular image sequence with 256 256 resolution for each scene.
To begin our evaluation, we first establish the correlation pairs between the ground truth label and our predicted groups. For each group, we assign the ground truth label with the highest number of pixels corresponding to it in the first 10 frames. More details are included in the appendix. We calculate the mean Intersection over Union (IOU) for the assigned label mask with its corresponding ground truth mask over the entire image sequence. It is noted that achieving high mIOU score over the entire sequence requires more than just the ability to accurately distinguish each individual part. It also necessitates the capacity to consistently track each part throughout the sequence.
8

---

Published as a conference paper at ICLR 2024
We present 10 examples in Figure 0 showcasing both quantitative mIOU and qualitative visual- ization and comparisons. Despite the variation in the scene configurations, our method achieves an mIOU score of over 85% on most scenes, clearly demonstrating its robustness over the dataset. Moreover, the high mIOU score indicates that our method can generate accurate part segmentation results and continuously track speeific parts throughout the sequences, see the learned trajectories in Figure ensuring both temporal and multi-view consistency of the discovered parts. Furthermore, our method is capable of dealing with complex geometry and topology. Holes (cable in example 5) and geometry details are nicely revealed by our method. By utilizing our motion- based grouping approach, our method can accurately segment objects even if they are spatially separated- see the gloves (example 1) and 3-car (example 7) in Figure
5.4 APPLICATION: STRUCTURED SCENE MODELING BY ROBOTIC MANIPULATIION
Observation and interaction are crucial for human beings to learn from the real world. this section, we show that our method can identify objects and understand the functionality of their parts by observing physical interac- tion procedures. To demonstrate this, capture set of
Robotic Manipulation
robotic manipulation sequences with a similar monocular camera setting (Pumarola al. (2020). As shown in Figure 7 above, by observing the robot's work process, like picking up a toy or inserting peg, our method can accurately identify the manipulated object, as well as the
links and joints of the robot. Note that since the robotic
arms' trajectories are different in the two sequences, the joints discovered by motion are also different. The dis- covered 3D parts with their Lagrangian motion could pro- vide a strong prior for downstream functionality reason- ing and robotic reinforcement learning tasks.
Rendered Sequence
Predicted Groups
Real-world Scenes Editing
5.5 APPLICATION: SCENE EDITING
In addition to scene understanding, with the learned struc- tural representation of dynamic scenes, our method can also edit scenes and generate new renderings from the scene. Figure 7 below presents a few scene-editing ap- plications supported by our approach in HyperNeRF real- world sequence (Park et al. (2021b). Since our method conducts grouping in the 3D canonical space, the consis- tency can be maintained not only across multiple views but also across time steps. We show the removal or mod- ification of specific objects in these two real scenes and demonstrate the scalability of our method to real- call-wrldd applications.
Removal Medification
Figure 7: Applications. Our method directly apply to real-scene struc- tural modeling and editing.
6 CONCLUSION
In this paper, we present MovingParts, novel method for 3D dynamic scene reconstruction and part discovery. Inspired by fluid simulation, we observe the motion in the scene from both the Eulerian view and the Lagrangian view. In the particle-based Lagrangian view, we constrain the motion pattern of the particles to be a few rigid transformations, so that we successfully perform part discovery. To ensure fast convergence during training, we utilize hybrid feature volume and neural network representation, for both views which are efficiently supervised by a ccylle-cosiisenccy loss. What is more, the learned part representation could directly be applied to downstream tasks, e.g., object tracking, structured scene modeling, editing, etc.
Limitations. Motion modeling at a specific location can be considered as a sequence decoding task. In this paper, we explicitly store the motion features in low-dimensional vectors, which makes challenging to model motion on very long sequences. Although we can circumvent the issue by manually splitting long sequences into shorter ones, unified long sequence mccodingdeccoding scheme will still be a more elegant and efficient solution. We defer the exploration of this challenging setting to future work.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
Thanks for the real-scenario robotic data built by Litian Liang (UCSD) and Liuyu Bian (THU). This work was supported by the National Natural Science Foundation of China (NSFC) under Grants 62076230 and the Fundamental Research Funds for the Central Universities under Grant WK3490000008.
REFERENCES
Antonio Agudo and Francesc Moreno-Noguer, Robust spatio-temporal clustering and reconstruction of multiple deformable bodies. IEEE TPAMI, 41(4), 2019.
Christoph Bregler, Aaron Hertzmann, and Henning Biermann. Recovering non-rigid 3d shape from image streams. In CVPR, 2000.
Ang Cao and Justin Johnson. Hexplane: A fast representation for dynamic scenes, In CVPR, 2023,
Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In ECCV, 2022.
Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas B. McHugh, and Vincent Vanhoucke. Google scanned objects: A high-quality dataset of 3d scanned household items. In ICRA, 2022.
Yilun Du, Yinan Zhang, Hong-Xing Yu, Joshua B. Tenenbaum, and Jiajun Wu. Neural radiance flow for 4d view synthesis and video processing. In ICCV, 2021.
Bernhard Egger, William A. P. Smith, Ayush Tewari, Stefanie Wuhrer, Michael Zollhoefer, Thabo Beeler. Florian Bernard, Timo Bolkart, Adam Kortylewski, Sami Romdhani, Christian Theobalt, Volker Blanz, and Thomas Vetter. 3d morphable face models past, present, and future. In Siggraph (TOG), 2021.
Jiemin Fang, Taoran Yi, Xinggang Wang. Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Matthias Nießner, and Qi Tian. Fast dynamic radiance fields with time-aware neural voxels. In Siggruph Asia (TOG), 2022.
Ronald Fedkiw, Jos Stam, and Henrik Wann Jensen. Visual simulation of smoke. In Siggraph, 2001.
Sara Fridovich-Keil. Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance. In CVPR, 2023.
Chen Gao, Ayush Saraf, Johannes Kopf, and fia-Bin Huang. Dynamic view synthesis from dynamic monocular video. ICCV, 2021.
Paulo FU Gotardo and Aleix M Martinez. Non-rigid structure from motion with complementary
rank-3 spaces. In CVPR, 2011.
Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David J Fleet, Dan Gnanapragasam, Florian Golemo, Charles Herrmann, Thomas Kipf, Abhijit Kundu, Dmitry Lagun, Issam Laradji, Hsueh-Ti (Derek) Liu, Henning Meyer, Yishu Miao, Derek Nowrouzezahrai, Cengiz Oztireli, Etienne Pot, Noha Radwan, Daniel Rebain, Sara Sabour, Mehdi S. M. Sajjadi, Matan Sela, Vincent Sitzmann, Austin Stone, Deqing Sun, Suhani Vora, Ziyu Wang, Tianhao Wu, Kwang Moo Yi, Fangcheng Zhong, and Andrea Tagliasacchi. Kubric: a scalable dataset generator. In CVPR, 2022.
Xiang Guo, Guanying Chen, Yuchao Dai, Xiaoqing Ye, Jiadai Sun, Xiao Tan, and Errui Ding. Neural deformable voxel grid for fast optimization dynamic view synthesis. ACCV, 2022.
Mare Habermann, Weipeng Xu, Michael Zollhoefer, Gerard Pons-Moll, and Christian Theobalt. Livecap: Real-time human performance capture from monocular video. In ACM TOG, 2019.
Zhang Jiakai, Liu Xinhang, Ye Xinyi, Zhao Fuqiang. Zhang Yanshun, Wu Minye, Zhang Yingliang, Xu Lan, and Yu Jingyi. Editable free-viewpoint video using a layered neural representation. In Siggraph (TOG), 2021.
10

---

Published as a conference paper at ICLR 2024
Yuki Kawana, YUSUKE Mukuta, and Tatsuya Harada. Unsupervised pose-aware part decomposi- tion for 3d articulated objects. In ECCV, 2022.
Margret Keuper, Bjoern Andres, and Thomas Brox. Motion trajectory segmentation via minimum cost multicuts. In ICCV, 2015.
Muhammed Kocabas, Nikos Athanasiou, and Michael J. Black. Vibe: Video inference for human body pose and shape estimation. In CVPR, 2020.
Chen Kong and Simon Lucey. Deep non-rigid structure from motion. In ICCV, 2019.
Tianye Li, Mira Slavcheva, Michael Zollhöfer, Simon Green, Christoph Lassner, Changil Kim, Tan- ner Schmidt, Steven Lovegrove, Michael Goesele, Richard Newcombe, and Zhaoyang Lv. Neural 3d video synthesis from multi-view video. In CVPR, 2022.
Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. In CVPR, 2021.
Jia-Wei Liu, Yan-Pei Cao. Weijia Mao, Wenqiao Zhang, David Junhao Zhang, Jussi Keppo, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Devrf: Fast deformable voxel radiance fields for dynamic scenes. In NeurlPS, 2022.
Miles Macklin, Matthias Müller, Nuttapong Chentanez, and Tae- Yong Kim. Unified particle physics for real-time applications. ACM TOG, 2014.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
Tongzhou Mu, Zhan Ling, Fanbo Xiang, Derek Cathera Yang. Xuanlin Li, Stone Tao, Zhiao Huang, Zhiwei Jia, and Hao Su. Maniskill: Generalizable manipulation skill benchmark with large-scale demonstrations. In NeurIPS, 2021.
Atsuhiro Noguchi, Xiao Sun, Stephen Lin, and Tatsuya Harada. Neural articulated radiance field. In ICCV, 2021.
Atsuhiro Noguchi, Umar Iqbal, Jonathan Tremblay, Tatsuya Harada, and Orazio Gallo. Watch it move: Unsupervised discovery of 3D joints for re-posing of articulated objects. In CVPR, 2022.
Martin Ralf Oswald, Jan Stühmer, and Daniel Cremers. Generalized connectivity constraints for spatio-temporal 3d reconstruction. In ECCV, 2014.
Anestis Papazoglou and Vittorio Ferrari. Fast object segmentation in unconstrained video. In ICCV, 2013.
Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Steven M. Scitz, and Ricardo
Nerfies: Deformable neural radiance fields. ICCV, 2021a.
Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T. Barron, Sofien Bouaziz, Dan B Gold- man, Ricardo Martin-Brualla, and Steven M. Seitz. Hypernerf: nigher-dimensiooal representa- tion for topologically varying neural radiance fields. ACM TOG, 40(6), 2021b.
Sida Peng, Chen Geng, Yuanqing Zhang, Yinghao Xu, Qianqian Wang, Qing Shuai, Xiaowei Zhou, and Hujun Bao. Implicit neural representations with structured latent codes for human body modeling. IEEE TPAMI, 2023.
Erik Learned-Mlllrr Pia Bideau. It's moving! a probabilistic model for causal motion segmentation in moving camera videos. In ECCV, 2016.
Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francese Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In CVPR, 2020.
Zhang Richard, Isola Phillip, Alexei Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In CVPR, 2018.
11

---

Published as a conference paper at ICLR 2024
Yahao Shi, Xinyu Cao, and Bin Zhou. Self-supervised learning of part mobility from point cloud sequence. In Computer Graphics forum, 2021.
Vikramjit Sidhu, Edgar Tretschk, Vladislav Golyanik, Antonio Agudo, and Christian Theobalt. Neu- ral dense non-rigid structure from motion with latent space constraints. In ECCV, 2020.
Cheng Sun, Min Sun, and Hwann-Tzong Chen. Direct voxel grid optimization: Super-fast conver- gence for radiance fields reconstruction. In CVPR, 2022.
Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, and Christian Theobalt. Non-rigid neural radiance fields: Reconstruction and novel view synthesis of dynamic scene from monocular video. In ICCV, 2021.
Vadim Tschernezki, Diane Larlus, and Andrea Vedaldi. NeuralDiff: Segmenting 3D objects that move in egocentric videos. In 3DV, 2021.
Tony Tung, Shohei Nobuhara, and Takashi Matsuyama. Complete multi-view reconstruction of dynamic scenes from probabilistic fusion of narrow and wide baseline stereo. ICCV, 2009.
Fangyin Wei, Rohan Chabra, Lingni Ma, Christoph Lassner, Michael Zollhoefer, Szymon Rusinkiewiez, Chris Sweeney, Richard Newcombe, and Mira Slaveheva. Self-supervised neu- ral articulated shape and appearance models. In CVPR, 2022.
Chung-Yi Yi Weng, Brian Curless, Pratul P. Srinivasan, Jonathan T. Barron, and Ira Kemelmacher- Shlizerman. HumanNeRF: Free-viewpoint rendering of moving people from monocular video. In CVPR, 2022.
Tianhao Walter Wu, Fangcheng Zhong, Andrea Tagliasacchi, Forrester Cole, and Cengiz Oztireli. D²2neRF: Self-supervised decoupling of dynamic and static objects from a monocular video. In NeurlPS, 2022.
Wenqi Xian, Jia-Bin Huang, Johannes Kopf, and Changil Kim. Space-time neural irradiance fields for free-viewpoint video. In CVPR, 2021.
Christopher Xie, Yu Xiang, Zaid Harchaoui, and Dieter Fox. Object discovery in videos as fore- ground motion clustering. CVPR, 2019.
Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong
Wang. Groupvit: Semantic segmentation emerges from text supervision. In CVPR, 2022.
Charig Yang, Hala Lamdouar, Erika Lu, Andrew Zisserman, and Weidi Xie. Self-supervised video object segmentation by motion grouping. In ICCV, 2021.
Alex Yu, Sara Fridovich-Keil, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In CVPR, 2022.
Wentao Yuan, Zhaoyang Lx, Tanner Schmidt, and Steven Lovegrove. Star: Self-supervised tracking and reconstruction of rigid objects in motion with neural rendering. In CVPR, 2021.
Li Zhang, Brian Curless, and Steven M. Seitz. Spacetime stereo: Shape recovery for dynamic scenes. In CVPR, 2003.
Wang Zhou, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility structural similarity. IEEE TIP. 2004.
Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. On the continuity of rotation representations in neural networks. In CVPR, 2019.
12

---

Published as a conference paper at ICLR 2024
Hook
Jumping Jacks
T-Rex
Renderod
Image
Depth
Groups
Merged Groups
Figure 8: More visualization results of reconstruction and part discovery on D-NeRF synthetic
dataset. By analyzing the motion patterns, our method could discover parts that are consistent with common sense, such as human arms and legs.
Table 2:
Ablation study about components and losses, including the Lagrangian module, cycle
consistency loss, TV loss, per-point loss, and cross-entropy loss.
Lyyele feature-based
LTv
enttopp
Eper-pt
PSNR 1
SSIM f
LPIPS
AI (Ours)
32.183
0.971
0.036
A2
X
32.333
0.972
0.036
A3
displacement-based
/
V
31.968
0.971
0.037
A4
fcature-based
30.017
0.957
0.064
A5
feature-based
V
x
31.988
0.971
0.037
A6
feature-based
32.082
0.970
0.038
A
APPENDIX
A.1
ABLATION STUDY
In this section, we conduct ablation experiments the Lagrangian module and the loss functions to showcase their effectiveness. These ablations were conducted on the D-NeRF synthetic dataset and we report their averaged metric values (PSNR, SSIM and I.PIS) with their corresponding model settings (A1 7) in Table 2 Moreover, we show the rendered results of these experiments on the Bouncing Balls scene in Fig. to illustrate the influence of individual modules on rendering and part discovery. We present our full model in Experiment Al.
Lagrangian module. Our Lagrangian module is mainly designed for automatic part discovery, which is the main focus of the this work. Without this module, the model cannot achieve part discovery at all while dynamic reconstruction and novel view rendering can still be done. We eval- uate how the Lagrangian module is affecting the final rendering with the ablated model setting A2, where we set the weight of the cycle consistency loss to 0, which essentially disables the Lagrangian module. As shown in Table the absence of rigid part motion constraints in the Lagrangian module leads to only a slightly higher PSNR (less than 0.2db difference). In general, our Lagrangian module enables automatic part discovery while retaining high rendering quality.
Cycle Consistency loss. To validate the effectiveness of the motion feature- based cycle consistency loss, we compare with the displacement-baane based cycle loss in A3. We enforce that the displacement modeled by the Eulerian and Lagrangian modules remains consistent, akin to the Deformation Cycle Consistency in Liu al. (2022):
deyele-disp
1x- ( tg) + tt.
(14)
|Pansl
xEPaay
As demonstrated in Table 2 there is decline in PSNR for displacement-basedd cycle consistency (A3) when compared to the feature-level consistency used in our paper (A1). More importantly, the discovery of parts is contingent upon grouping Lagrangian motion features. The inherent ambiguity
13

---

Published as a conference paper at ICLR 2024
Al: Feature-hased Cycle Loss
A2: Languagian Module
A3: Displacmeet-baseed Cycle Loss
A4:
A5:
A6; Without Per-point Loss
Entropy Loss
(Our Full Model)
Figure 9: The rendering results of ablations on Bouncing Balls scene.
between motion features and displacement, where multiple motion features may correspond to the same displacement, poses a challenge in attaining precise part discovery results. As result, the A3 model often fails to distinguish different parts in the scene (see A. in Figure
Total variation loss. The total variation loss imposes constraints on the feature similarity among neighboring grids in the motion volume, effectively ensuring that adjacent particles in space exhibit similar motion patterns. Experiment A4 demonstrates the critical role of this regularization in motion modeling. In its absence, there notable decline in image rendering quality. Furthermore, due to the unrestrained movement of near-neighbor particles, the discovered parts lack the characteristic localized nature in 3D space, as illustrated in A4 of Figure 9
Additional losses. The removal of either the cross- cross-entropy loss (A5) or the per-point RGB loss (A6) lead to decline rendering quality. Notably, the cross-entropy loss plays a crucial role in regulat- ing foreground- -wwkuddd-bbkoounddd probability. and its omission may lead to ghosting in the background, as evident in A6 of Figure is noteworthy that even the absence these additional regularization losses, our method can still produce reasonable part discovery results.
A.2 MORE RECONSTRUCTION AND GROUP RESULTS
In this subsection, we report the per-scene results of the D-NeRF synthetic dataset in Table 3 We also show more dynamic scene reconstruction and part discovery results in Figure g It can be that our method can achieve high-quality dynamic reconstruction with different motion patterns and also shows the ability that reasonably segment the moving regions and obtain meaningful parts. like the legs and arms. Note that our approach does not explicitly introduce any category and geometric priors. Therefore, it necessary to separate parts that are perceptuallysemmattiaall distinguish- able but have no relative motion with other parts in the video capture, like the head and body.
A.3 VISUALIZATION OF GROUP MERGING
Due to the excessive number of groups and non- linear neural networks, the same rigid transfor- mation trajectory could be easily represented by different high-level motion features. Therefore, we introduce group merging module to reduce the number of the group plausible level. In this subsection, we visualize two examples of all steps of group merging, as shown Figure 10 At each merge step, the pair of groups with the current most similar motion patterns are ag- gregated; the entire process actually builds bi- nary tree of the original groups The whole pro- cess stops when all groups combined into single one, i.c. the step with a single color. Fi- nally, the most reasonable step is selected by
Two examples of the group merging process. The groups with similar motion patterns are gradually merged, and then the most reason- able step could be selected.
Figure 10:
14

---

Published as a conference paper at ICLR 2024
Table 3: Per-scene results of D-NeRF synthetic dataset.
Hell Warrior
Mutant SSIM
Hook SSIM
Bouncing Balls
Method
PSNR
SSIM
I.PIPS
PSNR
LPIPS
PSNR
I.PIPS
PSNR
SSIM
I.PIPS
T-NeRF
23.19
0.93
0.08
30.56
0.96
0.04
27.21
0.94
0.06
37.81
0.98
0.12
D-NeRF
25.02
0.95
0.06
31.29
0.97
0.02
29.25
0.96
0.11
38.93
0.98
0.10
TiNeu Vox
28.17
0.97
0.07
33.61
0.98
0.03
31.45
0.97
0.05
40.73
0.99
0.04
NDVG
25.53
0.95
0.07
35,53
0.99
0.01
29.80
0.96
0.04
34.58
0.97
0.11
K- K-Planes
25.70
0.95
33.79
0.98
28.50
0.95
41.22
0.99
HexPlane
24.24
0.94
0.07
33.79
0.98
0.03
28.71
0.96
0.05
39.69
0.99
0.03
WIM
12.35
0.81
0.21
16.20
0.85
0.16
14.16
0.82
0.19
15.82
0.84
0.29
Ours
28.66
0.97
0.04
34.42
0.98
0.02
31.39
0.97
0.04
38, 38,99
0.99
0.04
Lego
T-Rex
Stand Up
Jumping Jacks
Method
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
T-NeRF
23.82
0.90
0.15
30.19
0.96
0.13
31.24
0.97
0.02
32.01
0.97
0.03
D-NeRF
21.64
0.83
0.16
31.75
0.97
0.03
32.79
0.98
0.02
32.80
0.03
TiNeu Vox
25.02
0.92
0.07
32.70
0,98
0.03
35,43
0.99
0.02
34.23
0.98
0.03
NDVG
25.23
0.93
0.05
30.15
0.97
0.05
34.05
0.98
0.02
29.45
0.96
0.08
K-Planes
25.48
0.95
31.79
0,98
33.72
0.98
32.64
0.98
HexPlane
25.22
0.94
0.04
30.67
0.98
0.03
34,36
0.98
0.02
31.65
0.97
0.04
WIM
13.95
0.72
0.28
19.05
0.87
0.14
16.26
0.89
0.12
17.95
0.87
0.16
Ours
25.08
0.92
0.07
32.24
0.98
0.03
34,46
0.98
0.02
32.22
0.98
0.03
the merging threshold discussed in our paper, shown as the rectangles in the figure. Note that the same color in different steps does not denote the same group.
A.4 MOTION GROUPING EVALUATION DETAILS.
We utilized Kubric (Greff et al. (2022), data generation pipeline for multi-object videos with annotation, to create our evaluation dataset. The MOVi-C pipeline from Kubric was employed to generate scenes, utilizing realistic, textured objects from the GSO dataset (Downs al. (2022). ). To create a diverse set of scenes, a random HDRI was used to generate the background. The number of objects each scene varied randomly between one to five, and object shadows were not considered due to their highly non-rigid nature. Objects were placed randomly in space with randomized initial velocity. To capture each scene, the camera was positioned above the objects with camera movement setting similar to PoPumrrola et al. (2020). Finally, each scene was simulated and rendered into a 120-frame image sequence, with ground truth instance masks. To evaluate the mean Intersee- tion over Union (mIOU) between our grouping result and the ground truth mask, it is necessary to establish correspondence between them. To achieve this, we utilized the first 10 frames in the se- quence determine the correspondence, which then extended to the whole sequence. For each group, we tallied the number of pixels belonging to each label in the first ten frames and identified the ground truth label with the highest number of pixels to associate it with the group. This operation enabled us to map each group to a label and convert the group map into a label map. Subsequently, we calculated the fer-frame IOU between the converted label map and the ground truth mask. To obtain the final mIOU, we averaged the IOUs of all labels over the entire image sequence.
A.5 TRAINING DETAILS AND HYPEERPPARAMEE PARAMETER SETTINGS
We use 50x 50 50 voxels to construct the Eulerian volume V and the Lagrangian volume Vz with a feature dimension of 20. The canonical volume V. is constructed with a 160 x 160 x 160 voxel. Following Fang et al. (2022). the feature dimension of V is set as 6. To alleviate the optimization difficulty and speed up training, we set the initial resolution of the canonical volume to 40 and upsample it at the 4k, 6k, and 8k iterations. For the MI.s in our framework, we set the channel number of all hidden layers to 128. We use two-layer MLPs for the motion extractors Eg and &g. The parameters of the rigid transformation decoders, Dg and Dr, are shared. This sharing ensures that the decoded motion parameters exhibit consistency when the motion features are consistent. Two separated linear layers are used for the decoders to predict the 6D rotation and 3D translation with biases as (1, O, O, 0, 1, 0) and (0, 0, 0). respectively so that the initial deformation is an identity. For motion grouping, we set the slot number 12 and initialize the slots from standard normal distribution. We use the Adam optimizer for a total of 20k iterations, by sampling 4096 rays from a randomly sampled image in each iteration. To reduce the learning difficulty, we add images into the training set progressively at the early training stage. We set the learning rate as 0.08 for the Eulerian and Lagrangian volumes, 0.01 for the canonical volume, 6 10 for È and D. 8 x 10 for other networks. For the overall loss function, we described it as:
= Lphcce + Weyete + Wper.pt Eperpp + Wentropy Centropy + Wwy Lay-
(15)
15

---

Published as a conference paper at ICLR 2024
We use Eperpp and Lentopy to supervise the color of sampled points and regularize the background probability, respectively. We set Wper pt and Westropy to 0.01 and 0.001. To encourage reciprocity of motion between the Eulerian and Lagrangian views, weight parameter Weycle of 0.1. Additionally, we apply the total variation loss to smooth the features of the motion volumes VE and V!:
the cycle consistency loss Leyele with
Σ (VAAVV N
1
+ WE A
(16)
Where N denotes the number of parameters each motion volume, A represents the square differ- ence between neighboring values in the motion volume, we denotes the additional weight between VE and Vz. We set the W/y = 0.01 and we = 1 for the D-NeRF synthetic dataset. For motion grouping evaluation, we decrease these two weights due the more complex object geometry and motion patterns, we set the Why 0.001 and wpp = 0.1.
A.6 MORE RESULTS OF APPLICATIONS ON SYNTHETIC DATA.
Except for the real data application, we also con- ducted structural scene modeling and editing on robotic synthetic data. We utilized subset of the robotic manipulation environments in ManiSkill (Mu et al. (202T) to simulate robot-object interac- tions with similar monocular setting (Pumarola ict al. (2020)). As shown Figure II We conducted experiments using two different setups: articulated object manipulation and bidd manipulation. In the first setup, the robot performs operations on particular movable part of an articulated object, as shown in Figure TT above. In the second setup, the robot is tasked with grasping a specific object and moving it target position, as illustrated in Fig- ure I below. In both setups, through observation, our method can accurately identify the manipulated object or object part, such the drawer (orange) and the cabinet door (yellow), as well as the links and joints of the robot. What is more, benefit from the learned structural scene representation, we could apply direct high-level scene editing. In Figure II] we show the editing operations like duplication, re- moval, scaling of specific object in the scene, and the control of an articulated robot arm. Note that these operations all performed in 3D space and rendered into 2D images for visualization.
Structural scene modeling robotic munipulation
Scene oditing with the robotic environment
Rescaling
Re- Re-posing
Duplication
Removal
Figure 11: Structural scene modeling and editing in simulated robotic manipulation en- vironments.
A.7 IMPLEMENTATION DETAILS OF SCENE EDITING
Based on our part-level representation, we could di- rectly edit the 3D dynamic scenes. In this paper, we present four scene-editing applications: Removal, Duplication, Rescaling, and Re-posing. Here, we will provide more details about how to implement these applications.
Midada
Rendey
Deletion
Maakk
Berigread
As shown in Figure 12 these application implemen- tations could be divided into 2 ways. For Removal, after ray marching, the sampled points are fed into the Eulerian module be transformed into canon- ical space. Then these canonical coordinates are grouped by our motion grouping network. To delete
Andky
Mo-
Duplication, Rescaling, Re-posing Figure 12: Illustraiion of the application im- plementation.
16

---

Published as a conference paper at ICLR 2024
the specified group, simply assign the density of the points in this group from the canonical space. After rendering, the specific group will be deleted from the image.
remove it
Duplication, rescaling, and re-posing could be implemented in the same way. We do this by pro- cessing the group to be edited and the background (all the other groups) respectively. Specifically, the sampled points by ray marching are fed into the background branch and the edit branch. In the background branch, we leave the background group fixed and delete the group to be edited, which is the as the Removal application. In the edit branch, group, such as rotation, translation, scaling, etc. We achieve this by first performing manually specified transformation on the sampled points. Then these transformed points fed into the Eule- rian module and the motion grouping network obtains the canonical coordinates and corresponding group index. After that, we set the density the points in the background groups to 0 so that the transformation only acts on the target group. Finally, we integrate the corresponding points of the two branches render them into the 2D image, the integration formula is follows:
perform transformations for the target
Thg Oedit
(17)
c =
(bg Cbg + Tedit -eesst)
Note that by introducing additional edit branches, we could make different edits to different groups simultaneously.
17


Reviews:
[{"review_id": "d8059fUNte", "reviewer": "Reviewer_5UoR", "summary": "This paper presents an approach for dynamic scene reconstruction that allows the discovery of parts as a result of the factorization of the scene motion. The approach is based on monocular NeRF. A part corresponds to particles that share a common motion pattern. Scene motion is parameterized with both an Eulerian view and a Lagrangian view. The Lagrangian view allows tracking the particles on objects. Since the particles on a rigid part share a common rigid transformation pattern, a motion grouping module is used to detect the parts. The method is based on three modules, namely: the Eulerian module, the Lagrangian module and the canonical module. The Eulerian\nmodule and the Lagrangian module observe the motion of specific spatial locations and specific particles, respectively. The canonical module serves to reconstruct the geometry and appearance for volume rendering. The the reconstruction and part discovery performance  of the method is experimentally evaluated on a synthetic dataset provided by D-NeRF. The metrics used for evaluation were PSNR and SSIM. The motion grouping is evaluated on a synthetic dataset created using Kubric toolkit. In both evaluations the method obtained performances better or at the level of the state-of-the-art.", "questions": "--Can you use other motion models, other than rigid ones?\n--What is the relationship between the cycle consistency between the Eulerian and Lagrangian modules and the material derivative?\n--How are the coordinates in the Eulerian and Lagrangian modules related?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The strength of the paper stems from being a method based on monocular NeRF that besides allowing to perform dynamic scene reconstruction, allows also the discovery of rigid parts. Another strength of the approach is that it used hybrid feature volume and neural network representation, which allows  fast convergence during training.", "weaknesses": "The grouping of the parts is based on predicted rigid motion. Rigid motion can be a significant constraint in many applications. The approach is inspired by fluid simulation, with the scene motion being observed from both the Eulerian view and Lagrangian view. The Eulerian and Lagrangian descriptions of a flow field are related by the material derivative. It is unclear whether the method ensures that such relationship is verified or met by the proposed approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aEDZ9WsNAo", "reviewer": "Reviewer_WRpS", "summary": "This paper proposes a method for part-aware dynamic NeRF. While dynamic NeRF (D-NeRF) encodes the temporal information as well as the spatial radiance fields, it has not been easy to extract the object part information. To address this problem, this paper uses an idea similar to traditional motion segmentation, which combines the flow between adjacent frames (i.e., Eulerian flow in terms of particle flow simulation) as well as the clusters of motion trajectories (i.e., Lagrangian flow). Experiments show that the proposed method achieves near-SOTA accuracies for novel view synthesis with part-aware representation.", "questions": "Q. Related to the above, since the proposed method focuses on monocular input, readers may wonder if motion segmentation is applicable to their input sequence. In this regard, what happens if they apply motion segmentation to input sequences and then reconstruct *part-wise* NeRF?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "+ The proposed method correctly brings the idea of using motion trajectories to NeRF-based representations. In particular, the use of a cycle consistency loss that improves the agreement between Eulerian and Lagrangian motion shows a nice coupling of traditional ideas and modern neural networks.\n\n+ The proposed method focuses on scenarios using a monocular moving camera, which makes the proposed method practical. In fact, the paper shows two practical examples of applications to robotics and scene editing.", "weaknesses": "- Using long-term motion trajectories is itself a traditional idea in computer vision, especially for motion segmentation (e.g. [a]). While I think that using Eulerian vs. Lagrangian views is a nice introduction to temporal motion representations, it would be nicer if they also briefly discussed the relationship of their idea to these traditional attempts. In my opinion, this would not sacrifice their technical contribution, but rather strengthen the theoretical and academic value of their approach.\n\n[a] Keuper, Margret, Bjoern Andres, and Thomas Brox. \"Motion trajectory segmentation via minimum cost multicuts.\" Proceedings of the IEEE International Conference on Computer Vision. 2015.\n\n- Since their technical contribution may be to combine short- and long-term motion trajectories, readers may want to see the ablation studies comparing with and without these modules.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jKEjnykIrS", "reviewer": "Reviewer_NTzN", "summary": "In this paper is proposed a NeRF-based approach for simultaneous non-rigid scene reconstruction and rigid part grouping from monocular pictures. To this end, the method combines location-based Eulerian and particle-based Lagrangian interpretations, being a slot attention-based motion grouping network as part of the Lagrangian module. Both interpretations are a hybrid representation of volumetric features, which are supervised by a cycle-consistency loss. Every motion group is expressed by a rigid transformation per temporal frame. Experimental results are provided on synthetic and real datasets, obtaining a good trade-off between accuracy and computational cost in comparison with state-of-the-art techniques.", "questions": "The motivation of the paper is good and the contributions are clearly stated. The technical section is well written and is clear enough. Most of the relevant details are provided as well as the explanations and discussions of every module the authors use. \n\nThe combination of Eulerian and Lagrangian formulations to handle the time-varying problem is a very interesting contribution. The authors exploit both points of view in a unified formulation. \n\nIn comparison with other NeRF algorithms to handle non-rigid objects, the algorithm in this paper can also infer a segmentation, exploiting the local rigidity of the clusters. This is a good point, but also shows us the type of deformation to be considered is a bit limited, as just piecewise rigid deformations are possible. In any case, the joint estimation is a good contribution. Similar to NeRF approaches, the camera poses are assumed in advance, representing, probably, a strong prior in this context. \n\nThe proposed method provides a good trade-off between accuracy and computational cost in comparison with other competing recent works. Moreover, the method can infer the part-based grouping automatically, as Noguchi et al. CVPR 2022 did. The results in terms of computational cost are really relevant. \n\nLack of realistic experimental evaluation. Unfortunately, the method is mainly tested on the D-NeRF synthetic dataset. I would like to see experimental results on real and challenging sequences. In addition to that, in spite of claiming non-rigid motions, the level of deformation on that dataset is very limited, and therefore I consider it is not the best one for evaluating non-rigid scenarios. In other words, the type of motion is piecewise rigid (as it was considered previously), that essentially is not the same as non-rigid. In fact, this can be observed in the estimation of groups. As it can be seen, the number of groups is very reduced and the global behavior in terms of deformations of those groups is quite simple. Again, the use of the dataset ManiSkill is not the best way to validate the strengths of the method due to the simplicity of the motion. In contrast, the authors could consider some real videos with dynamic scenarios to finish the full analysis and claim validation. I like real experiments in the context of scene editing. \n\nThe video is complete and definitely can help the reader.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper addresses an important problem in computer vision and learning. The proposed solution is a good combination of multidisciplinary areas. \n\n- In general terms, the paper is well written and clear enough.", "weaknesses": "- The experimental evaluation is provided on simple synthetic datasets where the amount of deformation is a bit limited. Considering the paper formulation, I believe the method could handle strong deformations with no problem, but that is never validated. \n\n- No full evaluation on real monocular image sequences.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9TrLjlAb3E", "reviewer": "Reviewer_qQod", "summary": "This submission takes monocular dynamic NeRF one steps further and tries to discover rigid objects parts. It uses D-NeRF-style monocular RGB input and aims for general articulated scenes. To this end, it uses forward and backward deformation modeling and a cycle consistency loss between them on the feature level (doing this on the features instead of offsets is novel). Then, at test time, the forward motions are merged into an adaptively determined number of rigid groups. Experiments evaluate the reconstruction and part segmentation quality. The results are almost exclusively on quasi-multi-view/D-NeRF-style synthetic scenes.", "questions": "My questions are intertwined with the Weaknesses, please see above. \n\nOverall, I unfortunately lean towards reject. I only have minor concerns about the method and related work, which can be easily addressed. However, the experimental evaluation is too lacking in multiple important regards. Still, if these concerns are addressed well in a rebuttal, I would most likely increase my score to accept.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "Extending dynamic NeRF towards part discovery is interesting, as also evidenced by a few related works in this direction. \n\nThe proposed method is quite straightforward and thus elegant. It does not appear to have overly involved components.\n\nUsing cycle consistency on motion features rathter than the final offsets is interesting and novel.\n\nThe reconstruction quality on the D-NeRF dataset is high (other datasets are unclear). The same goes for the part discovery, which gives very accurate looking qualitative results.\n\nA few simple downstream applications like basic editing or tracking are presented.", "weaknesses": "Related Work and Method:\n\n(A) I am not convinced that \"Eulerian\" and \"Lagrangian\" are the right terms. Standard deformable NeRFs like D-NeRF or Nerfies use backward deformation modeling. The paper proposes forward modeling instead. These terms should at the very least be mentioned explicitly somewhere. Both forward and backward modeling focus on particles and are direct functional inverses of each other. They thus resemble the Lagrangian perspective. As for example mentioned in Tretschk et al. State of the Art in Dense Monocular Non-Rigid 3D Reconstruction, I think it's more accurate to think of methods like NSFF that do not have a proper canonical model and rather directly regress the radiance field at each coordinate point in time and space as Eulerian. Such methods do not have a notion of particle, while both forward and backward deformation methods do. Or, to be more precise, they only use temporal consistency losses with auxiliary scene flow between *neighboring* frames and thus only have a notion of *instantaneous* flow to *temporally neighboring* frames at coordinate points in world space, which is exactly what Eulerian flow is. All of this is of course not ultimately relevant for the point of the paper in any way, only the presentation/writing, and I'm willing to hear why the notions proposed in the paper are more appropriate.\n\n(B) The submission argues from the intractability of tracking a particle with backward deformation modeling. But for example Cai et al. Neural Surface Reconstruction of Dynamic Scenes with Monocular RGB-D Camera use an invertible deformation model that would allow to long-term track a particle in a straightforward manner. They lack expressivity and are computationally expensive but that's a more involved argument than the one in the submission.\n\n(C) Liu et al. also propose cycle consistency of forward and backward modeling for dynamic NeRF in their NeurIPS 2022 paper DeVRF. It does this on the motion and not the feature level though. I think this should be discussed in the paper.\n\n(D) It might be good to discuss how the proposed method relates to articulated NeRF methods like Wei et al. Self-supervised Neural Articulated Shape and Appearance Models. \n\n(E) While not necessary, please consider citing Nerfies when introducing the idea of using an SE(3) parametrization, since they were the first to do that for dynamic NeRFs.\n\n(F) Maybe mention that category-specific methods, e.g. those focusing on the human body or face, do quite easily allow for part understanding and editing/re-posing, etc. and that the proposed method focuses on general dynamic scenes.\n\n(G) The lower half of page 5 talks about averaging all the features in a group. I assume all these features come from all the points in the batch? But what is in the batch? An entire image? I.e. how is good, reproducible coverage ensured such that averaging in this manner always gives pretty much the same final feature for a certain group across different views?\n\n(H) What loss function provides a gradient for D_L, the Lagrangian decoder? All losses except for the cycle consistency loss only go into the canonical model and the Eulerian part. Where is the total variation loss applied exactly?\n\n(I) I don't understand what the group merging module is used for. My only guess is this: During training, the initial, too-large number of groups is fixed/always used. Then, after training is finished, the group merging module is used to determine the final number of groups? So the APE cost is never used for backpropagation?\n\n(J) The Lagrangian/forward module assumes articulated rather than fully non-rigid motion due to its per-group averaging operation. Please state this explicitly.\n\nExperiments:\n\n(K) I am concerned that video results are only shown for the synthetic D-NeRF dataset, which is effectively a multi-view dataset (as discussed in Gao et al. Monocular Dynamic View Synthesis: A Reality Check). I'd much prefer to see the video results of the real-world monocular scenes used in the paper. And the datasets from Figures 6 and 7 (although these also use the quasi-multi-view setting). Otherwise, I will assume that reconstruction only works for quasi-multi-view synthetic scenes and not for input that is truly monocular or real.\n\n(L) Please add video results for part discovery on multiple scenes of the HyperNeRF dataset. Currently, only a handful of still images are shown. Otherwise, I will assume that these are severely cherry-picked and part discovery does not work on real-world scenes.\n\n(M) The same applies for tracking. Using a well-textured canonical space like here https://github.com/pablopalafox/npms for example would help in visualizing the tracking/correspondence quality.\n\n(N) Please add LPIPS scores everywhere. It's standard for NeRF papers (see e.g. the original NeRF paper) and PSNR and SSIM are quite similar, while LPIPS is much more perceptual.\n\n(O) There appear to be no ablations. For example, applying the cycle consistency loss on features vs. positions. Or using SE(3) vs. offsets. And loss ablations, e.g.: Why is the per-point color loss useful? What happens without the total variation loss? Also, an ablation of whether the forward model helps *in reconstructing* would be good. Is this additional component useful? If it isn't, then the proposed part discovery portion of the method (i.e. the group merging module) might be added flexibly as a post-processing step to existing backward dynamic NeRFs to discover parts, which would be interesting to know.\n\n(P) Can Watch-It-Move, the most relevant related work, not be applied to the D-NeRF dataset? I'm curious about the parts it would discover.\n\n(Q) I assume that part segmentation is evaluated in 2D image space rather than in 3D due issues with the unobservable insides of objects?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)