DynaVol: Unsupervised Learning for Dynamic Scenes through Object-Centric Voxelization
Authors: Yanpeng Zhao
Siyu Gao
Yunbo Wang
Xiaokang Yang
Venue: ICLR 2024 poster (2024)
Keywords: Unsupervised Learning
Object-Centric Representation Learning
Dynamic Scene Decomposition
Inverse Rendering
Primary area: generative models

Abstract: Unsupervised learning of object-centric representations in dynamic visual scenes is challenging. Unlike most previous approaches that learn to decompose 2D images, we present DynaVol, a 3D scene generative model that unifies geometric structures and object-centric learning in a differentiable volume rendering framework. The key idea is to perform object-centric voxelization to capture the 3D nature of the scene, which infers the probability distribution over objects at individual spatial locations. These voxel features evolve over time through a canonical-space deformation function, forming the basis for global representation learning via slot attention. The voxel features and global features are complementary and are both leveraged by a compositional NeRF decoder for volume rendering. DynaVol remarkably outperforms existing approaches for unsupervised dynamic scene decomposition. Once trained, the explicitly meaningful voxel features enable additional capabilities that 2D scene decomposition methods cannot achieve: it is possible to freely edit the geometric shapes or manipulate the motion trajectories of the objects.

Published as a conference paper at ICLR 2024
DYNAVOL: UNSUPERVISED LEARNING FOR DYNAMIC SCENES THROUGH OBJECT-CENTRIC VOXELIZATION
Yanpeng Zhad Siyu Gao" Yunbo Wang Xiaokang Yang
MoE Key Lab of Artificial Intelligence, Al Institute. Shanghai Jiao Tong University (zhao-yan-peng, si siyu. - g yunbow, xkyang) @ jtu edu.cn https://sites..ont e. www.wwwwwwyyn////acccceeeoo
ABSTRACT
Unsupervised learning of object-centric representations in dynamic visual scenes is challenging. Unlike most previous approaches that learn to decompose 2D images, we present DynaVol, 3D scene generative model that unifies geometric structures and object-centric learning in a differentiable volume rendering framework. The key idea is to perform object-centric voxelization to capture the 3D nature of the scene, which infers the probability distribution over objects at individual spatial locations. These voxel features evolve through a canonical-space deformation function, forming the basis for global representation learning via slot attention. The voxel and global features are complementary and leveraged by compositional NeRF decoder for volume rendering. Dyna Vol remarkably outperforms existing approaches for unsupervised dynamic scene decomposition. Once trained, the explicitly meaningful voxel features enable additional capabilities that 2D scene decomposition methods cannot achieve: it is possible to freely edit the geometric shapes or manipulate the motion trajectories of the objects.
INTRODUCTION
Unsupervised learning of the physical world is of great importance but challenging due to the intricate entanglement between the spatial and temporal information (Wu et al. 2015; Santoro et al. 2017] Greff et al. 2020). Existing approaches primarily leverage the consistency of the dynamic information
across consecutive video frames but tend ignore the 3D nature, resulting in a multi-view mismatch of 2D object segmentation (Kabra et al.) 2021 Elsayed et al., 2022; Singh et al. 2002].. In this paper, we explore novel research problem of unsupervised 3D dynamic scene decomposition. Different from the previous 2D counterparts, our method naturally ensures 3D-consistent scene understanding and can motivate downstream tasks such as scene editing. We believe that an effective object-centric representation learning method should satisfy two conditions: First, it should capture the time-varying local structures of the visual scene in a 3D-consistent way, which requires the ability to decouple the underlying dynamics of each object from visual appearance; Second, it should obtain global understanding of each object, which is crucial for downstream tasks such as relational reasoning.
Accordingly, we propose to learn two sets of object-centric representations: one that represents the local spatial structures using time-aware voxel grids, and another that represents the time-invariant global features of each object. To achieve this, we introduce Dyna Vol, which incorporates object- centric voxelization into the unsupervised learning framework of inverse rendering. The key idea is that object-centric voxelization allows to infer the probability distribution spatial locations, thereby naturally facilitating 3D-consistent scene decomposition (see Figure 1}.
objects at individual
Dyna Vol consists of three network components in the test phase: (i) A dynamics module that learns the transitions of voxel grid features in canonical space over time; (ii) A volume slot attention module that progressively refines the object-level, time-invariant global features by aggregating the voxel representations; (iii) An object-centric, compositional neural radiance field (NeRF) for view synthesis, driven by both the local and global object representations. contrast prior research that focuses on
"Equal contribution.
Corresponding author; Yunbo Wang.

---

Published as a conference paper at ICLR 2024
Monocular video
Existing approaches
Latent codes
2D Reconstruction
decomposition
3D scene decomposition
Image encoder
Slot attention
Image decoder
Novel view synthesis
Scene editing
DynaVol
Object occupancy: (Ponj 0.7, Pouja 0.01, Patj3 0.02....)
View synthesis
Object-centric
Volume Slot attention
Object-centric
voxelization
NeRF
Figure 1: Dyna explores an unsupervised object-centric voxelization approach for dynamic scene decomposition. Unlike its 2D counterparts, such as SAVi (Kipf et al., (022), DynaVi Vol ensures 3D) consistency and provides additional capabilities, e.g., novel view synthesis and scene editing.
decomposing 2D images (Kipf et al. 2022 Sajjadi al. 2022 Elsayed al. 2022), ) our unsupervised
voxelization approach provides two additional advantages beyond novel view synthesis. First, it allows for fine-grained separation of object-centric information 3D space without any geometric priors. Second, it enables direct scene editing (e.g., object removal, replacement, and trajectory modification) that not feasible existing video decomposition methods. This done by directly manipulating the voxel grids the learned deformation function without the need for further training.
In our experiments, we initially evaluate DynaVol on simulated 3D dynamic scenes that contain different numbers of objects, diverse motions, shapes (such cubes, spheres, and real-world shapes), and materials (such as rubber and metal). On the simulated dataset, we can directly assess the performance of Dyna Vol for scene decomposition by projecting the object-centric volumetric representations onto 2D planes, and compare with existing approaches, such as SAVi (Kipf et al.) 2022) and uORF (Yu al. 2022). Additionally, demonstrate the effectiveness of Dyna Vol in novel view synthesis and dynamic scene editing using real-world videos.
2 PROBLEM SETUP
We assume a sparse of visual observations of dynamic timestamp and a set of RGB images (I, TT)T= collected with a moving monocular camera, where Iž TR. xxxWxx are images acquired under camera poses TV IR 4x4 T is the length of video frames and V is the number of views at the initial timestamp. The goal is to decompose each object in the scene by harnessing the underlying space-time structures present in the visual data without additional information. Please note that we also consider scenarios where (i) only one camera pose is available the first timestamp (i.e., V 1) and (ii) images are captured by monocular camera along smooth moving trajectory. These configurations can be easily achieved in real-world scenes.
T the initial
METHOD
In this section, we first discuss the overall framework of Dyna Subsequently, we introduce the concept of object-centric voxel representations and provide the details of each network component. Finally, we present the three-stage training procedure of our approach.
3.1 OVERVIEW OF DYNAVOL
Dyna Vol is trained in an inverse graphics framework to synthesize (T D and {I,TT> without any further supervision. Formally, the goal is to learn an object-centric projection of (x, d, t) {(c.. o..)) where x (z, y, 2) 3D point sampled by the neural renderer and is the predefined number of slots, which is assumed to be larger than the number of objects. The core of our approach is to introduce an object-centric 4D voxel representation, denoted by V2 E R x N₂ N, N., which maintains the time-vary density J.W of each possible object. The renderer estimates the density and color for each object at view direction and re-combines {(c.. A to approach the target pixel value. As shown in Figure ? DynaVol consists of three network components: (i) The
2

---

Published as a conference paper at ICLR 2024
3D Voxel Warmup
Learnable params
[(xe de)}
Finetuned params
Air
ft(tt)
No
J Sg
Dynamics module
SE(1)
Ad
N₂. M
Comp/non-con comp NeRF
Zw
Attention block with GRU
3D-to-40 4D Voxel Expansion
Eg
3D encoder
((x, d:))
Interp
P
(!)
Egg
1/2
Occupancy Probabiliy
4D Voxel Optimization
Z 3
X3
Figure 2: The architecture of Dyna Vol consists of three groups of network components: the bi- directional dynamics modules (fe. f')) the slot attention module with GRU and a 3D volume encoder (Zw., Ea), and neural rendering modules (Ny, N₂) based on 3D and 4D voxels respectively, The training scheme involves a 3D voxel warmup stage (top), a 3D-to-4D voxel expansion stage (middle, whose pseudocode is given in Appendix B and 4D voxel optimization stage (bottom).
bi-directional deformation networks La and I that learn the canonical-space transitions over time of the object-centric voxel representations contained within V4: (ii) The volume encoder Eg and slot attention block Zu that work collaboratively to refine a set of global slot features S, = {s} V iteratively, which convey time-invariant object information; (iii) The image renderers, which include compositional NeRF denoted by N& that jointly uses V, and S; to generate the observed images, and non-compositional NeRF denoted by No that is only used to initialize the voxel representations.
Dyna Vol involves three stages in the training phase: First, a 3D voxel warmup stage that learns to obtain a preliminary understanding of the geometric and dynamic priors. Second. 3D-to-4D voxel expansion stage that extends the 3D density grids Ft=1 E N.N. X,,N, to 4D dimension, initializing V==1 using the connected components algorithm. Third, 4D voxel optimization stage that optimizes (V=11 St=u, fv, Eo, Zw, No} to refine the object-centric voxel representations.
3.2 OBJECT-CENTRE VOXEL REPRESENTATIONS
We extend the idea of using 3D voxel grids to maintain the volume density for neural rendering with 4D voxel grids, denoted V. The additional dimension indicates the occupancy probabilities of each object within each grid cell. The occupancy probability {a, at an arbitrary 3D location can be efficiently queried through the trilinear interpolation sampling method:
Interp(xx, V.) R NeNN...NN,NNN)). R.
where (Nz, Ny, N₂) are the resolutions of Vg- To achieve sharp decision boundaries during training, we apply the Softplus activation function the output trilinear interpolation.
3.3 MODEL COMPONENTS
Canonical-space dynamics modeling. As shown in Figure 2 we use dynamics module f&x to learn the deformation field from Vi=1 at the initial timestamp to its canonical space variations over time. Notably, in the warmup stage, fø learns to transform the density values in F,. Given 3D point Vx; E {x} at an arbitrary time, (X,, t) predicts a position movement Ax; so that we can transform to the scene position at the first moment by x; Ax, We then query the occupancy probability from Vi=1 by D. Interp ((x, (x, t)) Vz=1). Notably, we encode x; and into higher dimensions via positional embedding. Additionally, in the warmup stage, we use another dynamics module fe to model the forward movement Ar' from initial movement to timestamp This module enables the calculation of cole-coonistenccy loss, enhancing the coherence of the learned canonical-span -soncc transitions. Furthermore, it provides useful input features representing the forward dynamics starting from the initial time step for the subsequent connect components algorithm, which can improve the initialization of Vi=1-
Volume slot attention. To progressively derive a set of global iine-invariant object-level repre- sentations from the local time-varying voxel grid features, employ volume slot attention in approach. Specifically, we use a set of latent codes referred to as "slots" to represent these object-level features. This terminology is in line with prior literature 2D static scene decomposition (Locatello
3

---

Published as a conference paper at ICLR 2024
al. 2020). The slots are randomly initialized from normal distribution and progressively refined episode by episode throughout our second training stage. The episode refers to training period that iterates from the initial timestamp to the end of the sequence. We denote the collection of slot features by S E RRND xD where e is the index of the training episode and D is the feature dimensionality.
At the beginning of each episode, we initialize the slot features as =i=0 S = + -
where $1.7' is the average of slot features across all timestamps in the previous episode. Each slot feature captures the live-invariant properties such as the appearance of each object, which enables the manipulation of the scene's content and relationships between objects. To bind the voxel grid representations to the corresponding object, at timestamp t, we pass V. through 3D CNN encoder Eg, which consists of 3 convolutional layers with ReLU. It outputs N flattened features h, E RMx,, where M represents the size of the voxel grids that have been reduced in dimensionality by the encoder. From an optimization perspective, a set of well-decoupled global slot features can benefit the separation of the object-centric volumetric representations. To refine the slot features, we employ the itcrative attention block denoted by Zw to incorporate the flattened local features h, In a single round of slot attention at timestamp & 1, we have:
A softMaxN (1) k(ba)) ((S.-))) W
U: WT v(he),
(2)
EE
where S denotes Si-1 in short, (g, k v) are learnable linear projections RDD Luong et al. 2015). such that Ar E RMN and Ue E RNDD, and /D fixed softmax temperature (Vaswani et al. 2017). The resulted slots features are then updated by GRU as GRU(4, S,-1). We update the slot features and obtain S, by repeating the attention computation 3 times each timestamp. For more in-depth analyses volume slot attention, please refer to Appendix F
Object-centric renderer. Previous compositional NeRF, like in uORF Yu et al. 2022), typically uses an ML.P to learn a continuous mapping from sampling point x, viewing direction d and slot features to the emitted densities } and colors {c,, } of different slots. Our neural renderer takes as inputs { N Ja=1 mcan(S. S, Six') ) at timestamp t. As discussed above. Siz' is the averaged slot features in the previous episode, which can more stable than the frequently refined features S,.
We perform object-centric projections using MLP: N₂ (x, d. (8 - {cn). and query {an) mean to
directly from the voxel grids 14 at the corresponding timestamp. We use compose the predictions of c, and on for different objects, such that:
N
N
Wn On/ / Onn
T
W
Σ
WnCan
(3)
n=1
n=1
where v and e the output density and the color of a sampling point. We estimate the color C(r) of
a sampling ray with the quadrature rule Max 1995y: Ö(r) = Σn= Ti (1 - exp(-ō,6.) T, where
T; - ,, P is the number of sampling points in certain ray, and &; is the distance between adjacent samples along the ray.
3.4 TRAINING
We train the entire model of Dyna using neural rendering objective functions. At a specific timestamp, we take the rendering loss LRender between the predicted and observed pixel colors, the background entropy loss LEar- and the per-point RGB loss Lpiin following DVGO (Sun et al.) 2022) as basic objective terms. Lem can be viewed regularization to encourage the renderer to concentrate on either foreground or background. To enhance dynamics learning in the warmup stage, we design a novel cycle loss between fo and fe:
LRender
1
Σ FR
[((T)
(((r)
LEM
1
Σ
of log(u)) (1 - (t) log(1 w
R
[
PER
P. جل ((r)
P
Lpoint
Σ
Leye
1
fo () (1 1)
f((2),
R
P.
R
P
FER
i=0
PER
1=0
where 2/ xi + (x 1),4 E [0. Pr). R is the set of sampled rays in batch, P, is the number of sampling points along ray r, and uf is the color contribution of the last sampling point obtained by exp(-opt pp )). We discuss the three stages in the training phase below.
2/ = TT, (1 exp(
4

---

3D-to-4D voxel expansion stage. We extend the 3D voxel grids F:=1 to 4D voxel grids V using the connected components algorithm, in which the expanded dimension corresponds the number of slots (N). The input features for the connected components algorithm involve the forward canonical-space transitions generated by fg and the emitted colors by Ngr. The output clusters Published as a conference paper at ICLR 2024
3D voxel warmup stage. To reduce the difficulty of learning the object-centric 4D occupancy grids, we optimize the 3D density grids Ft=1 RN, x N,N,N and warmup fy using (I: 1V and [I To enhance the coherence of the learned canonical-space dynamics, we train an additional module to capture the forward deformation field A using Lcye in Eq. 4 Az is used in the subsequent connect components algorithm to improve the initialization of V1 We train the bi-directional (S': (w) and the non-compositional neural renderer Ng based on Ft The overall
objective defined as Lwam ΣT= (ERender + ag Coint + ae + acLoye). The hyperparameter values are adopted from prior literature (Liu et al. 2022).
symbolize different objects, based on our assumption that voxels of the same object have similar motion and appearance features. More details are given in Appendix
4D voxel optimization stage. In this stage, we finetune fa and Vi=1 obtained in the first two stages respectively. Following an end-to-end training scheme, the dynamics module, volume slot attention mechanism, and compositional renderer collaboratively contribute to refining the object-centric voxel grids. The loss function in this stage defined as LDyn = Ei= (LRender + Op Opiiti + Oge (Em)) where finetune (Vio+ 46) and train (0, w, from the scratch.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Datasets. We build the synthetic dynamic scenes in Table using the Kubric simulator (Greff et al. 2022). Each scene spans 60 timestamps and contains different numbers of objects in various colors, shapes, and textures. The objects have diverse motion patterns and initial velocities. All images have dimensions of 512 x 512 pixels We also adopt 4 real-world scenes from HyperNeRF (Park) et al. 2021) and DrNeRF Wu et al. 2022), as shown in Table 2 For the synthetic scenes, we follow (Pumarola al. 2020) employ images collected viewpoints randomly sampled on the upper hemisphere. In contrast, real-world scenes are captured using a mobile monocular camera.
Metrics. For novel view synthesis, we report PSNR and SSIM (Wang et al. 2004). To compare the scene decomposition results with the 2D methods, we employ the Foreground Adjusted Rand Index (FG-ARI) (Rand 1971 Hubert & Arabie 1985). It measures the similarity of clustering results based on the foreground object masks to the ground truth, ranging from - (higher is better).
Table 1: Novel view synthesis results of our approach compared with D-NeRF (Pumarola et al. 2020), DeVRF (Liu al. 2022), and their variants align with our experimental configurations (see text for details). We evaluate the results averaged over 60 novel views(one view per timestamp).
30BJFALL
3OBJRAND
30RJMETAL. PSNR1
3FAII+3STILL
METHOD
PSNR
SSIM
PSNR
SSIM
SSIM
PSNR
SSIM
D-NERF (V=1)
28.54
0.946
12.62
0.853
27.83
0.945
24.56
0.908
D-NERF (V=
29.15
0.954
27.44
0.943
28.59
0.953
25.03
0.913
DEVRF
24.92
0.927
22.27
0.912
25.24
0.931
24.80
0.931
DEVRE-DYN
18.81
0.799
18.43
0.799
17.24
0.769
17.78
0.765
OURS (V 1)
31.83
0.967
31.10
0.966
29.28
0.954
27.83
0.942
OURS (V =
32.11
0.969
30.70
0.964
29.31
0.953
28.96
0.945
60BIFALL
8OBJFALL PSNR1
30B/REALSIMP PSNR1
30BJREALCMPX
METHOD
PSNRT
SSIM1
SSIM
SSIM
PSNR1
SSIM
D-NERF =V=
28.27
0.940
27.44
0.923
27.04
0.927
20.73
0.864
D-NERF -V=
27.20
0.928
26.97
0.919
27.49
0.931
22.72
0.874
DEVRE DEVRF-DYN
24.83
0.905
24.87
0.915
24.81
0.922
21.77
0.891
17.35
0.738
16.19
0.711
18.64
0.717
17.40
0.778
OURS (V =1)
29.70
0.948
29.86
0.945
30.20
0.952
26.80
0.917
OURS (V 55
29.98
0.950
29.78
0.945
30.13
0.952
27.25
0.918
5

---

Published as a conference paper at ICLR 2024
Table 2: Quantitative comparisons for novel view synthesis in real-world scenes from HyperNeRF and DeN. On average, our approach outperforms HyperNeRF by 5.7% in PSNR and 7.2% in SSIM. also outperforms NeR, another neural renderer with disentanglement learning.
CHICKEN
BROOM PSNR
BANAABANANA SSIM1
DUCK
AvG.
METHOD
PSNR
SSIM
SSIM
PSNRT
PSNR1
SSIMT
PSNR1
SSIM
NEURDIFF
21.17
0.822
17.75
0.468
19.43
0.748
21.92
0.862
20.07
0.725
HYPERNERF
26.90
0.948
19.30
0.591
22.10
0.780
20.64
0.830
22.23
0.787
D'NERF
24.27
0.890
20.66
0.712
21.35
0.820
22.07
0.856
22.09
0.820
OURS
27.01
0.934
21.49
0.702
24.07
0.863
21.43
0.878
23.50
0.844
30bjRand,
3Fall+3Still, t=50
30bjRealCpmx, t=50
D-NeRF
DeVRF
Ours
Chicken
Peel-Banana
Figure 3: Novel view synthesis results in synthetic and real- world scenes. D-NeRF fails in 3ObjRand. We also visualize corresponding decomposition maps for real-world scenes. It's important to note that decomposing moving objects in real-world scene often requires additional geometric priors. Relying solely on space-time continuity within single scene may not be sufficient to separate objects like hand and banana, especially when they are contact throughout the video.
Compared methods. In the novel view synthesis task for synthetic scenes, we evaluate Dyna against D-NeRF (Pumarola et al.) 2020) and DeVRF (Liu et al. 2022). For real-world scenes, we compare it with NeuralDiff (Tschernezki et al., 2021). HyperNeRF (Park al. (021), and DeN (Wu et al., 2022). In the scene decomposition task, we use established 2D/3D object-centric representation learning methods, SAVi (Kipf al. 2022) and uORF YYu al. 2022). as the baselines. These models are pre-trained on MOVi-A (Greff et al. 2022) and CLEVR-56" (Yu et al., (2022 synthetic scenes. Besides, also include pretrained and publicly available SAM model (Krillovv et al. 2023) to compare the segmentation performance.
respectively, which similar
4.2 NOVEL VIEW SYNTHESIS
Synthetic scenes. We evaluate the performance of Dyna Vol on the novel view synthesis task with the other two 3D benchmarks (D-NeRF and DeVRF). For fair comparison, we implement "D-NeRF (V = 5)" which is trained using multi-view frames at the first timestamp. Besides, since DeVRF trained with V 60 views at the initial timestamp and 4 views for the subsequent timestamps, we additionally train DeVRF model in a data configuration similar to ours. That is, we only have access to one image per timestamp (1 2 1) a dynamic view randomly sampled from the upper hemisphere. This model is termed "DeVRF-Dyie achieves the best results in terms of PSNR and SSIM. Notably, there is no remarkable difference between the performance of approach when trained with = 1 or V 5 initial images, which demonstrates that our model can be potentially applied to scenarios with monocular video data. In contrast, DeVRF-Dyn has significant decline in performance compared to its standard baseline due to its heavy dependence on accurate initial scene understanding.
As shown in Table T Dyna Vol consistently
Real-world scenes. We also evaluate the performance of Dyna Vol in real-world scenes with the cutting-edge NeRF-based methods (NeurDiff, HyperNeRI, and D²NeRI), As shown in Table 2 DynaVol outperforms the previous ofate-of-the-arr method HyperNeRF by 5.7% in PSNR and 7.2% in SSIM average. It also outperforms the other neural rendering method that learns disentan- gled representations, NeR, by 6.4% and 2.9% the respective metrics. For more real-world experiments, please refer Appendix

---

Published as a conference paper at ICLR 2024
Table 3: Results in FG-ARI for object-centric scene decomposition. Since SAVi requires videos with fixed viewpoints, we generate an image sequence with stationary camera and also evaluate Dyna (FixCam) using the fixed viewpoints. Notably, unlike DynaVol, the compared models including SAM are fully generalizing models trained beyond the test scenes. In contrast, our model enables 3D scene
decomposition by overfitting each individual scene in unsupervised manner.
METHOD
3FALL
3RAND
3METAL
3F+3S
6FALL
SFALL
3SIMP
3CMPX
SAVI
3.74
4.38
3.38
6.12
6.85
7.87
3.10
4.82
OURS (FIXCAM, 5)
94.53
93.30
94.91
93.20
93.42
93.42
91.22
91.84
UORF
28.77
38.65
22.58
36.70
29.23
31,93
38.26
33.76
SAM
70.77
55.52
46.80
47.36
62.66
71.68
56.65
51.91
OURS (V 1)
96.89
96.11
85.78
92.76
95.61
93.38
94.28
95.02
OURS (V = 5)
96.95
96.01
96.06
94.40
94.73
95.10
93.96
95.26
Observed
Learned
Slots/segments
image
Mask
SAVI
uORF
SAM
Ours
Figure 4: Visualization of scene decompostion results for 60bjFall.
Qualitative results. Figure showcases the rendered images arbitrary timestamp from a novel view. On the synthetic dataset, it shows that DynaVol captures 3D geometries and the motion patterns of different objects more accurately than the compared methods. In contrast, D-NeRF struggles to capture intricate dynamics and fails to model the complex motion 3ObjRand. Additionally, generates blurry results in 3ObjRealCpmx. DeVRF, on the other hand, also fails accurately model the trajectóries of the moving object with complex textures, as highlighted by the red box in 30bjRealCmpx. The real-world examples on the right side of Figure F demonnttrtee that Dyna Vol produces high-quality novel view images by understanding the stereo object-centric nature of the scene. This is further illustrated by the scene decomposition maps projected onto 2D) planes.
4.3 SCENE DECOMPOSITION
Quantitative results. To obtain quantitative results for 2D segmentation as well as the scene decomposition maps, we assign the casted rays in volume rendering to different slots according to each slot's contribution to the ray's final color. proviids comparison between Dyna Vol with 2D/3D unsupervised object-centric decomposition methods (SAVi/uORF) and a pretrained Segment Anything Model (SAM). For fair comparison with SAVi, which works on 2D video inputs, we implement Dyna Vol model using consecutive images with fixed camera view (termed as FixCam). For uORF and SAM, as they are primarily designed for static scenes, we preprocess the dynamic sequence into T individual static scenes the inputs of these models and evaluate their average performance on the whole sequence. The FG-ARI results in Table@sshow that DynaVol (V = and Dyna Vol (V 5) outperform all of the compared models by large margins.
Qualitative results. In Figure we randomly select a timestamp on 6ObjFall and present the object-centric decomposition maps of SAVi, uORF, SAM, and Dyna Vol. We have the following three observations from the visualized examples. First, compared with the 2D models (SAVi and SAM), DynaVol Vol can effectively handle severe occlusions between objects in 3D space. It shows the ability to infer the complete shape of the objects, as illustrated in the 1st and 5th slots. Second, compared with the 3D decomposition uORF, Dyna Volcan better segment the dynamic scene by leveraging explicitly meaningful spatiotemporal representations, while uORF only learns latent representations for each object. Last but not least, Dyna Vol adaptively work with redundant slots. This means that the
7

---

Published as a conference paper at ICLR 2024
Remove object
Modify trajectory
Figure 5: Showcases of dynamic editing for both wolld (top) and synthetic scenes (bottom).
We visualize the edited decomposition map and the corresponding rendering results.
Table 4: Ablation studies of the key components and the training stages in Dyna Vol.
VolaVol.
30BIFALL
60BJFALL
30BIREAL.MMP
PSNR
FG-ARI
PSNR
FG-ART1
PSNR1
FG-ARIf
w/O FORWARD DEFORMATION ft())
31.50
95.66
29.75
92.94
27.35
94.90
w/o VOLUME SLOT ATTENTION
31.71
95.33
30.09
92.32
27.16
94.86
w/o AGERAGE SLOTS 1.T
31.68
96.19
29.90
92.97
27.22
94.90
w/o 4D VOXEI OPTIMIZATION
29.69
92.03
27.87
89.85
24.98
92.91
4D VOXEL OPTIM. FROM SCRATCH
31.69
31.81
29.37
20.82
28.55
33.89
FULL MODEL
32.11
96.95
29.98
94.73
27.25
95.26
pre-defined number of slots can be larger than the actual number of objects in the scene. In such cases, the additional slots can learn to disentangle noise visual observations or learn to not contribute significantly to image rendering, enhancing the model's flexibility and robustness.
4.4 DYNAMIC SCENE EDITING
After the training period, the object-centric voxel representations learned by Dyna can be readily used in downstream tasks such as scene editing without the need for additional model tuning. Dyna Vol allows for easy manipulation of the observed scene by directly modifying the object occupancy values within the voxel grids or switching the learned deformation function to pre-defined one. This flexibility empowers users to make various scene edits and modifications. For instance, in the first example in 5 we remove the hand that is pinching the toys. In the second example, modify dynamics of the shoe from falling to rotating. More showcases are included the appendix.
4.5 FURTHER ANALYSES
Ablation studies. We present ablation study results in Table 4 First, we can find that all network components are crucial to the final rendering and decomposition results. Furthermore, the absence of the 4D voxel optimization stage, the performance of Dyna Vol significantly degrades, highlighting the importance of refining the object-centric voxel representation with slot-based renderer. Addition- ally, we conduct an ablation study that performs 4D voxel optimization from scratch with randomly initialized Vz=1 and (-). The results clearly show that excluding the warmup stage has substantial impact the final performance, especially for the scene decomposition results.
first study the impact the slot number. From Table we can see that find that the presence of redundant slots has only minor impact on both rendering and decomposition results. Figure [ we explore the convergence the slot values during the training
Analysis the slot features.
Table 5: The analysis of the impact of the number of slots (N) on the performance of our approach.
30BIFALL PSNRT
60BJFALL
30B/REALCMPX PSNRT
# SLOTS
FG-ARI
PSNR1
FG-ARTt
FG-ARI
5
31.68
95.82
29.99
81.12
27.22
94.70
10 (OurS)
32.11
96.95
29.98
94.73
95.26 27.25
31.71
95.22
30.51
84.85
27.27
94.85
8

---

Published as a conference paper at ICLR 2024
3ObjRand
60bjFall
over feature dimensions
slot
0.2
Averaged slot values
slot
1.--
slot
0.0
-0.2
slots
0.4
slot
0.-
00.6
slot
slot 7
-0.8
slot 8
over
other slots
-1.0-
100
200
300
400
500
600
100
200
300
400
500
600
Training episode
Training episode
Figure 6: Visualization of the convergence of Si: during training.
process. Specifically, select the top slots (out of 10) that contribute most image rendering in 3ObjRand and 60bjFall. We present the average value across all dimensions of each slot ((x)) () at different training episodes. The results demonstrate that each slot effectively converges to stable value, indicating that the features become progressively refined and successfully learn time-invariant information about the scene. Furthermore, the noticeable divergence among different slots indicates that Dyna Vol successfully learned distinct and object-specific features.
time
RELATED WORK
Unsupervised 2D scene decomposition. Most existing methods in this area (Greff al. 2016
2019, Burgess et al. 2019 Engeleke et al.] (0200) use latent features to represent objects in 2D scenes.
The slot attention method (Locatello et al. 2020) extracts object-centric latents with an attention block and repeatedly refines them using GRUs (Cho et al.) 2014). SAVi (Kipf et al.) (022) extends slot attention to dynamic scenes by updating slots at each frame and using optical flow as the training
target. STEVE (Singh al. 2022) improves SAVi by replacing its spatial broadcast decoder with an autoregressive Transformer. SAVi++ (Elsayed et al. 2022) improves SAVi by incorporating depth information, enabling the modeling of static scenes with camera motion.
Unsupervised 3D scene decomposition. Recent methods (Kabra al. 202 Chen et al. 202
Stelzner et al. 2021 Yu et al., 20222 Sajjadi et al., 2022) combine object-centric representations with view-dependent scene modeling techniques like neural radiance fields (NeRFs) (Mildenhall ct al. 2020). ObSuRF (Stelzner et al. 2021) adopts the spatial broadcast decoder and takes depth information training supervision. uORF YYu et al., (2022) extracts the background latent and foreground latents from an input static image to handle background and foreground objects separately. For dynamic scenes, Guan al. (2022) proposed to use set of particle-based explicit representations in the NeRF-based inverse rendering framework, which is particularly designed for fluid physics modeling. Driess et al.) (2022) explored the combination of object-centric auto-encoder and volume rendering for dynamic scenes, which is relevant to our work. However, different from our unsupervised learning approach, it requires pre-prepared 2D object segments.
Dynamic scene rendering based on NeRFs. There is another line of work that models 3D dynamics using -eRed methods (Pumarola et al. 2020 Li et al., 2021 Liu et al. 2022; Wu et al.. 2022 Guo et al. 2022 Li et al.) 2023). D-NeRF (Pumarola al. 2020) uses a deformation network to map the coordinates of the dynamic fields to the canonical space. I et al. (202T) extended the original MLP in NeRF to incorporate the dynamics information and determine the 3D correspondence of the sampling points at nearby time steps. Li et al. (2023) achieved significant improvements dynamic scene benchmarks by representing motion trajectories by finding 3D correspondences for sampling points in nearby views. DeVRF (Liu et al., 2022) models dynamic scenes with volume grid features (Sun et al. 2022) and voxel deformation fields. NeRF (Wu et al. (022) presents a motion decoupling framework. However, unlike DynaVol, it cannot segment multiple moving objects.
CONCLUSION
In this paper, presented Vola an inverse graphics method designed to understand 3D dynamic scenes using object-centric volumetric representations. Our approach demonstrates superior perfor- mance over existing techniques in unsupervised scene decomposition in both synthetic and real-wor world scenarios. Moreover, it goes beyond the 2D counterparts by providing additional capabilities, such as novel view synthesis and dynamic scene editing, which greatly expand its application prospects.

---

Published as a conference paper at ICLR 2024
ACKNOWLEGMENTS
This work was supported by the National Natural Science Foundation of China (Grant No. 62250062, 62106144), the Shanghai Municipal Science and Technology Major Project (Grant No. 202 (02ISZZZX0002) the Fundamental Research Funds for the Central Universities, the Shanghai Sailing Program (Grant No. 217510202133) 33)), and the CCF-Tencent Rhino-Bird Open Research Fund.
REFERENCES
Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick. and Alexander Lerchner. Monet: Unsupervised scene decomposition and representation. In CVPR, 2019.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. Proceedings of the IEEE/CVF international conference computer vision, pp. 9650-9660, 2021.
Chang Chen, Fei Deng, and Sungjin Ahn. ROOTS: Object-centric representation and rendering of 3D scenes. Journal of Machine Leurning Research, 2021.
Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülgehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In EMNLP, 2014.
Danny Driess, Zhiao Huang, Yunzhu Li, Russ Tedrake, and Mare Toussaint, Learning multi-object dynamics with compositional neural radiance fields. arXiv preprint arXiv: 1855, 2022.
Gamaleldin F. Elsayed, Aravindh Mahendran, Sjoerd van Steenkiste, Klaus Greff, Michael Curtis Mozer, and Thomas Kipf. SAVi++: Towards end-to-end object-centric learning from real-world videos. In NeurIPS, 2022.
Martin Engelcke, Adam R Kosiorek, Oiwi Parker Jones, and Ingmar Posner. Genesis: Generative scene inference and sampling with object-centric latent representations. In ICLR, 2020.
Klaus Greff, Antti Rasmus, Mathias Berglund, Tele Hao, Harri Valpola, and Jürgen Schmidhuber. Tagger: Deep unsupervised perceptual grouping. In NeurIPS, 2016.
Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. In ICML, 2019.
Klaus Greff, Sjoerd Van Steenkiste, and Jürgen Schmidhuber. On the binding problem in artificial neural networks. arXiv preprint arXiv:2012.052000 2020.
Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David J. Fleet, Dan Gnanapragasam, Florian Golemo, Charles Herrmann Thomas Kipf, Ab- hijit Kundu, Dmitry Lagun, Issam H. Laradji. Hsuch-Ti Liu, Henning Meyer, Yishu Miao. Derek Nowrouzezahrai, Cengiz Oztireli, Etienne Pot, Noha Radwan, Daniel Rebain, Sara Sabour, Mehdi S. M. Sajjadi, Matan Sela, Vincent Sitzmann, Austin Stone, Deqing Sun, Suhani Vora, Ziyu Wang. Tianhao Wu, Kwang Moo Yi, Fangcheng Zhong, and Andrea Tagliasacchi. Kubric: A scalable dataset generator. CVPR, 2022.
Shanyan Guan, Huayu Deng, Yunbo Wang, and Xiaokang Yang. NeuroFluid: Fluid dynamics grounding with particle-driven neural radiance fields. In ICML, 2022.
Xiang Guo, Guanying Chen, Yuchao Dai, Xiaoqing Ye, Jiadai Sun, Xiao Tan, and Errui Ding. Neural deformable voxel grid for fast optimization of dynamic view synthesis. In ACCV, 2022.
Lawrence Hubert and Phipps Arabie. Comparing partitions. Journal of classification, 2:193-218, 1985.
10

---

Published as a conference paper at ICLR 2024
Rishabh Kabra, Daniel Zoran, Goker Erdogan, Loic Matthey, Antonia Creswell, Matt Botvinick, Alexander Lerchner, and Chris Burgess. SIMONe: View-invariant, temporally-abstraeted object representations via unsupervised video decomposition. In NeurlPS, 2021.
Thomas Kipf, Gamaleldin F Elsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff. Conditional object-centric learning from video. In ICLR, 2022.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan- Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. wwXiv:230.002444 2023.
Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. In CVPR, pp. 6498-6508, 2021.
Zhengqi Li, Qianqian Wang, Forrester Cole, Richard Tucker, and Noah Snavely. Dynibar: Neural 2023.
dynamic image-based rendering. In CVPR, pp.
Haotong Lin, Sida Peng. Zhen Xu, Yunzhi Yan, Qing Shuai, Hujun Bao, and Xiaowei Zhou. Efficient neural radiance fields for interactive free-viewpoint video. In SIGGRAPH Asia 2022 Conference Papers, pp. 1-9, 2022.
Wei Liu, Yan-Pei Cao, Weijia Mao, Wenqiao Zhang, David Junhao Zhang, Jussi Keppo, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. DeVRE: Fast deformable voxel radiance fields for dynamic scenes. In NeurIPS, 2022.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. In NeurIPS, 2020.
Minh-Thang Luong, Hieu Pham, and Christopher D Manning. Effective approaches attention-based neural machine translation. arXiv preprint (eti::150..444 2015.
Nelson Max. Optical models for direct volume rendering. IEEE Transactions on Visualization and Computer Graphics, 1995.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T. Barron, Sofien Bouaziz, Dan Goldman, Ricardo Martin-Brualla, and Steven M. Seitz. Hypernerf: A higher-dimensionall representation for topologically varying neural radiance fields. ACM Trans. Graph., 40(6), dec 2021.
Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francese Moreno-Noguer. D-NeRF: Neural radiance fields for dynamic scenes. In CVPR, 2020.
William M Rand. Objective criteria for the evaluation of clustering methods. Journal of the American Statistical association, 66(336):846885) 850, 1971.
Mchdi SM Sajjadi, Daniel Duckworth, Aravindh Mahendran, Sjoerd van Steenkiste, Filip Pavetić, Mario Lučić, Leonidas Guibas, Klaus Greff, and Thomas Kipf. Object scene representation transformer. In NeurlPS, 2022.
Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. A simple neural network module for relational reasoning. In NeurlPS, volume 30, 2017.
Gautam Singh, Yi-Fu Wu, and Sungjin Ahn. Simple unsupervised object-centric learning for complex and naturalistic videos. In NeurlPS, 2022.
Karl Stelzner, Kristian Kersting, and Adam R Kosiorek. Decomposing 3D scenes into objects via unsupervised volume segmentation. arXiv preprint arXiv:2 148, 2021.
Cheng Sun, Min Sun, and Hwamn- -oaann-Toonggg Chen. Direct voxel grid optimization: Super-fast conver- gence for radiance fields reconstruction. In CVPR, 2022.

---

Published as a conference paper at ICLR 2024
Vadim Tschernezki, Diane Larlus, and Andrea Vedaldi. Neuraldiff: Segmenting 3d objects that move in egocentric videos. In 2021 International Conference on 3D Vision (3DV), pp. 910-919. IEEE, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.
Zhou Wang, Alan Conrad Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 2004.
Jiajun Wu, liker Yildirim, Joseph Lim, Bill Freeman, and Josh Tenenbaum. Galileo: Perceiv- ing physical object properties by integrating physics engine with deep learning. In NeurIPS, volume 28, 2015.
Tianhao Wu, Fangcheng Zhong, Andrea Tagliasacchi, Forrester Cole, and Cengiz Öztireli. D2NeRF: Self-supervised decoupling of dynamic and static objects from monocular video. In NeurIPS, 2022.
Junyu Xie, Weidi Xie, and Andrew Zisserman. Segmenting moving objects via an object-centric layered representation. Advances Neural Information Processing Systems, 35:28023-280336 2022.
Zhen Xu, Sida Peng. Haotong Lin, Guangzhao He, Jiaming Sun, Yujun Shen, Hujum Bao, and Xiaowei Zhou. 4k4d: Real-time 4d view synthesis at 4k resolution. arXiv preprint arXiv:2310. /148, 2023.
Hong-Xing Yu, Leonidas J Guibas, and Jiajun Wu. Unsupervised discovery of object radiance fields. In ICLR, 2022.
12

---

Published as a conference paper at ICLR 2024
APPENDIX
A DATA DESCRIPTION
30bjFall. The scene consists of two cubes and cylinder. Initially, these objects are positioned randomly within the scene, and then undergo a free-fall motion along the Z-axis.
3ObjRand. We use random initial velocities along the and Y axes for each object in 3ObjFall.
30bjMetal. We change the material of each object in 3OhjFall from "Rubber" to "Metal".
3Sall+3Still. We add another three static objects with complex geometry to 3ObjFall.
60bjFall & Fall.all. We increase the number of objects in 3ObjFall to 6 and 8.
30bjRealSimp. We modify 3ObjFall with real-world objects that have simple textures.
3ObjReelCmmpx We modify 3ObjFall with real-world objects that have complex textures. Real-world data, We take Chicken, Peel-Banana and Broom from HyperNeRF (Park al. 2021 and Duck from DeNeRF (Wu et al. 2022).
B 3D-TO-4D VOXEL EXPANSION ALGORITHM
There are two steps in voxel expansion: the feature graph generation and (2) the connected components computation. We input the voxel density set {X&} and voxel position set {Xa} with size of N,, where N, N₂ x N2 Ng. Also, represents the density value threshold, digb represents the RGB distance threshold, dval represents the velocity distance threshold, and N, represents the number of sampling rays. By filtering out invalid locations from [x&} with density values below predefined threshold ddens we can build feature graph which incorporates information related to the geometry, color, and dynamics of the valid voxels. We assume that voxels belonging to the same object should have similar motion and appearance features. Conversely, voxels corresponding to objects in different spatial locations should be separated and exhibit diverse features. Subsequently, we expand the 3D voxel grid to 4D with connected components algorithm.
four parameters in advance, where den
C FURTHER EXPERIMENTS ON REAL-WORLD SCENES
To further validate Dyna'sl's capability on real-world scenes, we conduct additional experiments using the ENeRF-Outdook dataset(Lin al. (022). As shown in Table 6 DynaVol outperforms the HyperNeRF by 10.9% in PSNR and 11.0% in SSIM on average. Figure showcases the novel view synthesis results at arbitrary timestamp from novel view. Our findings reveal that Dyna Vol produces clearer results than HyperNeRF, especially in rendering shadows and objects held in hands. However, it's important note that our approach falls short compared state-of-the-art methods, such as 4K4D (Xu et al./ (023),, which are based on spherical harmonics. This disparity primarily stems from different focuses approach: the latter prioritizes renderer quality, while our emphasis lies in object-centric representation learning. Combining both approaches remains a prospect for future research.
Moreover, we showcase scene decomposition results in Figure where we compare Dyna Vol against the state-of-the-art unsupervised video segmentation approach, OCLR (Xie et al. 2022). Additionally, OCLR employs DINO features (Caron et al. 2021) for test-time adaptation (referred to OCLR adap)). Our findings indicate that Dyna Vol produces clearer segmentation, particularly on object boundaries, while maintaining temporal consistency. Conversely, OCLR exhibits challenges in maintaining temporal consistency. Even with test adaptation (OCLR(test adap)) still performs not well in maintaining consistency over extended durations.
D IMPLEMENTATION DETAILS
We set the size of the voxel grid to 1103, the assumed number of maximum objects N 10, and the dimension of slot features to = 64. We use 4 hidden layers with 64 channels in the renderer, and use the Adam optimizer with batch of 1,024 rays in the two training stages. The base learning
13

---

Published as a conference paper at ICLR 2024
Algorithm Pseudocode of the 3D-to-4D voxel expansion algorithm
1: Input: X' = { k=1 (xx J hyperparameter ddens Grgb Svet. N,, slot number N
2: Output: V1=1
3: N = [k|Xx > ddaa ko E [1,]
> Filter out invalid locations
4: for i - to N, do
5: Sample ray randomly with direction d;
Get color information
6: Calculate emitted color Ci. = Nor (X,, d.) for N
7: end for
8: Get averaged color C; = 1 N, E Ctj for EN
9: for + to T - do
Calculate velocity Vz.j f/( (x,, - f)( for E N end for
Get velocity information
N 12: for (p.g) E v)lu E E N do
Diga(p, q) = 12(C,, Cg)
Get color relationship
14:
Dra(p. q) = max(12(V(..p Ve.g) for & + 1 to To if neighboor(p.q) and Digo (p. q) < digh and Dva(p. q) 8vel then
> Get velocity relationship
16:
E(p, =1
D Create edge of the graph
else
18:
E(p, 4) =0
19: end if
20: end for
21: Generate graph feature with node V and edge relationship E
22: Obtain the cluster set {cl. M
k=1 CommectedCommoonens(((] > Get number of the smallest clusters to be merged Sort clusters by size and merge Ni cluster set
23: Ni- = M N +
24: [clx)
sort-and-merge(
25: for k + 1-110 to N do
Vx X2 for clk
Voxel expansion by assigning the density value of F to V
27: end for
rates are 0.1 for the voxel grids and 1e-3 for all model parameters the warmup stage and then adjusted to 0.08 and Se in the second training stage. The two training stages last for 50k and 35k iterations respectively. The hyperparameters in the loss functions are set to Op = 0.1, oa = 0.01, NVIDIA RTX3090 GPU and last for about hours.
O'g 1.0, a. 1.0. All experiments
E EXPERIMENTAL DETAILS OF SCENE DECOMPOSITION AND EDITING
Scene decomposition. To get the 2D segmentation results, assign the rays to different slots according to the contribution of each slot to the final color the ray. Specifically, suppose oi is the density of slot point have
P Bn
W
Onn
Σ 1=1
T:(1 -xp((o;0p)
(5)
En=o
where Win is the corresponding density probability and 8n is the color contribution the final color of slot n. We can then predict the label of 2D segmentation by ((r) = argmax, (Bn).
Scene editing. The object-centric representations acquired through Dyna Vol demonstrate the capability for seamless integration into scene editing workflows, eliminating the need for additional
Table 6: Quantitative comparisons for novel view synthesis in real-world scenes from ENeRF-Outdord On average, approach outperforms HyperNeRF by 10.9% in PSNR and 11.0% in SSIM.
ACTOR1 4
ACTOR2_3
ACTOR5.
AVG.
METHOD
PSNR1 SSIM
PSNRT SSIM
PSNR1
SSIM
PSNRT
SSIM
HYPERNERE
22.79
0.759
22.85
0.773
25.22
0.806
23.62
0.779
OURS
26.44
0.871
26.24
0.864
25.97
0.861
26.21
0.865
14

---

Published as a conference paper at ICLR 2024
t= 1
61
t= 89
GT
HyperNeRF
Ours
(a) Novel view synthesis results scene Actorl
t=1
t= 43
7=
GT
HyperNeRF (b) Novel view synthesis results on scene Actor2_3,
Ours
Figure 7: Novel view synthesis results on ENeRF-Qutdoor dataset.
training. By manipulating the 4D voxel grid Vdeesity learned Dyna objects can be replaced, removed, duplicated, added according specifie requirements. For example, swap the
15

---

Published as a conference paper at ICLR 2024
Table 7: Hyperparameter choice of the threshold value for splitting the foreground and background in 3D-to-4D voxel expansion stage.
30BIFALL
60B/FALL PSNR
30BIREALCMPX PSNR1
THRESH
PSNR1
FG-ARIf
FG-ARI
FG-ARIf
0.1
31.69
95.73
29.95
93.08
27.44
94.37
0.001
31.55
95.64
29.95
93.15
27.37
95.33
0.01(OURS)
32.11
96.95
29.98
94.73
27.25
95.26
Table 8: Hyperparameter choice of the weight of the Lpoin loss.
30BIFALL
60BJFALL PSNRT
3OB/REALCMPX FG-ARI1
Op
PSNRT
FG-ARI1
FG-ARI
PSNR1
0.0
31.49
95.30
30.46
95.06
26.84
95.18
29.90
89.33
29.25
93.65
25.49
55.52
0. (OURS))))
32.11
96.95
29.98
94.73
27.25
95.26
color between objects by swapping the corresponding slots assigned to each object. Moreover, the deformation field of individual objects can be replaced with user-defined trajectories (e.g., rotations and translations), enabling precise animation of the objects in the scene.
F FURTHER EXPERIMENTAL RESULTS
Ablation study of the hyperparameter choice. We study the hyperparameter choice of the threshold value for splitting the foreground and background in the warmup stage in Table and the weight(a,,) of per-point RGB loss in Table S It can found that our model is robust to the threshold, and the performance of Dyna Vol remains largely unaffected with different thresholds. Furthermore, can be found that ap 0.1 performs better than a, 0.0, indicating that a conservative use of Lipiint improve the performance. A possible reason is that it eases the training process by moderately penalizing the discrepancy of nearby sampling points on the same ray. When the weight of Cpoint increases ap 1.0, the performance of the method decreases significantly. Such substantial emphasis on Lpoint is intuitively unreasonable and can potentially introduce bias to the neural rendering process, thereby negatively impacting the final results.
Ablation study of 3D volume encoder. To derive object-level global representations from the 4D occupancy grids V,, we employ the 3D volume encoder within the volume slot attention mechanism. We evaluate Dyna Vol without the 3D encoder in Table which shows using 3D encoder to connect the global and local object-centric features is beneficial the performance of Dyna Vol.
Table 9: Ablation study of 3D volume encoder within volume slot attention.
30BJFALL PSNR
60BJFALL PSNR1
8OBIFALL PSNR1
METHOD
FG-ARI
FG-ARIT
FG-ARII
w/O 3D ENCODER
31.26
96.52
29.90 29.98
93.91
29.21 29.78
93.67
FULL MODEL
32.11
96.95
94.73
95.10
Ablation study of the slot attention steps Nit. The number of iterations in the slot attention is to refine the slots for more accurate object-centric features. We choose Ni. = in our method, as larger number of update steps at single timestamp may result in the overfitting problem. Table TT presents quantitative comparisons across various numbers of slot attention steps, revealing that Dyna Vol achieves optimal performance at Nir = 3.
Per-slot rendering results. We present the per-slot rendering results in Figure demonstrating DynaVoll Vol's robustness to varying slot numbers. Our findings indicate that redundant slots remains empty, thus mitigating the issue of over-segmentation.
16

---

Published as a conference paper at ICLR 2024
Table 10: Ablation study of the slot attention steps Nw repeated each timestamp.
3OBIFALL PSNR1
60B/FALL PSNRT
8OB/FALL
NITR
FG-ARIf
FG-ARI
PSNR*
FG-ARI
31.39
96.26
29.90
94.88
29.58
93.65
5
31.59
96.42
29.65
94.77
29.69
93.55
3(OURS)
32.11
96.95
29.98
94.73
29.78
95.10
Recon
Slot
Slot
Slot
Slot 4
Slot 5
N 5
Recon
Slot 1
Slot 2
Slot 3
Slot
Slot
N 10
Slot 6
Slot 7
Slot 8
Slot
Slot
Recon
Slot 1
Slot 2
Slot
Slot
Slot
Slot
Slot
Slot
Slot
1
Slot 11
Slot 12
Slot 13
Slot 14
Slot 15
Figure 8: Per-slot rendering results for scene 3ObjFall with 5, 10, and 15 slots, respectively.
Error bars assess the performance stability of Dyna Vol, we perform separate training processes using three distinct seeds. The results, presented in Table IT showcase the mean and standard devia- tion of the PSNR (Peak Signal-to-Noise Ratio) and FG-ARÍ (Adjusted Rand Index for Foreground) values for novel view synthesis and scene decomposition tasks. These additional results serve as valuable supplement to those presented in Table [ and in the main manuscript, demonstrating the consistent and reliable performance of Dyna Vol across multiple training trials.
G ENLARGED VISUALIZATION OF EDITED DYNAMIC SCENES
In Figures 10 13
we provide enlarged visualizations of the edited dynamic scenes for better clarity and observation. These figures provide a closer look at the specific changes made to the dynamic scenes, enabling better understanding of the editing process.
17

---

Published as a conference paper at ICLR 2024
t=5
=
t= 44
=
Image
OCLR
OCLR(test adap)
Ours
Figure 9: Scene decomposition results on ENeRF-Outdoor dataset.
Table 11: Quantitative results with error bars (mean std) of Dyna Vol.
30BIFALL
3OB/RAND
30B/METAL
3FALL+3STILL
PSNR
FG-ARI
PSNR
FG-ARI
PSNR
FG-ARI
PSNR
FG-ARI
32.0510.006 97.0310.08 70.7910.11 96 96.04L0.04 29.3710.07
96.0210.11 28.97-0.02
94.36-10.06
60BIFALL
8OBIFALL
30BJREALSIMP
30BJREALCMPX
PSNR
FG-ARI
PSNR
FG-ARI
PSNR
FG-ARI
PSNR FG-ARI
29.9910.07 94.7510.17
29.7910.06
95.12:1.03 30.1710.07 1010.13
27.1420.10
95.2310.05
For more examples, including object adding and duplication, please refer project page: https //s //sites. google. com/ooo////ooooooo ht///viw////////// http//ieww/dynvoo///// which demonstration showcases further instances of scene editing using Dyna Vol, providing comprehensive overview of its capabilities.
18

---

Published as a conference paper at ICLR 2024
Remove object
Figure 10: Real-world decomposition and scene editing. The left images illustrate the original scene before any editing, while the right images present the scene after object removal. is an enlarged version of Figure in the main manuscript.
Modify trajectory
Figure 11: Trajectory modification based on 30bjRealCmpx (Left: before editing: Right: after editing). is an enlarged version of Figure 5 the main manuscript.
19

---

Published as a conference paper at ICLR 2024
Replace object
Figure 12: Object replacement based on 30bjFall and 30bjRealCmpx (Left: before editing; Right: after editing).
Swap color
Figure 13: Color swapping based on 60bjFall (Left: before editing; Right: after editing).
20


Reviews:
[{"review_id": "RNWsOLHhHW", "reviewer": "Reviewer_8F9K", "summary": "This paper presents an optimization-based method for taking a monocular video as input and producing an object-centric 3D dynamic representation as output. The idea is to optimize a 3D voxel grid to learn the 3D occupancy/density field at each timestep, along with the warp that links each timestep to the zeroth timestep. Then, these occupancies are divided into a set of object representations using connected components, and then slot-based optimization proceeds, where  features associated with objects are averaged across the time axis, and per-timestep features are assigned to the objects via iterated slot attention. The representation is trained by rendering to the given views with a nerf-style setup, using a color loss, an entropy loss aiming to separate foreground/background, and a cycle-consistency objective on the scene flow fields. The results are impressive, both in simulation and in the two real videos.", "questions": "This is more of a personal curiosity, but: instead of learning two deformation networks (one forward, one backward) and then asking them to be cycle-consistent, would it work to learn an invertible mapping here instead? (Like with Real NVP.)", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "I like this paper. It is well written, it is interesting, the method seems sensible (thought I have a few questions), and the results look good.", "weaknesses": "I have a variety of fairly low-level questions, which I hope the authors can address. \n\nOverall I think the method would be easier to understand if the stages were presented in a more central way, instead of at the very end. After reading everything I am still a bit unclear on the meaning of an \"episode\". \n\nSection 2 emphasizes that \"we also consider scenarios where only one camera pose is available at the first timestamp\". What does it mean for only one camera pose to be available? Camera poses are relative to something. A single pose might as well be the identity matrix. \n\nThe beginning of Section 3.1 says the method will be trained \"without any further supervision\", without defining any initial supervision. So I suppose the question is: further to what?\n\nSection 3.1 introduces the number of objects N, as \"the assumed number of objects\". Is this manually chosen? Later on there is a connected components step, so I wonder if connected components could choose N instead.\n\n\nIt is unclear to me what exactly is happening in the warmup stage vs. the dynamics stage. From the names I might guess that the warmup stage does not involve any dynamics, but this is wrong, because it uses all timesteps and it also trains the forward and backward dynamics modules. The section also says \"To obtain the dynamics information, we train an additional module f′ξ\" -- but dynamics information is already obtained by f_\\psi, so there is some mixup here, unless the authors do not see f_\\psi as obtaining dynamics information. Finally, it's unclear how exactly the connected components and the \"feature graph\" connect with the rest of the method.\n\nL_point seems like it is never defined, and only a citation is given. It would be great to describe in English what is happening there.\n\nSection 3.3 seems to complain that prior compositional nerfs used an MLP to learn a mapping from position+direction+feature to color+density, and then proposes to do the exact same thing. Have I missed something here? \n\nOverall I like the paper and I hope it can be cleaned up. The video results show that the segmentations are a little messy, and it seems like only 2 real videos are used, but still I think the paper is interesting and useful to build on.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HZscre9LDO", "reviewer": "Reviewer_B2an", "summary": "This paper introduces DynaVol, an attempt at unsupervised learning for object-centric representations in dynamic scenes. While most existing solutions focus on 2D image decomposition, DynaVol aims to offer a 3D perspective by leveraging object-centric voxelization within a volume rendering framework. It has a few steps to achieve this, including but not limited to canonical-space deformation optimization, and global representation learning with slo attention. The approach combines both local voxel and global features as conditions for a NeRF decoder. The proposed method achieves good results in a simulation environment and demonstrates its applicability in one real-world dataset.", "questions": "1. Based on the results presented in Table 1, it seems that models utilizing 5 views (V=5) don't consistently surpass the performance of those with just a single view. Could the authors shed light on the underlying reason for this unexpected behavior?\n\n2. Significance of the Warmup Stage: The paper doesn't delve deeply into the role of the warmup stage in the ablation studies. How crucial is this phase to the overall performance and efficiency of DynaVol?\n\n3. Performance of Warmup-Stage-Only: It's intriguing to observe that the 'warmup-stage-only' variant appears to perform quite competently even though it's restricted to initial timestep observations. It's even better than other baselines reported in Table 1. Could the authors explain the factors or mechanisms behind this seemingly robust performance?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Good attempt. The proposed DynaVol approach introduces a shift from 2D image decomposition by bringing in a 3D perspective (via NeRF). NeRF as a 3D representation beyond view synthesis is an underexplored region and I think this work makes a good effort towards this direction by doing object-centric voxelization within a volume rendering framework. Although the canonical-space deformation operation is widely seen in dynamic NeRF works, it still brings value to object-centric learning.\n\n2. The paper is well articulated in the introduction section. I do capture the underlying motivation for the whole work, but I found a hard time understanding the two-stage training at the beginning. The authors are suggested to make the overview section and the overview figure flow better.", "weaknesses": "Dataset Limitations: My primary concern is the robustness of DynaVol when applied to intricate real-world scenes. The paper predominantly employs what can be described as a 'toy' dataset to validate its model. The observations made from this kind of dataset often fail to generalize to real-world applications. The mention of the HyperNeRF dataset application does lend some credence to its real-world viability. However, a more thorough exploration, perhaps using more diverse and challenging datasets (one or two demos would be enough), would provide stronger evidence of the model's capability.\n\nAlso, I wonder how important the background loss is in the pipeline. Will this method work without this loss? Will the slot decompose the background?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Di0WlsSd2A", "reviewer": "Reviewer_FQjp", "summary": "The manuscript proposes a method, DynaVol, that learns to represent dynamic scenes as decomposed into the different moving parts (objects). The input to the method is a sequence of frames and their pose.\nThis decomposition is learned in 3D by learning to deform a canonical volume into the 3d states at the different observation points. Slot attention on 3d voxel grids is used to learn time-invariant object representations that are deformed by the deformation field. A NERF rendering loss guides allows learning this from just the input image sequence.", "questions": "Do I understand correctly that there is only one occupancy grid at t=0 and the other ts are reconstructed using the deformation field? A query at time t is warped back to t=0 and then interpolated into the feature volume?\n\nWhere is the GRU in Fig 2?\n\nFor the NERF rendering is it correct that the occupancy comes from the occupancy grid and the color comes from the slot features? Why this choice and not push occupancy into the slot features too?\n\nI couldnt find the per-point color loss in the DVGO paper? What am I missing?\n\nWhat happens without the warmup phase? \n\nHow are the occupancy grid and the deformation function represented? Just dense values on a grid? what resolution?\n\nHow is the feature graph computed after warmup?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The research direction of learning to decompose and represent a dynamic scene in terms of the motion of individual constituent parts (objects / slots) is important. \n\nThe results on simulated data are encouraging (although there are only a few scenes shown) and show cases where the model is indeed able to nicely separate out the different moving parts of the scene.\nFigure 4 exactly captures the advantage of the proposed dynamic object-centric scene representation in comparison to 2d methods (SAM): being able to represent unobservable parts of an image.\n\nThe proposed approach to dynamic scene representation has fundamental advantages as demonstrated by the qualitative results on scene editing (removing objects; modifying dynamics).\n\nThe visuals and writing in the manuscript are of high quality (the clarity could be improved - see weaknesses).", "weaknesses": "The proposed model DynaVol is complex. The text and the architecture figures were not easy to follow. Especially the parts about the slot updating and training are a bit opaque to me still. There are various unexplained variables in Fig. 1 which means the figure doesnt really add much to explaining the concepts and architecture to me.\n\nOne weakness is that DynaVol has to be overfited to each individual single scene. As such this is similar to NERF and okay. But when comparing such an overfit model to a fully generalizing model like SAM caution has to be use to clarify very precisely. \n\nThe experiments do suggest to me that DynaVol works well on easily segmented objects (from the simulation) but on real sequences DynaVol does not much outperform other methods. The small set of sequences evaluated in sim and real weakens the claims.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Z1cF3gjDCZ", "reviewer": "Reviewer_p1WX", "summary": "This paper presents an unsupervised learning method for 3D dynamic scene capture and decomposition. The presented method, DynaVol, utilizes a canonical NeRF as well as a time-conditioned implicit warping field for modeling a 3D dynamic scene. Learning such a representation uses a monocular video together with multi-view observation of a static frame, and several other objectives designed for retrieving better geometry and enforcing temporal consistency. Comparisons with previous state-of-the-arts are shown on the tasks of novel view synthesis and 3D scene decomposition, and improved results are reported.", "questions": "* Supervision?\n\nAlthough not a very big concern, the number of objects N seems to serve as input to the pipeline. This semantic could serve as additional supervision with strong prior, especially to the task of scene decomposition. It would be interesting to see how to address the problem of automatically determining the number of objects in the scene.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* Experiment\n\n Extensive experimental results are shown on tasks like novel view synthesis and 3D scene decomposition. Improved results are achieved compared to many existing baselines and state-of-the-arts.", "weaknesses": "* More insights from ablation study?\n\nIt is really good to see that the authors conduct a lot of ablation study on the work. However, some of them only consist of a chart with statistics, while lacking some insight analysis on them. It would further improve the quality of this work if more insights are discussed there.\n\n* Novelty?\n\nThe method proposed in the manuscript is pretty interesting, although it would be great to highlight the novelty and difference between the proposed method and previous literature. For example, on the dynamic modeling side, what is the relationship with [1, 2]. On the representation side, the design of volume slot attention is also not very well justified. It would be great if more insights and ablations are provided for the that part of design.\n\n* Introduction\n\nIn the introduction part, I feel the importance of the problem that the paper is working on is not well explained. Instead, the author directly starts to focus on what method is proposed and what experiments are conducted. It would better enhance the quality of the paper if more background and impact of solving this problem are provided and discussed.\n\n[1] Li, Zhengqi, et al. \"Neural scene flow fields for space-time view synthesis of dynamic scenes.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n[2] Li, Zhengqi, et al. \"Dynibar: Neural dynamic image-based rendering.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)