DyST: Towards Dynamic Neural Scene Representations on Real-World Videos
Authors: Maximilian Seitzer
Sjoerd van Steenkiste
Thomas Kipf
Klaus Greff
Mehdi S. M. Sajjadi
Venue: ICLR 2024 spotlight (2024)
Keywords: neural scene representations
scene representations
representation learning
novel view synthesis
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We introduce the Dynamic Scene Transformer, a model which learns scene representations from real videos and achieves separate control over camera and dynamics.

Abstract: Visual understanding of the world goes beyond the semantics and flat structure of individual images. In this work, we aim to capture both the 3D structure and dynamics of real-world scenes from monocular real-world videos. Our Dynamic Scene Transformer (DyST) model leverages recent work in neural scene representation to learn a latent decomposition of monocular real-world videos into scene content, per-view scene dynamics, and camera pose. This separation is achieved through a novel co-training scheme on monocular videos and our new synthetic dataset DySO. DyST learns tangible latent representations for dynamic scenes that enable view generation with separate control over the camera and the content of the scene.

Published as a conference paper at ICLR 2024
DYST: TOWARDS DYNAMIC NEURAL SCENE REPRESENTATIONS ON REAL- WORLD VIDEOS
Maximilian Seitzer
Sjoerd van Steenkiste
Thomas Kipf 3
Klaus Greff
Mehdi S. Sajjadi
MPI for Intelligent Systems
Google Research
Google DeepMind
ABSTRACT
Visual understanding of the world goes beyond the semanties and flat structure of individual images. In this work, we aim to capture both the 3D structure and dynamics of real-world scenes from monocular real-world videos. Our Dynamic Scene Transformer (DyST) model leverages recent work in neural scene represen- tation to learn a latent decomposition of monocular real-world videos into scene content, per-view view scene dynamics, and camera pose. This separation is achieved through a novel co-training scheme on monocular videos and our new synthetic dataset DySO. DyST learns tangible latent representations for dynamic scenes that enable view generation with separate control over the camera and the content the scene.
INTRODUCTION
The majority of research in visual representation learning focuses on capturing the semantics and 2D- -truutuuctrr of individual images. In this work, instead focus on simultaneously capturing the 3D structure as well as the dynamics of a scene, which is critical for planning, spatial and physical reasoning, and effective interactions with the real world. We draw upon the recent progress in generative modelling of 3D visual scenes, which has moved away from explicit representations such as voxel grids, point-clouds or textured meshes in favor of learning implicit representations by directly optimizing for novel view synthesis (NVS). For example, Neural Radiance Fields, though initially limited single scenes with hundreds of input images with controlled lighting, precise camera pose and long processing times (Mildenhall et al., 2020), have since been extended to handle variations in lighting Startin-Brually al., 2021), 1) generalize across scenes (Trevithick & Yang, 2021), work with few images (Niemeyer et al., 2022), missing cameras (Meng, 2021), and even dynamic scenes (Pumarola et al., 2020).
A related line of research focuses on learning global latent neural scene representations (Sitzmann et al., 2021; Sajjadi et al., 2022b; Kosiorek et al., 2021). These approaches offer several advantages, including the ability to generalize from few views, and improved scalability and efficiency due to amortized learning (Sajjadi et al., 2022b). Most importantly, their tangible latent representations can be readily used for downstream applications (Sajjadi al., 2022a; Driess et al., 2022; Jabri et al., 2023). However, each of these models is limited to static scenes, which not only ignores the important aspect of scene dynamics, but also disqualifies the vast majority of potential real-world video datasets that contain dynamic scenes.
In this work, we present first steps towards learning latent dynamic neural scene representations from monocular real-world videos. Building up on recent work, our method learns a separation of the scene into global content and per-view camera pose scene dynamics, thereby enabling independent control over these factors.
Our core contributions are as follows:
We propose the Dynamic Scene Transformer (DyST), a model that learns latent neural scene representations from monocular video and provides controlled view generation.
"Wor done during internship Google DeepMind. Project website: dyst-paper.gthhuiio.

---

Published as a conference paper at ICLR 2024
ddg
Camera Estimator
6j
Control Latents
Dynamics Estimator
dd
601 Vd
yd
Encoder
Decoder
2 2
Input Views
Scene Representation
Prediction
Target View
The DyST model. Input views of a scene encoded into the scene representation 2 cap- turing the scene content. The model is trained with an L2 loss by synthesizing novel target views from Z. To identify the target view v2, the camera and dynamics estimators produce the low-dimensional camera control latents é1, di from views with matching camera (váz) and dynamics (y ). This scheme, termed latent control swap, induces separation of camera and scene dynamics the latent space (see Sec. 3.1). We co- co-train DyST on synthetic multi-view scenes and video, transferring the latent structure and thereby enabling controlled generation on real videos.
Figure 1:
monocular
Through unique training scheme, DyST learns latent decomposition of the space into scene content as well as per-view scene dynamics and camera pose.
We present a detailed analysis of the model and its learned latent representations for scene dynamics and camera pose.
Finally, we propose DySO, novel synthetic dataset used for co-training DyST. We will publish DySO to the community for use in co-training and evaluation of future work on dynamic neural scene representations.
RELATED WORK
The field of neural rendering static scenes has recently experienced a significant gain in popularity with the introduction of Neural Radiance Field (NeRF) (Mildenhall et al., 2020) which optimizes an MLP through the novel view synthesis (NVS) task on posed RGB imagery. NeRF and similar scene-specific methods typically require very dense coverage of the scene along with accurate camera pose information. While there exists a great deal works tackling these shortcomings (e.g. Yu et al., 2021; Meng, 2021; Niemeyer al., 2022), this line of research largely focuses on view synthesis quality rather than learning latent scene representations. We refer the reader Tewari al. (2022) for recent overview.
More closely related to our approach are methods that learn 3D-aware latent scene representations from the NVS task. Examples in this line of work include GQN (Eslami et al., 2018), NeRF-VAB (Kosiorek et al., 2021), LFN (Sitzmann et al., 3021), and SRT (Sajjadi et al., 2022b). These methods combine a visual encoder with a camera pose-conditioned decoder network for novel view synthesis. More recently, Sajjadi et al. (2023) proposed RUST, which uses a Pose Estimator to train the model without any camera pose information. Contrary DyST, all of these methods are limited static scenes, and, with the exception of RUST, require accurate camera poses for training.
Learning scene representations from dynamic scenes comes with additional challenges: in the absence of calibrated multi-camera setups that record posed multi-view video, models have to learn to capture and disentangle scene dynamics from camera motion. In the easier case of posed multi-view video, high-fidelity solutions exist, such as D-NeRF (Pumarola et al., 2020), H-NeRF (Xu et al., 2021), NeRF-Dy (Li et al., 2022), and NeRFlow (Du et al., 2021). Acquiring the right data for these methods requires specialized recording setups and is not applicable to abundantly available monocular video. To overcome this limitation, several works including Neural Scene Flow Fields (Li et al., 2021), 1), Space- time Neural Irradiance Fields (Xian al., 2021), T8RF (Attal al., 2021). and Gao et al. (2021)
2

---

Common real-life videos monocular, i.c. both scene dynamics and camera vary together. but are strongly correlated over time, i.e. X' = {..}...> for time step f =1,, 1,..., TT complicating the learning of disentangled representations from this source of data. Our goal is learn latent neural Published as a conference paper at ICLR 2024
make use of additional priors obtained from optical flow depth to support NVS for small camera displacements directly from monocular video. NerFPlayer (Song et al., 2023) and RoDynRF (Liu et al., 2023) are able model dynamic scenes without depth or optical flow information by learning separate NeRFs for dynamic and static parts of the scene, though they require training a separate model for every video without generalizing between scenes. MonoNeRF (Fu et al., 2023) learns generalizable scene representations from pose- free monocular videos, but different from our method, assumes static underlying scene, i.e. it not applicable videos of dynamic scenes.
METHOD
Dynamic Scenes. A dynamic scene X consists of an arbitrary number of images or views x with associated camera pose and scene dynamics d,, i.e. A' = {-rh dg ......... The cameras c; define the extrinsics and intrinsics of the camera that this particular image has been taken with, while the
dynamics d define the position and shape of entities in the scene, e.g. the position of a moving car.
We note two special cases thereof: in static scenes, there exist no scene dynamics, hence only the camera view varies across views: X' {x-2 .. ...... This static-scene assumption significantly simplifies the setting by allowing the usage of basic photogrammetric constraints (Mildenhall et al., 2020), and it forms the basis for the majority of prior works on learned implicit representations (Sitzmann et al., 2021; Sajjadi et al., 2022b; Kosiorek et al., 2021).
scene representations on real-life video data while gaining independent control over both scene dynamics and the camera position.
Neural Scene Representations. Given dynamic scene X, we select a number of input views X [I.....} .I C X' that are encoded by the model into a set-based neural scene representation
Z = <* E Rd] EX))
(1)
We note that this setting is in contrast to most common approaches based on Neural Radiance Fields (NeRF) (Mildenhall al., 2020), which require training a model per scene and do not provide latent neural scene representations beyond the scene-specific MLP weights. Following Sajjadi et al. (2022b), the encoder Ency first applies a convolutional neural network each input view z X independently, then flattens the resulting feature maps into set of tokens that is jointly processed with an encoder transformer (Vaswani al., 2017).
To reproduce novel target view y E X of the same scene, we use transformer decoder Dec, that repeatedly cross-attends into the scene representation 2 to retrieve information about the scene relevant for this novel view. In order to do so, the decoder needs to be informed of the desired target view camera pose e and dynamics Most existing work only covers static scenes (Sitzmann et al., 2021; Kosiorek et al., 2021; Sajjadi et al., 2022b), and simply assumes known ground-truth camera position for querying novel views:
je Deco(e, 2).
Inferred latent camera poses. Access the ground-truth camera poses is strong assumption that does regularly not extend to real-world settings. In practice, camera parameters are often estimated using external sensors such as LIDAR, from the RGB views through cructure-from-Mottin (SIM) methods such as COLMAP (Schönberger & Frahm, 2016). However, these methods are generally noisy and regularly fail altogether (Meng, 2021; Sajjadi et al., 2023), especially so in dynamic scenes such as natural videos.
To lift any requirements for explicit camera poses on static scenes, Sajjadi al. (2023) introduce a Camera Estimator module CF2 that learns to extract latent camera pose è from the RGB data itself. More specifically, the camera estimator receives the target view y as input and queries parts of the scene representation Z' C Z to produce a low-dimensional camera control latent
غ CEo (v, Z') E N
(3)
3

---

Published as a conference paper at ICLR 2024
In practice, Z' contains tokens belonging to the first input view, encouraging the learned camera control latent to be relative to that specific view. We note that é lives in an abstract feature space, and it is not immediately compatible with explicit GT camera poses c, though such mapping can be learned from data (Sajjadi et al., 2023). This estimated camera control latent is then used to condition the decoder when generating an image during training. i.e. v Deco (é, 2), entirely removing the need for camera poses to train such a model. Sajjadi et al. (2023) show that the learned camera control latent distribution accurately follows the true poses, and that this model, on static scenes, achieves similar generation quality as baselines trained with ground truth pose information.
The network architecture of the camera estimator is similar to the encoder's: a CNN processes the target view, followed by transformer that alternates between cross-attending into the scene repre- sentation, and self-attending on the flattened feature maps. The final camera control latent is then produced by a global average pooling of the output tokens and a linear projection to N₂ dimensions.
3.1 STRUCTURING THE LATENT SPACE FOR DYNAMIC SCENE CONTROL
Up until this point, the setting in Eqs. (2) and (3) only admits static scenes. To apply our method on real-world videos, we need a way to allow variations in scene dynamics. We start with an approach similar to the learned camera control latents: a Dynamics Estimator DE sees the target view y and 2', and learns to extract a dynamics control latent d from the RGB views:
= DE Z') E R No.
(4)
The dynamics control latent d is used an additional query for the decoder reconstruct the target view: 05 Deca (č, d, Z).
Separating dynamics from camera pose. While this split between camera pose and scene dy- namies is convenient in theory, we note that thus far, there exist no structural differences between the ways in which and d are inferred or used. Hence, in practice, they likely will both learn to capture the camera pose and scene dynamics in a unified way, making it hard to control these aspects independently (as we confirm in practice in Sec. 4.4).
We propose to enforce this separation through a novel latent control swap training scheme that is outlined in Fig. 2. For training, we assume a scene X' where views are available for all combinations of camera poses } and scene dynamics (d As target frames, we choose the four views corresponding to all combinations of the camera poses C1, C2 and dynamics di, d2, i.e. our of tar- e2 er 22) J. For all remaining camera get views is {udj Mdy ddg Ddg poses and scene dynamics, we randomly sample subset
Target
Camera Powe
of views as input views.
In the following, we describe the decoding mechanism for the target view v the scheme for the remaining target views follows analogously. To generate this particular target view, the decoder needs access to (a latent estimate for) the camera pose ej and the scene dynamics d, Our Figure/ 2:
41
Soene Dynamics
Illustration of DySO scene.
key insight is that we do not necessarily need to estimate éj and dj from the target frame y itself. Instead, we can estimate the camera pose and scene dynamics from their respective counterparts:
G (V, 2"), d DE, (v z').
(5)
Crucially, we note that the most salient information y contains to render the target view y., is the camera pose and it is hence natural for the camera estimator CE to learn to extract camera pose information from its inputs. Similarly, y shares its scene dynamics dj with the target view to be generated, encouraging the dynamics estimator DE to extract scene dynamics. From these estimates, we render the target view:
02. Deca (CEo Z'), DEg (, 2)).
As consequence, the model is required to route all information regarding the camera pose through G, and all information regarding the scene dynamics through di, yielding the desired separation.
4

---

Published as a conference paper at ICLR 2024
3.2 SIM-TO-REAL TRANSFER
The approach outlined requires the availability of a special kind of multi-view, multi-dynamies dataset. In the real world, multi-view video would satisfy the requirements: the scene dynamics are synced through time, while each separate video instantiates a different view onto the same scene. However, acquiring such a dataset for training would narrow the applicability of the method to a limited set of domains. Instead, we suggest a sim-to-real setup: we generate synthetic dataset that contains the desired properties for inducing camera- -amer--- separation in the learned latent representations, while co-training on natural, monocular videos.
Synthetic dataset. We follow prior work using Kubric (Greff et al., 2022) and ShapeNet objects (Chang et al., 2015). Our dataset uses components from the MultiShapeNet (MSN) dataset (Sajjadi et al., 2022b) that has been introduced as challenging test bed for 3D representation learning from static scenes. MSN uses photo-realistic ray tracing, high resolution images as backgrounds, and has diverse set of realistic 3D objects.
Compared to MSN, our dataset has some distinct modifications. We place one object into each scene, as our primary goal is the separation of scene dynamics, in this case the object position & orientation, from the camera. The object is initialized with random position and pose the floor. To integrate dynamics, for each time step, we randomly jitter the object's 's position and further apply a random rotation to the object. MSN samples cameras on a half-sphere around the scene. Since real videos rarely feature 360 degree views, we use a set of more realistic camera motions: horizontal shifts, panning, and zooming, as well as sampling random camera points nearby fixed point.
We find this simple set of camera and object motions to be sufficient for successful sim-to-real transfer, and did not find the need to simulate more physically realistic object motions. Our dataset consists of IM dynamic scenes, each generated with all 25 combinations of 5 distinct scene dynamics and 5 camera views. We call this new dataset Dynamic Shapener Objects (DySO) and will make publicly available, as we expect it to be a useful tool for the community for training and evaluating dynamic neural scene representations in the future.
Co- fo-training To transfer the separation of dynamics and camera control latents from the synthetic scenes to natural monocular videos, we co-train on both types of data the same time. More specifically, we take alternating optimization steps on batches sampled from each of the two datasets. On the synthetic batches, we train the model according to Eq. (6), i.e. the camera pose and scene dynamics are estimated from views y and 2 different from the actual target view yd, to be rendered. On the batches with monocular videos, those additional views are not available, so we simply use the target view itself for dynamics and camera pose estimation:
(7) The latent control swap outlined in Sec. 3.1 is implemented efficiently by estimating the cameras and dynamics for all four (randomly sampled) target views in parallel before swapping the estimated é and dj control latents accordingly and decoding all four target views jointly.
0G = Dcc, (CE Z'), (VG 2'),
Model architecture. Our network architecture largely follows Sajjadi et al. (2023). For efficiency, we implement the Camera and Dynamics Estimators CE & DE as a single transformer that produces è and d simultaneously for a given target view (see also Fig. 8). The two control latents are produced differently: for the camera pose, we apply global average pooling to the transformer's 's output, and linearly project the result to produce é. For the scene dynamics, we add a learned token to the transformer's 's set of inputs, and apply a separate linear projection to to only the output for that token. This follows the intuition that the camera pose is a global property affecting all tokens (hence global pooling), whereas estimating the dynamics is a more localized task that only affects object tokens (hence the additional token that can learn attend to the respective subset of tokens). For the decoder Dec, we simply concatenate the two control latents to form a single query (é, d| which is used to cross-attend into the SLSR Z to generate the target view.
All model weights 0 are optimized end-to-end using the novel view synthesis (NVS) objective. Given a dataset DD = > of training scenes, for an arbitrary ground truth target view vå E X2, the model trained to minimize the L2 loss:
LNVS (8) = EX-D, (Xa))x [CEeco(CEo 2"), DE 2'), Enco(X)) &
(8)
where d tc, d' fd if X is synthetic, and d'=c, d' =d otherwise, i.e. on real-world videos.
5

---

Published as a conference paper at ICLR 2024
Input Views
DEg
Pred.
GT
Correct/ Matching
Wrong
CE2
y4
ya
Ma
Va
ya
DE
vā
vā
vā
Va
PSNR
26.7
26.6
26.0
26.0
18.3
22.3
Figure 3: NVS DySO. Left: Qualitative results. DyST is able to learn how extract camera & dynamics independently from the respective views of the scene, leading to the correct prediction of the GT image. Right: Quantitative performance for various inputs for CEg DEg. PSNR high even when camera or dynamics are only from matching views, showing that the model is capable of estimating camera & dynamics independently of the other.
Input Views
Time
Time
Recon GT
Figure 4: Frame synthesis on SSv2. We use the first, middle, and last frame as input views (marked in purple), and generate the intermediate frames based on the control latents estimated from them. DyST is able to render videos with challenging camera (left) and object motions (right).
4 EXPERIMENTS
Unless specified otherwise, DyST is always co-trained our synthetic DySO dataset and on real world videos as described Secs. 3.1 and 3.2. As a source of a real world videos, we use the Something- Soaeeihin v2 dataset (SSv2) (Goyal et al., 2017), which consists of roughly 170K training videos of humans performing basic actions with everyday objects. The videos contain both nontrivial camera motion and dynamic scene manipulation. From each video, we sub-sample clip of 64 frames. The model is trained using 3 randomly sampled input frames X to compute the scene representation Z (Xe(X) and randomly sampled target views for reconstruction. Following Sajjadi et al. (2022b), we train using batch size of 256 for 4M steps. For both control latents, we use dimensionality of No Na 8. We refer to Appendix A. for further implementation details.
4.1 NOVEL VIEW SYNTHESIS
We begin our investigations by testing the novel view synthesis (NVS) capabilities of the model on the test splits both the synthetic DySO and the real SSv2 datasets. We show several examples in Fig. (left). On DySO, the background is particularly well well-modelled, while the object is often less sharp. We attribute this to the difficulty of capturing 3D object appearance from only provided input views. For SSv2, we use the first, middle, and last frame of the videos as input frames and generate intermediate frames using control latents estimated from the intermediates. Qualitative results are shown in Fig. We find that our model accurately captures changes in both camera viewpoint and object dynamics.
Quantitatively, we evaluate the model on DySO by computing the PSNR against the ground truth target view V while varying the control latents, see Fig. 3 (right). We observe that there is only minor difference in PSNR between using the target y itself to estimate the control latents (y5/v)) versus estimating the latent camera from view with different scene dynamics (y²) and vice versa (vS) ). This further confirms that DyST can successfully estimate and re-combine camera pose and scene dynamics to synthesize the desired novel views.

---

Published as a conference paper at ICLR 2024
Camera Control Latent
Dynamics Control Latent
Camera Control Latent
Dynamics Control Latent
Frame
Frame
Frame
Figure 5: Latent distance analysis. Left: average L2 distances in the latent space between pairs of views on DySO, for camera (left) and dynamics control latents (right). Right: frame-to-frame L2 distances for a real world video, for camera (left) and dynamics control latents (right). The distances closely follow events in the video: grasping (second frame), turning (third frame) and placing the cup (last frame) are visible as distinct areas in the dynamics latent distances. The slow panning camera movements is reflected the broad diagonal stripe for the camera latent distances. Notably, the grasp has low distance to the similar placement motion despite the different camera positions, indicating that the model has learned to encode dynamics independently of the camera pose.
4.2 LEARNED CAMERA AND SCENE DYNAMICS CONTROL LATENTS
In this section, our experiments aim to answer three questions. First, does our training with latent control swapping induce a meaningful separation of camera pose and scene dynamics on synthetic data? Secondly, does the latent space separation transfer to real videos? And finally, what is the structure of the learned latent spaces?
Separation of Camera and Dynamics. To analyze the separation into camera and scene dy- namics quantitatively, we develop a metric which we call contrastiveness. Let Deam (y, v', 2') ||CE(y, 2 CF 2')ll denote the 1.2 distance camera latent space between views y and y/" Ddya is defined accordingly for the dynamics latent space. Then, compute the average ratio gaam between the distance camera latent space of a reference view yf to a matching view ya and to non-matching view v/a:
Deam (yaa yo 2')
Ream
E EX-D (X (X.VS joi
with e + c'and d + d'
Deam (2 yo 2')
Analogously, we define payn using vå as matching and y as non-matching view. the contrastive- ness metric Ream is z 0, this indicates that views with the same camera are mapped onto the same point in camera latent space regardless of their dynamics; Heam is Re 1, views with non-matching camera not be distinguished based on latent distance average. If H is >> 1, information from the dynamics latent space is "leaking" into the camera latent space, because views with the same dynamics d but different camera c' are average closer to y than views with different dynamics d', but the same camera c.
We find that the learned latent spaces well separated average, with little influence of one control latent on the other. In particular, our model achieves a camera contrastiveness of peam=0.06 (corr. to 16.7x less distance of matching non-matching views), and dynamics contrastiveness of 2d.m=0.42 (corr. to 2.4x less distance). The fact that Raam is notably smaller than 1d4y indicates that it is easier for the model to estimate the camera pose compared the object pose. This might be intuitive, as the camera pose can be estimated directly from all background pixels (unaffected by the scene dynamics), while the appearance of scene dynamics in 2D images is the result of combination of the actual scene dynamics and the camera pose. In contrast, when training without latent control swapping, the model only achieves m2aa==0.72 and pdya=1.26,, indicating that separation is drastically worse there (see also Tab. 1). This is further evidence that latent control swap training scheme induces the desired disentanglement of camera pose and scene dynamics. We also visualize the average latent distances heatmap in Fig. 5, and find that matching (blue squares) and non-matching views (red squares) are easily distinguishable.
Transfer to Real Video. To see whether the camera-dynamica separation transfers to real videos, measure the frame-to-frame latent distances example video of one of the authors grasping,

---

Published as a conference paper at ICLR 2024
Input
Camera
Dynamics
Horizontal
Height
Rotation
Rotation
Horizontal
Height
DySO
SSv2
Figure 6: Interpolating along the principal components camera and scene dynamics latent spaces on the synthetic DySO (first row) and the real world SSv2 dataset (second row) reveals that the model learns meaningful, isolated motions for both camera and dynamics.
lifting and turning a coffee cup. before placing back its original position. The resulting distance matrix is visualized in Fig. 5, We find both the slow panning motion of the camera and the distinct events in the video are clearly visible in the distance maps. Moreover, we point out that similar object poses lead to similar dynamics latents despite the vastly different camera positions. We conclude that the model is able to successfully disentangle camera pose from scene dynamics in real-world videos.
Learned Structure. To investigate the structure of the two learned control latent spaces, we apply principal component analysis (PCA) to a set of latent cameras è and scene dynamics d collected from 300 test scenes of the DySO dataset. Starting from the extracted latents è & d of some view, we then linearly interpolate along the principal components of each space and show the decoded images in Fig. 6. We find that the learned spaces capture meaningful movements: the first three components for the camera pose capture horizontal and vertical movements and planar rotation, mirroring similar observations by Sajjadi et al. (2023). For the scene dynamics, the first component captures object rotation, the second represents horizontal and the third vertical movements. Furthermore, we observe additional evidence that camera and dynamics are well separated: changing è does not modify the object's pose; similarly, changing d does not modify the camera. Lastly, also estimate the principal components on the SSv2 dataset, and find that they capture similar effects for the camera; for the dynamics, movements are more constrained to plausible dynamics fitting to the input frames.
We conclude that our model is able to disentangle camera and scene dynamics, and that learns a meaningful structure for both. We study how these capabilities can be used for test-time video control in the next section.
4.3 CONTROLLABLE VIDEO MANIPULATION
After establishing that DyST learns a disentangled representation camera and scene dynamics, we now show how this capability can be used to manipulate real-world video post-hoc. After encoding target video into the scene representation Z using the the first, middle and last frame, we synthesize counterfactual variations for this video by manipulating the control latents è & d. We study two variants: motion freezing and video-to-video motion transfer. For the former, we compute è or d from frame of the target video, and keep it fixed throughout the video. This freezes the corresponding camera or object motion from that frame throughout the video, creating a "bullet time" effect. For the latter, we replace è, d, or both, with the latents computed from different video, effectively transferring the motion in that video to the target frame video.
Figure shows examples of such manipulations. First, we find that inserting the latent camera è or scene dynamics of source frame correctly keeps the camera/ / object stable throughout the video, but with the original object camera trajectory. Second, we find that transferring the camera from a video with a camera zoom horizontal pan copies the corresponding motion onto target frame. Note how on the shift motion, the left edge of the frame becomes blurred - the respective areas are visible in the target video. and thus DyST fills in average guess due to the L2 loss. It is notable that both types of manipulations work in this fashion, as the manipulated control latents are fully -iitr---iittibttionn for the decoder given the scene representation.
8

---

Published as a conference paper at ICLR 2024
Targets
Zoom
Dynamics Camera
Figure 7: Video manipulations. Left: we re- re-generate the video (row 1) while fixing the camera è (row 2) or dynamics control latent (row 3) to one estimated from two frames of the video (columns I & 2) throughout. The fixed camera or dynamics appear frozen in place, whereas the corresponding other motion is unchanged. Right: transferring the camera pose è of source videos with zoom (row 1) and horizontal pan (row 2) to a target frame, creating a video with the respective camera motions.
Method
PPSR
[LPIPS
I.Raaa
RRyyn
Table 1: Evaluation of alternative training tech- niques evaluated for the y case (Fig. 3) on DySO. The alternatives perform worse across all image- based and contrastiveness-be based met- rics, showing that our latent control swap ap- proach (Sec. 3.1) leads to the best separation between latent camera and dynamics.
No swap
18.6
0.45
0.72
1.26
50% swap
25.4
0.36
0.26
1.07
Latent averaging
22.9
0.42
0.54
0.96
DyST
26.0
0.34
0.06
0.42
4.4 ABLATION STUDY
We investigate the importance of the latent control swap (Eq. (6)). We measure PSNR under latent con- trol swapping as described in Sec. 4.1, and latent space contrastiveness as described in Sec. 4.2 on the DySO dataset. The results are listed in Tab. 1. We compare several variations: applying no swap all (i.e., decoding the target el using control latents of yd, itself), swapping randomly with a 50% proba- bility, averaging the control latents of the target view and the swapped view together (e.g., averaging camera latents from yd1 and ydz to decode yd, The latter is motivated by the idea that the latents of matching views should eventually converge the same point. We find that all variations perform sig- nificantly worse than the latent control swap: without swapping, latent space separation does not occur as expected because there is no structural incentive that enforces PSNR is low. When swapping only 50% of the time, some separation emerges, but the model still has some chance to mix information from the target view into both control latents. For averaging the la- tents, also find substantial entanglement of camera and dynamics. Thus, our latent control swap is the most suitable training scheme for consistent separation of camera and dynamics in the latent space.
consequently, the recombination
CONCLUSION
In this work, we propose DyST, novel approach to generative modeling of dynamic 3D visual scenes that admits separate control over the camera and the content of the scene. We demonstrate how DyST can be applied to real-world videos of dynamic scenes despite not having access ground-truth camera poses via sim-to-real transfer. DyST shows promising view synthesis and scene control capabilities real-world videos of dynamic scenes.
As a next step, we would like to apply our method to more complex types of videos, for example with several independent moving objects, longer camera trajectories, changing lighting conditions challenges which exceed the scope of this work. Indeed, we expect that further model innovations are needed tackle them. Another open problem concerns the novel view synthesis aspect of our model; there is still room for improvement in view generation quality, especially for dynamic objects, which currently limited due the L2 loss. Future work could improve the model's generative capabilities: using diffusion or GAN-like approaches should lead to more plausibly imputed missing information.
We believe that DyST contributes significant step towards learning neural scene representations from real-world scenes encountered in the wild. This opens the door for exciting down wwwn-ttream applications, especially when combined with the potential of training on large-scale video collections.

---

Published as a conference paper at ICLR 2024
Author contributions & Acknowledgements. Maximilian Seitzer: conception, implementation, datasets, experiments, evaluation, writing. Sjoerd van Steenkiste: writing, advising. Thomas Kipf: writing, advising. Klaus Greff: project co-lead, conception, evaluation, writing. Mehdi S. M. Sajjadi: project lead, conception, datasets, early experiments, evaluation, writing.
We thank Aravindh Mahendran for detailed feedback the manuscript and Alexey Dosovitskiy for fruitful discussions on the project. The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Maximilian Seitzer.
REFERENCES
Benjamin Attal, Eliot Laidlaw, Aaron Gokaslan, Changil Kim, Christian Richardt, James Tompkin, and Matthew O'Toole. Törf: Time-of-fligh radiance fields for dynamic scene view synthesis. NeurIPS, 34, 2021.
Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. ShapeNet: An Information-Rich 3D Model Repository. arXiv preprint 1rXiv:111..03012, 2015.
Danny Driess, Zhiao Huang, Yunzhu Li, Russ Tedrake, and Mare Toussaint. Learning Multi-Object dynamics with compositional neural radiance fields. CoRL., 2022.
Yilun Du, Yinan Zhang, Hong-Xing Yu, Joshua B. Tenenbaum, and Jiajun Wu. Neural radiance flow for 4d view synthesis and video processing. In ICCV, 2021.
SM Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S Morcos, Marta Gar- nelo, Avraham Ruderman, Andrei A Rusu, Ivo Danihelka, Karol Gregor, et al. Neural scene representation and rendering. Science, 2018.
Yang Fu, Ishan Misra, and Xiaolong Wang. Mononerf: Learning generalizable nerfs from monocular videos without camera poses. ICML, 2023.
Chen Gao, Ayush Saraf, Johannes Kopf, and Jia-Bin Huang. Dynamic view synthesis from dynamic monocular video. In ICCV, pp. 5721, 2021.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fründ, Peter N. Yianilos, Moritz Mueller-Freitag. Florian Hoppe, Christian Thurau, Ingo Bax, and Roland Memisevic. The "something something" video database for learning and evaluating visual common sense. 2017 IEEE International Conference on Computer Vision (ICCV), 2017. URL http:: org/ abs/ /0as/1700 1006. 04261.
Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David Fleet, Dan Gnanapragasam, Florian Golemo, Charles Herrmann, et al. Kubric: A Scalable Dataset Generator. In CVPR, 2022.
Allan Jabri, Sjoerd van Steenkiste, Emiel Hoogeboom, Mehdi SM Sajjadi, and Thomas Kipf. Dorsal: Diffusion for object-centric representations of scenes et al.. arXiv preprint arXiv:2306.0800 2023.
Adam Kosiorek, Heiko Strathmann, Daniel Zoran, Pol Moreno, Rosalia Schneider, Sona Mokrá, and Danilo Rezende. NeRF- A Geometry Aware 3D Scene Generative Model. In ICML, 2021.
Yunzhu Li, Shuang Li, Vincent Sitzmann, Pulkit Agrawal, and Antonio Torralba. 3d neural scene representations for visuomotor control. In Conference on Robot Learning, pp. 112-123 PMLR. 2022.
Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. In CVPR, pp. 6498 6508, 2021.
Yu-Lun Liu, Chen Gao, Andreas Meuleman, Hung Yu Tseng, Ayush Saraf, Changil Kim, Yung- Yu Chuang, Johannes Kopf, and Jia-Bin Huang. Robust dynamic radiance fields. In CVPR, 2023.
10

---

Published as a conference paper at ICLR 2024
Ricardo Martin-Bruallaa Noha Radwan, Mehdi S. M. Sajjadi, Jonathan Barron, Alexey Dosovitskiy, and Daniel Duckworth. NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections. In CVPR, 2021.
Quan Meng. GNeRE: GAN-Based Neural Radiance Field Without Posed Camera. In ICCV, 2021.
Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In ECCV, 2020.
Michael Niemeyer, Jonathan T Barron, Ben Mildenhall, Mchdi SM Sajjadi, Andreas Geiger, and Noha Radwan. RegNeRF: Regularizing neural radiance fields for view synthesis from sparse inputs. In CVPR, 2022.
Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-NeRF: Neural Radiance Fields for Dynamic Scenes. In CVPR, 2020.
Mehdi S. M. Sajjadi, Daniel Duckworth, Aravindh Mahendran, Sjoerd van Steenkiste, Filip Pavetic, Mario Lucic, Leonidas Guibas, Klaus Greff, and Thomas Kipf. Object Scene Representation Transformer. In NeurIPS, 2022a.
Mehdi S. M. Sajjadi, Henning Meyer, Etienne Pot, Urs Bergmann, Klaus Greff, Noha Radwan, Suhani Vora, Mario Lucie, Daniel Duckworth, Alexey Dosovitskiy, et al. Scene Representation Transformer: Geometry-Free Novel View Synthesis Through Set-Latent Scene Representations, In CVPR, 2022b.
Mehdi S. M. Sajjadi, Aravindh Mahendran, Thomas Kipf, Etienne Pot, Daniel Duckworth, Mario Lučié, and Klaus Greff. RUST: Latent Neural Scene Representations from Unposed Imagery. CVPR, 2023. URL https: /aatpp:///rxxae org/
/tt/abs////eeeaacccee./eee/.eaer abs 2211.14305.
Johannes Lutz Schönberger and Jan-Michael Frahm. Structureffrom-Motion Revisited. In CVPR, 2016.
Vincent Sitzmann, Semon Rezchikov, William Freeman, Joshua B Tenenbaum, and Fredo Durand. Light field networks: Neural scene representations with single-evaluation rendering. In NeurIPS, 2021.
Liangehen Song, Anpei Chen, Zhong Li, Zhang Chen, Lele Chen, Junsong Yuan, Yi Xu, and Andreas Geiger. Nerfplayer: A streamable dynamic scene representation with decomposed neural radiance fields. JEEE Transactions on Visualization and Computer Graphics, 29(5):2732-27) 2023.
Ayush Tewari, Justus Thies, Ben Mildenhall, Pratul Srinivasan, Edgar Tretschk, Yifan Wang, Christoph Lassner, Vincent Sitzmann, Ricardo Martin Martin-Brualla,, Stephen Lombardi, et al. Ad- vances in neural rendering. Computer Graphics Forum, (2), 2022.
Alex Trevithick and Bo Yang. GRF: Learning a General Radiance Field for 3D Scene Representation and Rendering. In ICCV, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurIPS, 2017.
Wenqi Xian, Jia-Bin Huang, Johannes Kopf, and Changil Kim. Space-time neural irradiance fields for free-viewpoint video. In CVPR, pp. 9421-9431, 2021.
Hongyi Xu, Thiemo Alldieck, and Cristian Sminchisescu. H-nerf: Neural radiance fields for rendering and temporal reconstruction of humans in motion. NewrIPS, 44:14955-14966, 2021.
Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixeINeRF: Neural Radiance Fields from One or Few Images. In CVPR, 2021.
11

---

Published as a conference paper at ICLR 2024
A
APPENDIX
è
d
Linear Projection
Linear Projection
Avg.
Camera Dynamics Estimator
MLP
Self Self-Attention
3x
Key
Value
Query
Cross-Altention
Key
Value
Query
Scene Repr.
Query Tokens
Dynamics
Taken
CNN
Target View
Figure 8: Architecture of joint camera and dynamics estimator. We implement camera and dynamics estimator as a single transformer-basee module that iterates cross- and self-attention layers between the target frame and the scene representation. The camera control latent è is produced from global average pooling of the output of the last transformer layer, and low-dimension projection. The dynamics control latent is produced from projecting only the output corresponding to the learned dynamics token.
A.1 MODEL DETAILS AND HYPERPARAMETERS
Unless mentioned otherwise, the architecture and hyperparameters follow RUST (Sajjadi et al., 2023).
We briefly summarize the main details in the following.
Input. The model trained on images of size 128x128. We always use input views compute the scene representation Z.
Encoder. The initial CNN uses layers that reduce the input views to patches of size 8x8. Flattening, adding 2D positional encoding, and linear transformation yields 768 tokens with dimensionality 768. The tokens corresponding to the first input view receive an additional embedding such that they can be differentiated from the remaining views. To compute the scene representation. we then apply a layer transformer encoder with 12 attention heads, MLP with hidden-dimension 1536, and nrennomalizzaion. The scene representation Z consists of 768 tokens of dimensionality 768.
Camera & Dynamics Estimator. We implement camera estimator and dynamics estimator jointly in a single module that is similar to RUST's pose estimator. See Fig. 8 for an overview of the architecture. Similar to the encoder, CNN first transforms the target view into patches, but with layers, resulting in a patch size of 16x 16. Adding 2D positional encoding yields 64 query tokens of dimensionality 768, which we add single learned "dynamics token", whose output is used to compute the latent scene dynamies d. A 3-layer transformer decoder then uses the queries to aross-attend into the tokens of the scene representation corresponding to the first input view 2', before performing self-attention among the queries themselves. produce the camera control latent è, the output of the last transformer layer (excluding the output corresponding to the dynamics token) global average pooled and linearly projected to N. = dimensions. To produce the dynamics control latent d, the output corresponding to the dynamics token is linearly projected to N₂ = dimensions.
12

---

Published as a conference paper at ICLR 2024
Pan
Shift
Zoom
Random
Object
Figure 9: Illustration of motions in the DySO dataset. Row 1-4 show examples the used camera motions: panning, shifting. zooming and randomly sampled. Row 5 shows an example of how object dynamics: random noise is added the object's position, and random rotation up to 90 degrees applied to the objects pose.
Decoder. The queries to the decoder are formed by concatenating the latent camera & dynamics d, then adding a 2D positional encoding for each pixel be decoded. The decoder processes each query independently by cross-attending into the scene representation using 2-layer transformer (same configuration in the encoder, but no selffattention). Different from RUST, the queries are linearly transformed to a dimensionality of 768 (instead of 256). Also different to RUST, the outputs of the transformer are mapped to RGB space by a 1-hidden layer MLP with hidden dimensionality 768 (instead of 128).
Training. During training, we always use input views and target views per scene, where we render 8192 pixels uniformly across the target views. On the synthetic DySO dataset, we apply latent control swap, where we sample the 4 target views such that capture exactly 2 distinct cameras and dynamics, and the 3 input views are randomly sampled from the remaining cameras and dynamics. We use a batch size of 256, where each data point correspond to scene. We alternate gradient steps between batches from the DySO and the SSv2 dataset, and train the model end-to-end using the Adam optimizer (with 3 = 0.9, 32 = 0.999, = 10-8) for 4 M steps. We note that the model has converged at that point, and we could see PSNR further improving while training longer. The learning rate is decayed from initially 1 1x10- to 1.6x 10 with an initial linear warmup in the first 2500 steps. We also clip gradients exceeding a norm of 0.1. As in RUST, we scale the gradients flowing to the camera & dynamics estimator by a factor of 0.2.
A.2 DATASETS
DySO.
We implement the DySO dataset based on the MultiShapeNet (Sajjadi et al., 2022b) dataset in the Kubric simulator (Greff et al., 2022). The DySO dataset consist of 1M scenes for training, 100K scenes for validation and 10K scenes for testing. Each scene has a randomly selected ShapeNet object (Chang et al., 2015), and consists of a 5 distinct camera positions observing the object moving for 5 steps. This yields 25 views per scene. Images are rendered with ray tracing and on top of 382 complex HDR backgrounds resolution of 128x 128.
13

---

Published as a conference paper at ICLR 2024
The cameras sampled on trajectories with one of 4 motion patterns: horizontal shifts, panning, zooming, and sampling random camera points nearby a fixed point. See Fig. for an illustration. Furthermore, add small Gaussian noise to all camera positions to make the camera trajectories less predictable. For the object dynamics, for each time step, we randomly jitter the object's position and further apply random rotation up to 90 degrees to the object. Initially, the object is placed the floor with a random position and pose.
SSv2. The SSv2 dataset showcases short videos of humans performing basic actions with everyday objects from an ego perspective. It consists of roughly 170K training videos, 24K validation videos, and 27K testing videos with 12 frames per second. For training, we sub- sample clip of up to 64 frames for each video. As the original videos only have resolution of x00 100 pixels, we upsample them to 128x 28. All examples shown in the paper are from the test split.
14


Reviews:
[{"review_id": "72ac6PQsWq", "reviewer": "Reviewer_whxo", "summary": "The goal of this paper is to estimate motion and object pose and shape from monocular videos using a latent neural representation. \nTwo modules are trained for camera parameter and object position and shape estimations. Then, these modules are then used as input to a third decoder module to generate novel views. To ensure the specialization of the modules, the authors proposed a training procedure where the data is organized to enforce which information is learned by each module. Evaluations are conducted on a newly created dataset which was used to train the model and qualitative results are presented with motion extraction, novel view generation, video manipulations where novel camera motions or object movements are generated.", "questions": "Is the latent dynamic space somehow interpretable, and is it possible to generate one instance based on the object position in space, or does the latent space always be inferred from an existing image contained in the processed sequence ? \nIn the later case, this means the approach cannot be used to recreate dynamics that do not exist yet in the data ?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Compared to the state of the art, this work investigates the more difficult setting of estimating moving objects and moving cameras only from a few motions pictures and a monocular camera. Moreover, they remove the assumptions of training one model for each scene. \n\nThe separation of 3D structure estimation and camera motion is an interesting property of the model. The training tricks illustrated by Eq. 5 and Eq. 6 provide an practical way of enforcing this while still retaining the benefit of end to end training.\n\nAlthough the videos are simple and are still far from the complexity of most real world data,it is still a good compromise as a next step toward more mature systems. Experiments shown in Fig.3 to assess the specialization of the different modules are convincing, it is also supported by the qualitative results shown in the videos in the supplementary material on video manipulation and image synthesis.\n\nExperiments seems reproducible, given code and training parameters, and available datasets will be provided.", "weaknesses": "The amplitude of the motion would probably limits the accuracy of the method. In Fig.7 the motion is tiny, and this is not evaluated by the authors. Although the encoder and decoder architectures are rather small for the \"simple\" cases covered by the paper, I have concerns on the scalability of this method to more real cases and more complex motions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yvG9QDOlfO", "reviewer": "Reviewer_Qe3y", "summary": "- The authors propose Dynamic Scene Transformer (DyST) which makes a model infer the target view’s control latents (camera pose, scene dynamics) with a pair of corresponding views (different camera pose and scene dynamics from target view).\n- Moreover, the authors propose a synthetic dataset, Dynamic Shapenet Objects (DySO), which consists of 5 scene dynamics and 5 camera views for each dynamic scene video to train the DyST.\n- By showing qualitative and quantitative results of the experiments on changing control latents, the authors validate that the DyST learns latent decomposition of the space into scene content.", "questions": "- The latent control swap training scheme needs 3 input views. It would be helpful why 3 input views are needed and how the performance changes with less or more than 3 input views.\n- It would be better if the authors discuss why it needs the contrastiveness metric and what the authors are trying to show with the swap in Table 1.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The authors co-train synthetic and real-world datasets to transfer the dynamics and camera control potential of synthetic scenes to natural monocular video and the results shown in Fig. 5 indicate that the model has learned to encode dynamics independently of the camera pose.\n- Since there is no architectural difference between camera pose and scene dynamics, the authors propose to enforce separation through a novel latent control swap training scheme, and the results in Fig. 3 demonstrate their method with a high improvement in PSNR scores.", "weaknesses": "- [Generalization in different types of motions] Additional experiments are needed to see if the proposed DyST model can generalize to camera poses and scene dynamics that were not seen during training. so, it would be better to provide qualitative results on how the controlled view looks like when horizontal shifts are input after training without horizontal shifts. (DySO’s camera motions consist of 4 horizontal shifts, panning, zooming motions, and random camera points)\n- [Cluttered background] Since the backgrounds of the DySO dataset in Fig. 2 and 3 are clean, the authors need to experiment to see if DyST can robustly control the view even when using videos with cluttered backgrounds. In addition, it would be better to have a distance analysis for unclean scenes to see how distinctly it separates camera pose and scene dynamics like the experiment in Figure 5.\n- [Quantitative comparison] As the authors mentioned in Sec. 5 Conclusion, unlike NeRF's output, the output of the proposed method has a quality gap, such as objects disappearing or blurring. Therefore, quantitative comparison results such as PSNR and LPIPS between NeRF and DyST are needed.\n- [Multiple objects] Also, as mentioned by the authors in the same section, the authors did not provide results for multiple object scenes. It would be helpful to see the results of latent distance analysis in Figure 5, PSNR, and LPIPS in Figure3 for multiple object scenes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hOuy77S4uB", "reviewer": "Reviewer_XJou", "summary": "This paper proposes Dynamic Scene Transformer that learns latent neural scene representations from monocular dynamic video without any pose information. Different from previous works, this paper mainly focuses on modeling the latent space for dynamic scenes. To achieve this, the authors utilized  a Camera Estimator and a Dynamics Estimator to produce the low-dimensional controllable latents for camera pose and scene dynamics. To separate dynamics from camera pose effectively, the author further design a swap training scheme and establish a multi-view, multi-dynamics dataset synthetic dataset.", "questions": "Reproducibility Statement should be put in the appendix.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well-motivated. The primary goal of this paper is the separation of scene dynamics and camera pose, while most of existing works only cover the static scenes.\n2. The authors proposes a novel training scheme that disentangles the camera pose from two views under the same camera while containing a moving object, and disentangles the scene dynamic from two views with still objects while under two different cameras. To fulfill this training strategy, the authors also establish a new synthetic data with multi-view, multi-dynamics data.", "weaknesses": "1. The method is quite similar to RUST[1]. The encoder, decoder and camera estimator are almost the same as the ones proposed in RUST. \n2. Inference procedure. From the method architecture, the target view is required to obtain the camera latents and dynamic latents. In this case,  I wonder if the specific novel view image is needed as the input to generate the novel view?  \n3. Control the latent code. In Fig7, the authors show the results of controlling camera latent and dynamic latent. The authors could explain how to control the latent code.\n4. Some quantitative results on real data should be provided.\n\n\n[1] RUST: Latent Neural Scene Representations from Unposed Imagery. CVPR, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)