Semantic Flow: Learning Semantic Fields of Dynamic Scenes from Monocular Videos
Authors: Fengrui Tian
Yueqi Duan
Angtian Wang
Jianfei Guo
Shaoyi Du
Venue: ICLR 2024 poster (2024)
Keywords: 3D vision
NeRF
semantic understanding
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We propose Semantic Flow for learning semantic fields of dynamic scenes from monocular videos.

Abstract: In this work, we pioneer Semantic Flow, a neural semantic representation of dynamic scenes from monocular videos. In contrast to previous NeRF methods that reconstruct dynamic scenes from the colors and volume densities of individual points, Semantic Flow learns semantics from continuous flows that contain rich 3D motion information. As there is 2D-to-3D ambiguity problem in the viewing direction when extracting 3D flow features from 2D video frames, we consider the volume densities as opacity priors that describe the contributions of flow features to the semantics on the frames. More specifically, we first learn a flow network to predict flows in the dynamic scene, and propose a flow feature aggregation module to extract flow features from video frames. Then, we propose a flow attention module to extract motion information from flow features, which is followed by a semantic network to output semantic logits of flows. We integrate the logits with
volume densities in the viewing direction to supervise the flow features with semantic labels on video frames. Experimental results show that our model is able to learn from multiple dynamic scenes and supports a series of new tasks such as instance-level scene editing, semantic completions, dynamic scene tracking and semantic adaption on novel scenes.

Published as a conference paper at ICLR 2024
SEMANTIC FLOW: LEARNING SEMANTIC FIELDS OF DYNAMIC SCENES FROM MONOCULAR VIDEOS
Fengrui Tian' Yueqi Duan? Angtian Wang" Jianfei Guo Shaoyi Du
'National Key Laboratory of Human--Maching Hybrid Augmented Intelligence, National Engineering Research Center for Visual Information and Applications, and Institute of Artificial Intelligence and Robotics, Xi an Jiaotong University TTinngua University 3 JJhhss Hopkins University 3Shanghai AI Laboratory duanyueqi www.nniuittinnhh..eaeo www.sinnghuuuuuuu.em..mm on t tianfr@stu. xitu. edu. quojianfei@@j1ab www.pprg..........ooooooooo org.cn angt iagaang jhu. edu dushaoyi @xitu, @xjtu.edu.cr enu.cn
ABSTRACT
In this work, we pioneer Semantic Flow, neural semantic representation of dy- namic scenes from monocular videos. In contrast to previous NeRF methods that reconstruct dynamic scenes from the colors and volume densities of individual points, Semantic Flow learns semantics from continuous flows that contain rich 3D motion information. As there is 2D-to-3D ambiguity problem the viewing direction when extracting 3D flow features from 2D video frames, we consider the volume densities as opacity priors that describe the contributions of flow features to the semantics on the frames. More specifically, we first learn a flow network to predict flows in the dynamic scene, and propose a flow feature aggregation mod- ule to extract flow features from video frames. Then, we propose flow attention module to extract motion information from flow features, which is followed by a semantic network to output semantic logits of flows. We integrate the logits with volume densities in the viewing direction to supervise the flow features with se- mantic labels on video frames. Experimental results show that our model able to learn from multiple dynamic scenes and supports series of new tasks such as instance-level scene editing, semantic completions, dynamic scene tracking and semantic adaption on novel scenes.
INTRODUCTION
In recent years, Neural Radiance Field (NeRF) (Mildenhall et al., 2020) methods bring storm on scene reconstruction tasks (Mildenhall et al., 2022: Chen et al., 2021; Yu et al., 2021; Niemeyer & Geiger, 2021; Deng et al., 2022). NeRF represents the static scene as an implicit neural network that takes an input of the position and viewing direction at a point in the 3D space, and outputs the corresponding color and density. The high reconstruction quality facilitates many researchers to extend its applications toward dynamic scenes (Gao et al., 2021; Wu et al., 2022; Park et al., 2021; Weng et al., 2022), which are mostly recorded by monocular cameras in the real world.
Dynamic scene reconstruction from monocular videos is a challenging problem. Since the fore- ground is dynamically changing in the scene, learning dynamic radiance fields from monocular videos suffers from 2D-to-3D ambiguity in the viewing direction when mapping the observed object motions on the 2D images (i.e. optical flows) dense object motions in the 3D scene (i.e., scene flows (Vedula et al., 1999)). Previous works mainly address the challenge and render the scene by designing spatio-tempont
constraints (Peng et al., 2021; Qiao al., 2022: Li et al., 2021; Gao al., 2021; Tian et al., 2023). Specifically, they assume that points are temporally moving in the scene, and reconstruct the scene with consistency constraints along point trajectories.
While the aforementioned dynamic NeRF methods successfully build radiance fields for dynamic scenes, the semantics these fields remain underexplored. Learning the semantic information in
"Corresponding author, Codes are available https:: thub. /Semant c- c-Foow/

---

Published as a conference paper at ICLR 2024
Input Scenes
Semantic Field
Instance-level Scene Editing
Dynamic Tracking
Semantic View Synthesis Unseen Frames
Semantic Completion
Unseen
Frames
Frames
Novel Scene
Semantic Adaption Novel Scane (500 finetuning steps)
Ground
DynNoRF+head
MonoNeRF
Figure I: Semantic Flow learns from flows capturing motion information in dynamic scenes. this way, Semantic Flow can learn semantics from multiple scenes and conduct instance-level editing (Top). It also supports dynamic scene tracking and semantic completion (Middle), both of which learn with few semantic labels. Compared to DynNeRF (Gao et al., 2021) and MonoNeRF (Tian et al., 2023), Semantic Flow can transfer to novel scenes with more accurate details (Bottom).
dynamic scenes considered beneficial since such information could support the further inter- pretability and applications of the reconstrueted scenes. The potential information in the flow of each moving object is important for many computer vision tasks such as trajectory prediction (Xu al., 2023), action recognition (Feichtenhofer al., 2019) and other applications (Trucco & Plakas, 2006; Tsai et al., 2016; Alahi et al., 2016; Hu et al., 2023).
To learn semantics in dynamic radiance fields, an intuitive solution is to add a semantic segmentation head to the previous dynamic NeRF methods. In this way, the semantics of each point is estimated from the position, timestamp, and other information related the point. Although such approach is sufficient for learning color and density, it cannot provide the motion information related to the scene. In other words, the motion of an object can be observed from continuous flow over time rather than single point at each moment. Due to the lack of motion information, predicting semantics from points forces the model to overfit to the training views the current scene, which limits its generalization performance when training with few annotated labels transferring novel scenes. Such limitation heavily restrains the model from wider range applications in the world.
In this paper, propose neural semantic representation for dynamic scenes. Instead of learning from separate points, we propose to learn from continuous flows that capture motion information of scenes. While extracting 3D flow features from 2D frames suffers from 2D-to-3I ambiguity problem, we consider the volume densities features the semantic labels on frames, so that the 2D semantic labels could be used to supervise the semantic field by integrating the flow features in the viewing direction.
opacity priors describing the contributions of flow
To achieve this, propose Semantic Flow for building semantic fields of dynamic scenes from flows. We first design flow network to predict flows in the dynamic scene. Then, we exploit the locations of the points on the flows as indexes to extract local image features from video frames, and aggregate the extracted local features and flow displacements as flow features. We design flow attention module to fully exploit motion information from flows and employ a semantic network to output the semantic logits of each flow. We aggregate the semantie logits in the viewing direction with volume densities and supervise the logits with semantic labels on the video frames.
In addition, we also develop dataset called Semantic Dynamic Scene Dataset based on the dy- namic scenes from the Dynamic Scenes dataset (Yoon et al., 2020) to conduct semantic tasks. In dataset, there seven forward-fan farrid dynamic scenes with complex foreground motions. We manually annotate the pixel-wise semantic labels each video, and conduct a series of experiments the dataset. As shown in Figure 1, Semantic Flow learn from multiple dynamic scenes, and
2

---

Published as a conference paper at ICLR 2024
present strong generalization ability various tasks such semantic adaption novel scenes, semantic completion, and dynamic scene tracking.
2 RELATED WORK
NeRF for semantics. In recent years, NeRF achieves great progress on the novel view synthesis task by representing scenes as neural implicit representations (Mildenhall et al., 2020; Liu et al., 2020; Gafni et al., 2021; Yu et al., 2021; Niemeyer & Geiger, 2021; Gu et al., 2022; Zhi et al., 2021; Martin- Brualla et al., 2021; Srinivasan et al., 2021). Such success in scene reconstruction tasks encourages many researchers to explore its possibility in scene understanding tasks. The first attempt is the Semantic NeRF (Zhi et al., 2021), which exploits many semantic tasks in the neural radiance field by adding a semantic head to the origin NeRF pipeline, Based on that, recent works focus on the semantic understanding of static scenes (Yang et al., 2021; Liu et al.. 2023a; Vora al., 2022; Fan et al., 2023; Kundu et al., 2022). Liu et al. (2023a) studied the generalization ability of semantic radiance field by introducing semantics into each ray. Vora al. (2022) proposed NeSF that is able to reason semantics from geometry stored in the radiance field. Fan et al. (2023) proposed NeRF-SOS that studies the segmentation problem in self-supervised way. Although above mentioned methods achieve promising performance, semantic learning in dynamic scenes remains underexplored.
Dynamic NeRF from monocular videos. The monocular videos are commonly captured by per- sonal phones in daily life. Hence many researchers make huge progress on building the dynamic radiance field from monocular videos (Li et al., 2021; Gao et al., 2021; Park et al., 2021; Pumarola et al., 2020; Peng et al., 2021; Qiao et al., 2022: Ost et al., 2021: Weng et al., 2022; Gafni et al., 2021). The major challenge behind scene reconstruction from monocular videos is the ambiguity problem, which means the same observed image sequences can be inferred from different scene reconstructions. Initially, some works extend the vanilla NeRF to dynamic scenes by introducing time dimension (Li et al., 2021; Gao al., 2021; Pumarola al., 2020). Based on that, series of studies try to reconstruct the dynamic parts of the scene more precisely with shape priors (Peng al., 2021; Qiao et al., 2022: Ost et al., 2021; Weng al., 2022: Gafni et al., 2021). Besides, Tian et al. (2023) studied the generalization problem and proposed MonoNeRF which builds a general- izable dynamic radiance field by jointly optimizing the spatial and temporal features. In this study, we explore the dynamic radiance field with semantic representations.
3 SEMANTIC FLOW
In this section, we introduce our model which we term as Semantic Flow. Given a monocular video I containing N frames (4,, ..,.., IN} with known camera poses {P1, P2....... .2......,, PN we denote 4,, Pt as the frame and pose at timestamp t and hence t foregrounds in each video frame, and reconstruct semantic fields of the dynamic foreground and static background separately. We follow previous works (Li et al., 2021; Gao et al., 2021; Tian et al., 2023) and supervise the model with optical flow and depth signals, which can be readily obtained from the state-of-the-arn pretrained models (Teed & Deng, 2020; Ranftl et al., 2022). For the semantic field of dynamic foreground, we first build a flow field based on the input video to calculate the flow of each point in the scene. Based that, exploit the moving points the flows as indexes to locate local image patches on each frame and extract the local image features. We combine the local features and flow displacements flow features and design flow attention module to uncover the semantic information related to the motion information of flows. To supervise the semantic field with labels on video frames, we employ semantic network output semantic logits of each flow, and render the logits of each pixel on the frames by integrating the semantic logits of flows in the viewing direction with volume densities. For the semantic field of static background, we directly sample the semantic features of each point from video frames. Our pipeline is shown in Figure 2. Before introducing our model, we first present a review of concurrent dynamic radiance fields.
NN. We use semantic labels to delineate
3.1 PRELIMINARIES
To build a neural radiance field for dynamic scenes, previous works (Gao et al., 2021; Li et al., 2021; Tian et al., 2023) build static and dynamic radiance fields for reconstructing backgrounds
3

---

Published as a conference paper at ICLR 2024
Encoder Fay.
Flow Network
v(-)
Predicted Flow t= ((0b,))
Monocular Video
Frame-wise Feature
Flow Embedding
Prediction
Semantic Network
Semantic Rendering
Fay (pnr)
dy
Fay Ct: )
Flow Feetures
Flow Attention
Pooling
Geometry
Fay (l: 1
Network
Flow Feature Sempling
The overview of the proposed model. We first design a flow network to predict flows the dynamic scene. Then, taking the orange flow (in the bottom left part) as an example, we aggregate the flow features from the feature map of each frame. We propose the flow attention module to reveal the motion information from the aggregated flow features. Finally, we design a semantic network to output the semantic logits of each flow, and predict semantics the frame by rendering the semantic logits along camera rays with volume densities ddy as opacity priors.
Figure 2:
and foregrounds separately and blend static and dynamic radiance fields to render the entire scene with learned blending weights. Specifically, for static radiance field, they learn an implicit function af : (a Car) = V (x)) that converts the input coordinate æ R to the corresponding color E ig and density det E R. Then they exploit the volume rendering technology to synthesize novel view images of static backgrounds. Given a ray r = o + ud emitting from the camera center R along the direction d IR3 through pixel on the image, the volume rendering function can be formulated as
Cae(r)= 24 (" Ta(u)oau(W)ssuu))
(1)
wherein Un, uy are the upper and lower bounds of the rendering depth range and (u) exp(- S", (s)()) is the accumulated transmittance. We simplify c(u) c(r(u)) and (u) (r(u)). For the dynamic radiance field, takes the position æ and time as input and learns an im- plicit function Mayy (a Cat, b) V dy (æ, f), where b is the blending weight that decides whether the point belongs to the dynamic foreground (Tian et al., 2023). Similarly, novel view images of dy- namic foreground could be also synthesized by (1). The static and dynamic radiance fields V st, V da collaboratively render the novel view images of the entire scene.
S.m. ("
(rul(())
(wull)) (u)
In this paper, we follow the principle to separately reconstruct the semantic fields of dynamic foreground and static background.
where full (fu((u)) (w) (1 (u) + beray
3.2 SEMANTIC FIELD FOR DYNAMIC FOREGROUND
In this section, we introduce our semantic field for dynamic foreground. We first describe our implicit flow field which generates the flows in the dynamic scenes. Then, introduce the flow feature aggregation module and flow attention module to predict semantics of each flow. In the end, present semantic rendering method.
Implicit flow field. We build a flow field to trace the flow on each point. Concretely, we first extract the video features Fag Edy (T) by video encoder Edy. Then, build flow field with a multi- layer perceptron (MLP) V stow based on Fdy- Given a point position x at timestamp t, the flowing point positions Φ Pc+1 E 3 at timestamps 1, + 1 can be calculated as
(1,,1, P++1) flow (Fayi $ 1),
(3)
4

---

Published as a conference paper at ICLR 2024
where L, = Then, the point trajectories p Pirr at timestamps t - r,, + could be inferred
from (3) by introducing P--r41, Pi+r-1 and t- r +, r- into V fiow. In this way, we can derive the flow related to point set of point positions with timestamps r = {T I2, 2,..... I where r = (.,,)). We follow previous works (Tian et al., 2023; Gao et al., 2021; Li et al., 2021) and employ optical flows the supervision signal. Concretely, for a point timestamp t, we obtain its flow with y flow- We estimate the optical flow by integrating the flow displacements of the points on the ray with densities da. The estimated flows are then supervised by the flows generated from the pretrained model (Teed & Deng, 2020). In the following, we build the flow features based on the flow of each point.
r(u) the ray at
Flow feature aggregation. Given a point with its position timestamp t and flow T. we sample the flow features from video features over the flow of the point. Specifically, for each r, (#;, (). we employ the camera pose P. to project the & onto the video frame I, as P. We extract the feature map of I, by Eay and exploit (+), as index to sample the point feature vector on the feature map with bilinear interpolation,
Fdy (I; ) =
(4)
where the function ((-) is the bilinear interpolation. Edy (I)) denotes the frame- wise feature map extracted by Edg- The flow features vector at any timestamp T could be built based on the point feature vector with the relative flow displacement AT = r, I,
TT) (Fay (I; Φ (AT)),
(5)
where the function 7(-) denotes the position embedding function. The initial flow features defined as
Fdy (I; = [Fay Fay() (I:
(6)
In the following, we introduce our flow attention module for uncovering the motion information of the flow.
Flow attention. As the flow features sampled from video features provide specific information for object motions from different views and timestamps, we propose a flow attention module to reason the semantic features related to flow motions. Concretely, for each flow feature (T;T)) our flow attention module is defined as
Q Fay (1;1) X Wo, = Fay(1;T) x WE, V = (1y x W.,
K())T v(h)), h (1,..., H),
A (h) softmax
Vdx
where de = C/H is the dimension of each head. Q,K, V E RCxC denote the query, key and value features extracted from (Ty by the fully connected layers Wq. We, W.. We employ multi-head attention module and Q [Q(), Q(2),... Q(H)), ,.K = [K(], K(2), ..., K(H),, V [V(1), (22),... ..., [(]] where (()), (M),, (h)) Nxd, A(h) is the output from h-th head. We exploit a semantic network implemented by MLP v. to obtain the final semantic feature vector from the outputs of all heads (A(1) ( "..,...... (N)).
Fay (1:1) (A(A (2
A(M).
(8)
wherein Faem (I:T) denotes the final semantic feature vector of the flow.
Semantic rendering. As there is ambiguity in the viewing direction when estimating precise se- mantie labels in 3D space from the semantic labels on 2D video frames, propose to supervise the semantic field by aggregating the semantics along the camera ray with the corresponding vol-
dy 8dy (T) ursem [Fay (I; r)) implemented by an MLP generate the semantic logits Say (T) of the flow. Given a ray r starting from the camera center through a pixel on the image, for each point r()) on the camera ray, we find its flow in the scene and calculate the corresponding semantic logits of the flow. The semantic logits of the ray are calculated by the following integration,
ume densities as opacity priors. Concretely, we design semantic network Teem.
Sdy (r)
Tay (u))ody (I (Tu
(9)
5

---

Published as a conference paper at ICLR 2024
where ddy (u)) denotes the semantic logits of point r(u) obtained from it flow feature vector. We simplity Say (T) = sdy (T(r()))) here. ddy is predicted by our geometric network V geo- We
supervise the rendered labels with ground truths by employing multi-class crossentropy loss:
Csem dy
[
(r) -r)=--
(r (r)}
(10)
1=1
wherein Play.?' p' are the semantic probabilities of class I from the prediction and ground truth map, respectively. is the total number of classes and 1 < <1 1 L.
3.3 SEMANTIC FIELD FOR STATIC BACKGROUND
In this section, introduce semantic field for the static background. Since some parts of the background may be occluded by the changing foreground, the semantic features extracted from video frames in these parts always imply the foreground information. To obtain the correct semantic features of occluded background parts, we choose to extract the features from non-occluded views as the following equation,
(11) where Est is the encoder of the static scene. 1 P denote the non-occluded frame and correspond- ing camera pose. Following Tian et al. (2023), strategy by effectively choosing one frame in the video. Then, the semantic logits in the static scene represented by MLP Vreem (a))
(T: = 92 (sr P*)x))),
employ the straightforward random sampling
- (Fst"(FF (Fs+(1:))), Similar to the semantic field for dynamic foreground described in Section 3.2, for each point r(u) on the ray, we exploit volume densities to render the semantic logits of the ray in the static scene,
(r)
=
(12)
where (rr(r( denotes the semantic logits of the point r(u). We supervise the logits by crossen- tropy loss:
L (
(r) - Σ (rood
(13)
1=1
where Phe is the predicted semantic probability of 1-th class in the static scene.
3.4
OPTIMIZATION
During the training phrase, we first pretrain the semantic field of the static background by using the semantic labels that belong the background, and then optimize the semantic field of the dynamic foreground combined with the pretrained static background field to render the entire semantic field,
sem full (T)
| Tuun(u)o yuu you (u)s (r())rr Un
(14)
where fuur (fut(())) (f) (u) (1 - (m) (r(u) + (u) (T(u)). b denotes whether the point belongs to the dynamic foreground or static background. The semantic loss of the entire scene is defined as
(.) = = Σ Σ
En' (r)oo))aarr))
[ 1=1
(15)
where f/yw is the predicted semantic probability of class l. In the following, we apply several optimization strategies to the model for rendering the semantic field with better quality.
Semantic consistency constraint. We suppose that the semantics of flows are consistent over time across the whole dynamic scene. Specifically, employing the flows of the points on the ray, i.e., (u) (r(w)),, where (r(w)) denotes the new position of ruu) when moving from the original timestamp to timestamp We supervise the semantic label of r, with the ground truth label of the ray r.
denotes the warped ray r at timestamp T by
Lconsis = - (T))
(16)
E=1
where (r+) is 1-th class semantic possibility of the ray r..

---

Published as a conference paper at ICLR 2024
Ground Trush
DynNeRF heed
MonoNeR Feehheea head
Sementic Flow
(a) Learning semantics from multiple dynamic scenes.
Truck
Ground Truth
DynNeRF+head
MonoNeRF+hend
Sementic Flow
Umbrella
(b) Semantic adaption on novel scenes (500 finetuning steps).
Bie Tosh1
Ground Trulh
DynNeRFF+haad
MorioNeRF +heea
Semantic Flow
Beloont
(c) Semantic completion (50% semantic labels). train the models with semantic labels video frames (J,, Is, J1o, In1, }12} and predict novel semantic views on the rest frames (Ia, Js.... Io].
Balloona
Ground Truth
DynNeRFthest
Mo thead
Semantic Flow
Skittigg
(d) Dynamic scene tracking (25% semantic labels). We use the semantic labels of frames {41 12, 1st to train the models, and conduct semantic predictions of novel views the frames {14, 1s (12).
Figure 3: Visualizations on various tasks. Different from DynNeRF (Gao et al., 2021) ( semantic head and MonoNeRF (Tian et al., + head that learn from point features, Semantic Flow learns from flow features for capturing motions. In this way, Semantic Flow predicts semantic labels of dynamic foregrounds with more accurate motions and clearer boundaries.
EXPERIMENTS
We evaluate our method by conducting experiments on various semantic tasks of dynamic scenes. We first introduce our new dataset and the implementation details, Then, we present experiments by training with full annotated labels. After that, we tested the performance on training with few labels.
4.1 DATASET AND IMPLEMENTATION DETAILS
Dataset. We introduce Semantic Dynamic Scene dataset which is built upon the Dynamic Scene dataset (Yoon et al., 2020). Dynamic Scene dataset contains 9 dynamic scenes captured by 12

---

Published as a conference paper at ICLR 2024
Table 1: Performance on semantic representation learning from multiple scenes and semantic adap- tion of novel scenes (500 finetuning steps). @ represents adding a semantic head to the original model. By learning from flow features, our model could reconstruct semantic fields and transfer to novel scenes with the state-of-the-art performance. More comparisons are in Table 9 in Appendix.
Learning semanties from multiple scenes Rallopn2
Semantic adaption of novel scenes Playground
Total Acch / mloU
Ralloon
Jumping
Skating
Umbrella
Truck
DynNeRF (Gao al, 2021)
0.767 /05515 0.459 / 0.229
0,912 0.550
0.955 /0.431
0,941 0.795
0.914/ 0.387
0.967 0.662
MonoNeRH (Tian et al., 2023)
0.907 70,760
0,967 / 0.616
0.929 / 0,576
0.973 / 0.590
0.961 / 0.685
0.879 0.393
0.968 0.425
Semantic Flow
0.919 0.844
0.967 / 0.839
0.938 0.703
0.970 0.608
0.970/ 0.884
0,937 00742
0,977 0.765
Table 2: Quantitative results on scene completion and dynamic scene tracking. à denotes that we
added a semantic head to the original model. The results show the superiority of our model when learning from with few labels. Detailed results are shown in Table 10 and Table 11 in Appendix.
Completion (50% labels)
Tracking (25% labels)
method
Total Acct
Avg Ace"
mloUt
Total Accf
Avg Acct
mloU?
DeAOT (Yang Yang, 3022)
0.816
0.600
0.412
0.776
0.533
0,359
DynNeRP9 (Gao et al., 2021)
0.934
0.849
0.738
0,896
0.760
0.660
MonoNeRF (Tian et al., 2023)
0.956
0.891
0.786
0.935
0.818
0,716
Semantic Flow
0.961
0.901
0.818
0.942
0.835
0.767
cameras with a camera rig. We manually annotate 7 dynamic scenes: Balloonl, Balloon2, Jumping, Skating, Playground, Truck and Umbrella. More details refer to Section D in Appendix.
Implementation details. We follow previous works (Yu et al., 2021; Liu et al., 2023a; Tian et al., 2023) to use ML.ss with residual links as our flow network V ftow and geometry network V geo- The semantic networks Vdem. V are implemented by three fully connected layers with ReLU activation. For the semantic field of dynamic foreground, we employ SlowOnly (Feichtenhofer et al., 2019) pretrained on Kinetics-400 dataset (Carreira & Zisserman, 2017) as our backbone encoder Edy with the frozen weights. We remove the final connectdd layer and combined the first, second, and third feature layers as the output of Edy (1)). We simplify Equation (6) where we only sample the flow feature vector F(T) timestamp from frames (4--1, [,, 1++1). We train the semantic field of dynamic foreground for 40,000 iterations with Adam optimizer (Kingma & Ba, 2015). For the semantic field of static background, we employ ResNet18 (He et al., 2016) pretrained on ImageNet (Deng et al., 2009) as our backbone encoder. We pretrain the semantic field of static foreground for 100,000 iterations. The learning rate is set x 10-4, More details refer to Section E in Appendix.
4.2 SEMANTIC VIEW SYNTHESIS WITH FULL LABELS
In this section, we evaluate our model with fully annotated labels. We first test the ability of se- mantic representation by learning from multiple scenes. Then, conduct ablation studies and instance-level scene editing application based on the learned representation. Finally, we evaluate the generalization ability by finetuning our model on novel scenes.
Learning semantics from multiple scenes. Similar to Tian et al. (2023), we train our model on two scenes simultaneously: 1) BalloonI and Balloon2; 2) Jumping and Skating. To compare with other state-of-the-art methods, we reimplement DynNeRF (Gao et al., 2021) and MonoNeRF (Tian al., 2023) from their official implementation and add a semantic head to their models for predict- ing semantics. As shown in Figure 3a, because DynNeRF learns the semantic field from position embedding, has limited generalization ability cross scenes, and hence the semantic information of two scenes is mixed together. As for MonoNeRF, due to the lack of motion information, the boundaries of dynamic objects are difficult to predict. Table 1, since the mloU matrix sensitive to boundary accuracy, Semantic Flow outperforms the above two methods with large margin.
Ablation studies. We conduct a series of ablation studies on Figure and Table by learning semantics from multiple scenes. It can be seen that with the proposed flow attention module, our model successfully extracts motion information in the flows and improves the performance on mloU matrix. dy provides important prior for integrating the semantic logits of dynamic objects. Our model renders the semantic field with finer details by using Leonsist- seem and Al contribute to reconstructing the semantics in the foregrounds.
8

---

Published as a conference paper at ICLR 2024
Table 3: Label percentages (Total Acct / mloUf).
Table 4: Flow displacements (Total Acct mloU?).
Table 5: Finetuning steps on
novel scene. o semantic head.
Labels
Completion
Tracking
Disp.
Complction
Tracking
Ft stops
MonoNeRF
Semantic Flow
25% 0.979
0.904 0.975 0.893
0.984 / 0,925 0.975 /0.893
100
0.819 0.296
0.906 0.689
50%
0.983 0.920 0.979 0.998
r
0.983 / 0,920 0.974 / 0.894
200
0.869
0.316
0.925 0.736
75%
0.985 0.934 0.985 0.934
T&AT 0.985 / 0.932 0.970 80.874
500
0.879 0.393
0.937 0.742
Table Numeric compar- isons dy prior, flow atten- tion module, Usem Loonsist and AT.
w/o, By prior
w/o
attn
w/o.
w/o. Ar
w/o. yssem
Acct / mloUf
day prior
flow
Loonsts AI
w/o, ddy prior
0.903 / 0.838
w/o. flow attn
0.921 / 0.643
wfo. grew
0.853 / 0,734
w/o. heonsist
0.917 / 0.741
w/o. AT
0.915 0.770
Figure 4: Ablation studies on Ody prior, flow attention module,
Semantic Flow
0.932 / 0.859
Lconsiet. AT and Visem
Instance-level scene editing, Our model could conduct instance-level scene editing applications with the learned semantic field. As shown in Figure our model can remove the dynamic instances in the foreground by forcing the volume densities of points related to target instances to zeros.
Semantic adaption on novel scenes. After training on multiple scenes, we finetune our model on unseen monocular videos to test the generalization ability on novel scenes. Specifically, we pre- train our model on Balloon1 and Balloon2 scenes with 10,000 training iterations and finetune the model on Umbrella, Playground and Truck scenes separately. We also evaluate the performance of DynNeRF (Gao et al., 2021) and MonoNeRF (Tian al., 2023) with the same experiment set- tings for fair comparison. As shown in Figure 3b, since model learns semantics from flow features, could predict more accurate boundaries of moving objects in dynamic foreground by capturing motion information. In contrast, the predictions from DynNeRF and MonoNeRF methods show blurring and even wrong boundaries, which causes many false positive predictions outside the performance drop mloU matrix in Table and Table 5.
moving objects and leads to
4.3 SEMANTIC VIEW SYNTHESIS WITH FEW LABELS
Since manually annotating semantic labels is a time-consuming and laborious task, in this sections, we introduce two strategies to evaluate the generalization ability of our model with few labels: (1) Semantic completion. We apply models to learn the semantic representation over the entire scene with the semantic labels in the first and last few frames {I1, Jz, Iz. T10. J11, [12]. }. (2) Dynamic scene tracking. We train on the front video frames {T,, I2, Iz] with fully annotated labels and directly fer the semantics over the rest frames. As shown in Figure 3c and Figure 3d, since DynNeRF builds the semantic field from position embedding, it cannot transfer unseen frames with accurate seman- tic predictions. As MonoNeRF predicts the semantic labels with point features, it meets difficulties in predicting the semantic labels of the moving balloons in the foreground. In Table 2, with the extracted flow features, Semantic Flow is easier to identify the dynamic parts and hence presents higher performance. Table and Table show that our model could extract motions in flows from various types of displacements and build semantic fields by using 25% labels.
CONCLUSION
In this paper, we build a semantic field dynamic scenes from monocular videos. We propose to leam from flow features that contain motion information and consider the volume densities as opac- ity prior for supervising the field with semantic labels on video frames. Experiments demonstrate the effectiveness of our model.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was supported by the National Key Research and Development Program of China under Grant No. 2020AAA0108100 the National Natural Science Foundation of China under Grant Nos. 62206147, 62327808 and 62088102. The authors also would like thank Dong Zhang, Zongwei Zhou, Wenxuan Li, Yaoyao Liu and Chuanruo Ning for their thoughtful suggestions.
REFERENCES
Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei-Fei, and Silvio Savarese. Social LSTM: Human trajectory prediction in crowded spaces. In CVPR, pp. 961-971, 2016.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In CVPR, pp. 6299-6308. 2017.
Anpei Chen, Zexiang Xu, Fugiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su. MVSNeRF: Fast generalizable radiance field reconstruction from multi-view stereo. In ICCV, pp. 14133, 2021.
Jia Deng. Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, pp. 248-255, 2009.
Kangle Deng, Andrew Liu, lun-Yan Zhu, and Deva Ramanan. Depth-supervised NeRF: Fewer views and faster training for free. In CVPR, June 2022.
A. Dosovitskiy, P. Fischer, E. 1lg, Häusser, C. Hazubaa, V. Golkov, v.d. Smagt, D. Cremers, and T. Brox. Flownet: Learning optical flow with convolutional networks. In ICCV, 2015.
Zhiwen Fan, Peihao Wang, Yifan Jiang, Xinyu Gong, Dejia Xu, and Zhangyang Wang. NeRF-
SOS: Any-view self-supervised object segmentation complex scenes. In ICLR, 2023, URL https: wwwpeeevee.eeeeeeeeeeee/ee.eeewe net / forum? (etrummid=xfffttoo dd=
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. SlowFast networks for video recognition. In ICCV, pp. 6201-6210. 2019. doi: 10. 10.1109/CC.22009.00000
Guy Gafni, Justus Thies, Michael Zollhöfer, and Matthias Nießner. Dynamic neural radiance fields for monocular 4d facial avatar reconstruction. In CVPR, pp. 8649-8658, June 2021.
Chen Gao, Ayush Saraf. Johannes Kopf, and fia-Bin Huang. Dynamic view synthesis from dynamic monocular video. ICCV, 2021.
Jiatao Gu, Lingjie Liu, Peng Wang, and Christian Theobalt. StyleNeRF: A style-based 3D aware generator for high-resolution image synthesis. In ICLR, 2022.
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, pp. 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollar, and Ross Girshick. Mask r-cnn. In ICCV, Oct 2017.
Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tian- wei Lin, Wenhai Wang, Lewei Lu, Xiaosong Jia, Qiang Liu, Jifeng Dai, Yu Qiao, and Hongyang Li. Planning-orienter autonomous driving. In CVPR, 2023.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Abhijit Kundu, Kyle Genova, Xiaoqi Yin, Alíreza Fathi, Caroline Pantofaru, Leonidas Guibas, An- drea Tagliasacchi, Frank Dellaert, and Thomas Funkhouser. Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representation. In CVPR, 2022.
Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-tima time view synthesis of dynamic scenes. In CVPR, 2021.
10

---

Published as a conference paper at ICLR 2024
Fangfu Liu, Chubin Zhang, Yu Zheng, and Yueqi Duan. Semantic Ray: Learning generalizable semantic field with rrosscrenoooecttoo attention. In CVPR, 2023a.
Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. In NeurlPS, volume 33, 2020.
Yu-Lun Liu, Chen Gao, Andreas Meuleman, Hung- Yu Tseng, Ayush Saraf, Changil Kim, Yung- Yu Chuang, Johannes Kopf, and Jia-Bin Huang. Robust dynamic radiance fields. In CVPR, 2023b.
Ricardo Martin BartinnBrualla, Noha Radwan, Mehdi S. M. Sajjadi, Jonathan T. Barron, Alexey Dosovit- skiy, and Daniel Duckworth. NeRF in the Wild: Neural radiance fields for unconstrained photo collections. In CVPR, 2021.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
Ben Mildenhall, Peter Hedman, Ricardo Martin Brruala,,, Pratul P. Srinivasan, and Jonathan Bar- ron. NeRF in the dark: High dynamic range view synthesis from noisy raw images. In CVPR, pp- doi: 10. WWW./CCVPR52288.22222000
16169-16178, 2022.
Michael Niemeyer and Andreas Geiger. GIRAFFE: Representing scenes as compositional genera- tive neural feature fields. In CVPR, pp. 11453-11464, June 2021.
Julian Ost, Fahim Mannan, Nils Thuerey, Julian Knodt, and Felix Heide. Neural scene graphs for dynamic scenes. In CVPR, pp. 2856-2865, June 2021.
Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Steven M. Seitz, and Ricardo Martin- Martin-Brualla, Nerfies: Deformable neural radiance fields. ICCV, 2021.
Sida Peng, Yuanqing Zhang, Yinghao Xu, Qianqian Wang, Qing Shuai, Hujun Bao, and Xiaowei Zhou. Neural Body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans. In CVPR, 2021.
Albert Pumarola, Enric Corona, Gerard Pons-Mol and Francesc Moreno-Noguer. D-NeRF: Neural Radiance Fields for Dynamic Scenes. In CVPR, 2020.
Yi-Ling Qiao et al. Neuphysics: Editable neural geometry and physics from monocular videos. In NeurlPS, 2022.
René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE TPAMI, 44(3):1623-1637, 2022.
Pratul P. Srinivasan, Boyang Deng, Xiuming Zhang, Matthew Tancik. Ben Mildenhall, and Jonathan T. Barron. NeRV: Neural reflectance and visibility fields for relighting and view synthe- sis. In CVPR, 2021.
Zachary Teed and Jia Deng. RAFT: Recurrent All-Pairs field transforms for optical flow. In Andrea Vedaldi. Horst Bischof, Thomas Brox, and Jan Michael Frahm (eds.), ECCV. pp. 402-419, 2020.
Fengrui Tian, Shaoyi Du, and Yueqi Duan. MonoNeRF: Learning generalizable dynamic radiance field from monocular videos. In ICCV, 2023.
Emanuele Trucco and Konstantinos Plakas. Video tracking: concise survey. IEEE Journal of oceanic engineering, 31(2):520-529, 2006.
Yi-Hsuan Tsai, Ming-Hsuan Yang, and Michael J Black. Video segmentation via object flow. In CVPR, pp.
2016.
S. Vedula, S. Baker, P. Rander, R. Collins, and T. Kanade. Three-dimensional scene flow. In ICCV, volume 2, pp. 722-729 vol.2, 1999, doi: (A110//CCVV.1999999222
Suhani Vora, Noha Radwan, Klaus Greff, Henning Meyer, Kyle Genova, Mehdi S. M. Sajjadi, Eti- enne Pot, Andrea Tagliasacchi, and Daniel Duckworth. NeSF: Neural semantic fields for general- izable semantic segmentation of 3d scenes. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL https: //oppe/eviieeeeeeeeeeeeeeeeeeeee net / hwtp:/nettoouum///// forum? dwwpmm/scdggggpcc.m.mmmmm

---

Published as a conference paper at ICLR 2024
Chung- Yi Weng, Brian Curless, Pratul P. Srinivasan, Jonathan T. Barron, and Ira Kemelmacher- Shlizerman. HumanNeRF: viewpiit rendering moving people from monocular video. In CVPR. pp. 16210-16220, June 2022.
Tianhao Wu, Fangcheng Zhong, Andrea Tagliasacchi, Forrester Cole, and Cengiz Öztireli. D2NeRF: Self-supervised decoupling of dynamic and static objects from a monocular video. In NeurIPS, 2022.
Yi Xu, Armin Bazarjani, Hyung-gum gun Chi, Chiho Choi, and Yun Fu. Uncovering the missing pattern: Unified framework towards trajectory imputation and prediction. In CVPR, pp. 9632-9643, 2023.
Bangbang Yang, Yinda Zhang, Yinghao Xu, Yijin Li, Han Zhou, Hujun Bao, Guofeng Zhang, and Zhaopeng Cui. Learning object-compositional neural radiance field for editable scene rendering. In ICCV, October 2021.
Zongxin Yang and Yi Yang. Decoupling features in hierarchical propagation for video object seg- mentation. In NeurlPS, 2022.
Jae Shin Yoon, Kihwan Kim, Orazio Gallo, Hyun Soo Park, and Jan Kautz. Novel view synthesis of dynamic scenes with globally coherent depths from monocular camera. In CVPR, June 2020.
Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelNeRF: Neural radiance fields from one few images. In CVPR, 2021.
Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and Andrew Davison. In-place scene labelling and understanding with implicit scene representation. In ICCV, 2021.
12

---

Published as a conference paper at ICLR 2024
A SUPPLEMENTAL VIDEO
We recommend readers to watch our supplemental video for more visual comparisons.
B CODES AND DATASET
The implementation codes of our model and our Semantic Dynamic Scene dataset are available at https: / comm////ttttt//tttttttoooooo
C
LOSS
Similiar to MonoNeRF (Tian et al., 2023), we use RGB loss and optical flow loss to supervise the static and dynamic radiance fields. Let L L denote the RGB color loss of static and dynamic field separately, L Lggb radiance fields, and Lopt denote the optical flow loss. The entire loss of the model is defined as
full denote the RGB color loss when jointly optimizing the static and dynamic
rgb rrgh rgb r.gb L 0 at se ody Ddy + fuil Luit
rgb rgb. full
+ Chopt Loop + of
sem sem full L
sem sem full + O Ly
(17)
+ at L + Aconsist Leonsists
where the hypar-parameters of each loss is listed in Table 7.
Param
rak
Ody
a
Gopt
aJull
Ody
acm
Oconsiet
Value
4
1
0.02
0.16
0.08
0.08
0.01
Table 7: Hyper-parameters of loss.
D ANNOTATION DETAILS
from Dynamic Scene dataset (Yoon al., 2020). For each scene, we classify two different types of semantic labels: foreground labels and background labels. All the foreground semantic labels in one scene formulate the foreground mask of the scene. The label details of each scene is shown in Table 8. The examples of annotated images are shown in Fig- ure 5, We follow previous works (Tian al., 2023; Gao al., 2021) and use video frames from different cameras to simulate the camera movement. All the cameras capture images at 12 different timestamps. During the training, each monocular video contains 12 frames, where the (-th frame sampled from (-th camera at time I.
We annotate dynamic
Table 8: Scene labels in our dataset.
Scene
Foreground labels
Background labels
Jumping Skating
personl. person2. person3, person4,
bag, lamp, ground, window, rellwall, gray_wall lamp, ground, window, wall, gray_wall
person, skate
Truck
truck umbrella
car, ground, building
Umbrella
wall, plants, ground red. wall grey wall, newspaper red wall grey wall, window gym device, ground
Balloon1
person, balloon
Balloon2
person, balloon
Playground
person, balloon
E IMPLEMENTATION DETAILS
In this section, we introduce the details of our Semantic Flow and experimental settings. The entire model is trained on a Nvidia RTX 3090 GPU with a total batch size of 1024 rays. The learning rate is 0.0005. We used Adam optimizer (Kingma & Ba, 2015) where betas is (0.9, 0.999). During the training. we first pretrain the semantic field of static background, and optimize the semantic field of dynamic foreground with the pretrained static semantic field.
13

---

Published as a conference paper at ICLR 2024
Figure 5: Annotation examples in Semantic Dynamic Scene dataset.
Details of semantic fields for dynamic foreground. We use the SlowOnly50 network (Feicht- enhofer al., 2019) as our encoder Edy and remove all the pooling layers the network. The encoder is pretrained on Kinetics-400 dataset (Carreira & Zisserman, 2017). We remove the final fully-connected layer in the encoder. We froze the pretrained weights and the point feature vector Fag (T; Φ.) is sampled from the first, second and third spatial feature maps of the encoder, which are concatenated in the channel dimension and fused with a fully connected layer. The vector has 256 channels. We follow previous works (Gao et al., 2021; Tian et al., 2023) and use the same embedding function 7-)). We follow Tian et al. (2023) and implement the flow network V flow and geometric network V geo by using MLPs with residual links and width 128. We employ 4 residual blocks to implement V// flow and V/ geo- The semantic network fydy sem is implemented by three fully connected layers with ReLU activation. As for flow attention module, the head number H is 4 and the number of channels C is 64. The inital training steps is 40,000 for each scene.
Details of semantic field for static backgrounds. We follow Tian et al. (2023) and use ResNet18 (He al., 2016) our encoder Est. The point feature (T; is sampled from four feature maps prior to four pooling layers of ResNet18 8 and concatenated in the channel dimension. We fused the concatenated features with fully connected layer form feature vector of size 256. The semantic network Ur is implemented by three fully connected layers with ReLU activation. The initial training steps 100,000 for each scene.
Details of learning from multiple scenes. We train our semantic field on two scenes simultane- ously. During training, there are 512 rays for each scene in a mini-batch. takes about 16 hours to learn the semantic field from two scenes.
Details of semantic adaption on novel scenes. After training from multiple scenes, we finetune our model on novel scenes by using the pretrained model on BalloonI and Balloon2 scenes with 10,000 training steps. The semantic fields foregrounds and backgrounds are separately finetuned with preferred finetuning steps. It takes about 10 minutes to finetune the model on novel scene with 500 steps.
Scene editing. We conduct instance-level scene editing applications by control the volume densities related to specific semantic labels. For instance, we delete the person| in the Jumping scene by forcing the volume densities of the points that has the label of person] to zeros.
Details of semantic completion. In this setting, after training the model on the frames that include semantic labels, directly employ the symantic view synthesis the rest unseen frames. We change the initial flow displacement to P&AT.
Details of dynamic scene tracking. Similar to semantic completion, directly conduct semantic view synthesis on the unseen frames by using the pretrained model on the frames with semantic labels. The initial flow displacement here AT.
14

---

Published as a conference paper at ICLR 2024
Table More comparisons on semantic representation learning from multiple scenes and semantic adaption of novel scenes (500 finetuning steps). represents adding semantic head to the original model. By learning from flow features, our model could reconstruct semantic fields and transfer to novel scenes with the state-of-the-arl performance.
Total / /
Balloon!
Learning semantics from multiple scenes
Balloon2
Jumping
Skating
Umbreela
Semuntic aduption of novel scenes
Playground
Trick
NeRF+dmee (Mildenhall et al,, 2020)
0,780 0.5529
0.455/0.211
0.913 0.566
0.857 / 00227
0.953 /07763
0.920 0.919/
/0.433
NSFT (Li al., 2021)
0.583 00.00
0.420
0.954/05 0.3337
0.910/0.62
0.661
RaDyNeRF2 (L.u al, 2023b)
0.567 0.459
0.531 /0,312
0.886 0.432
0.888 / 00.536
0.900/0.330
0.876 /.443
0.904 / /0.606
D.NeRF (Pamarola et 2020)
0.381 /0.076
0.700/ 0.221
0.3323
DynNeRE (Gan et ol., 2021)
0.767 / 70.515
0,459 0,459/0.222 /.222
0.912 0.570
0.955 / 00.431
0:941
70.795
00.387
0.967 /0.062
MomoNeRF (Tian al., 2023)
0.907 0,760
0,967 0.616
0.929/ 0.576
0,973/
0.961 08.68
0.879
00.425
Semantic Flow
0.919 / 0.844
0.967 / 0.839
0 0,703
0,970 0.608
0.970 / 0.884
0,937 0.742
0,977 / 0.765
F DETAILED RESULTS ON SEMANTIC COMPLETION AND DYNAMIC SCENE TRACKING
Table 10: Quantitative results on semantic completion. We train the semantic field with 50% frames with semantic labels and tested the semantic view synthesis performance on the rest frames. denotes that we added semantic head to the original model.
Acef mloo
Jumping
Skating
Truck
Umbrella
Balloent
Ballooe2
Playground
Average
DeATT (Yang & Yang, 2022)
0.733 / 0.292
0,863 /0.344
0,992/0 / 0.556
0,77 / 0.520
0.567 0.3
0337 /
0.866/ 0.415
0.816 0-122
DynNeRF (Cao 2021)
0.896 / 0.660
0,955 6.660
0,949 0.771
0.953 0.820
0.863/0.782
0.947/ 0.788
0,956 0.745
6.934/0.73
MonoNoRF 2023)
0.917
0.964 0064470.00)
0.975/0.769
0.966 0.844
0.912/0.800
0.978
0.962/0 /0776
(0.766
SemantieFow
0.940 0.771
0 /0.777
19776007766
0.967 00.660
0.922/ 0.543
0.983/ 00.920
7..769
0.818
Table 11: Quantitative results on dynamic scene tracking. We train the model on the front 25%
frames with semantic labels and evaluated the performance by render novel semantic views on the rest frames. 0 represents that we add a semantic head to the original model.
Acct /mloU?
Jumping
Skating
Umbeellaa
Balloon
Balloon
Playground
Average
DeAOT (Yang & Yang, 2022)
0.915/0.534
0.797
0.560:
0:24
70.47
0.776
DynNeR19 (Giao at al., 2021)
0336 / 0,595
0.912 70.563
0,914 0.637
0.266 0.767
0.825 0.099
0.925 / 0.729
0:922 70.648
0.896 / 0.660
MonNNeRF (Tian et at, 2023)
0,96 / 0.619
0.964 /.624
0,928 /0.623
0.95/0 0.766
0.897/ 708
0.954/ / 0.791
0.954 0.797
0.935/0.75 0-666
Semantic Flow
0.906/0.67
0,966 /0665
0,96600685
0.955 0,866
0,902 0.506
0.975/ 0.893
0.930/0.772
0,942/ 0.767
Semantic completion. We list the detailed performance of semantic completion in Table 10. The results are reported by training with 50% labels.
Dynamic Scene tracking. We present the detailed results of dynamic scene tracking in Table 11. The experiments are conducted by training the model with semantic labels on frames {41, 12, 13} i.., 25% labels.
G DISCUSSIONS
G.1 DISCUSSION ABOUT SCENE RECONSTRUCTION
In this section, we compare the scene reconstruction performance of our model with MonoN- cRF (Tian et al., 2023) and DynNeRF (Gao et al., 2021). Table 12 shows that model achieves better performance in PSNR, SSIM and LPIPS indexes. By reconstructing the dynamic scenes pre- cisely, model could conduct instance-level scene editing applications.
G.2 DISCUSSION ABOUT FLOW FIELD AND CORRESPONDENCE
We visualize the predicted flow fields and pixel correspondence of the dynamic foreground in three consecutive frames in Figure 6. Although we do not add extra restrictions to the mappings of flows in (16), the visualization of the flow fields and correspondence shows that our model is able to predict flows that map similar parts of the dynamic foreground across time.
15

---

Published as a conference paper at ICLR 2024
Table 12:
The results of scene reconstruction. Our model achieves better color rendering perfor- mance compared with other state-of-the-art methods.
PSNR SSIM t L/PIPS
Jumping
Skating
Average
DynNeRR (Gao et al., 2021)
21.91 / 0.6856 00.17
24.68 / 0.7866 0,5
23.30/ /.7331 / 0.117
MonoNeRF (Tion el al,, 2023)
22.41 / 0.7484 001445
26.18 / 0.8739 / 00,15
24.30 0.8102/////////// 0.130
Semantic Flow
22.86/ 0.7658 0,138
25.75 0,8774 0,113
24.31 0.8216/ 0.126
Rendered Image
Backward Flow
Forward Flow
Predicted Semantics
Correspondence
Framte
Frame #2
Framee 48
Figure 6: Visualization of rendered RGB images, estimated flow fields, semantic predictions and pixel correspondence three consecutive frames. the correspondence visualization, the color coding illustrates correspondences of the dynamic foreground across time.
G.3 DISCUSSION ABOUT OFTICAL FLOWS
Table 13:
Performance semantic representation learning from multiple scenes with optical flow maps generated from different flow estimation methods (Total Acc / mloU1).
learning semantics from multiple scenes Jumping
method
Balloon
Balloon2
Skating
RAFT (Toed & Deng. 2020)
0.919 11199/0.80 /0.844
0.967 / 0.839 0.936 / 0.733 0,970/ / 0.716
FlowNet (Dosavitskly et al., 2015)
0.926 / 0.855
0.974 / 0.766 0.921 / 0.636
0.965 / 0.591
Table 14: Performance on semantic representation learning from multiple scenes under the supervi- sion of noisy optical flow maps (Total Acct / mloU?).
Leaming semantics from multiple scenes
noise scule 8
Balloon1
Balloon2
Jumping
Skating
0
0.919 0.844
0.967 / 0,839
0.936 / 0.733
0.970 0.716
1%
0.923 0.844
0.963 0.820
0.937 / 0.715
0.770
5%
0.924 0.808
0.964/ 0.813
0.937 0.656
0.969/0.75
10 %6
0.922 0.777
0.964 / 0.666
0.936 / 0.690
0.969 0.670
Since Semantic Flow relies on the optical flows predicted from the pretrained models, in this sec- tion, we discuss the robustness of the model supervised by optical flow signals from two perspec- tives: choosing different optical flow estimation methods and adding noise to optical flow maps. We first compare the semantic prediction performance by using RAFT (Teed Deng, 2020) or FlowNet (Dosovitskiy et al., 2015). accroding to the experiments in RAFT paper, RAFT outper- forms FlowNet with a large margin in various optical flow estimation tasks. Therefore, it can be seen that there is about 10% performance drop in mIOU matrix when jointly optimizing Jumping and Skating scenes, where the dynamic foregrounds are drastically changing. However, our model still reaches comparable results when jointly optimizing Balloon1 and Balloon2 scenes with the flow maps estimated from FlowNet. We also test the performance of our model by manually adding different scales of noise to the predicted optical flow maps. Concretely, for each optical flow map estimated from two consecutive video frames, we denote o min. e Paaz as the minimum and maxi- mum numbers of the optical flows in the map, and the noisy flow be defined as the following
16

---

Published as a conference paper at ICLR 2024
equation,
noisy 4 Porigin + +. Inoise:
(18)
where 3 controls the scale of the noise and Enoise ~ 21(11 min, 4 maz) denotes the noise. Table 14 presents that model shows robustness with small scale of noise the optical flow maps.
G.4 DISCUSSION ABOUT OCCLUSION
Frame Image
Frame Backward Flow
Frame JI6 Forward Flow
Frame #G Prediction
Frame #5 Prediction
Frame #7 Prediction
Figure 7: Visualization of the occlusion situation. We manually occlude part of both static and dynamic regions in frame #6, which leads the wrong flow maps estimated by Teed & Deng (2020). Although Semantic Flow fails to predict the semantics of the occluded part in frame #6, successfully conducts accurate predictions in frames #5 and #7.
To test the performance of our model with occluded regions, manually add an occluded region to the frame #6 in the Balloon2 scenes. Concretely, shown in Figure 7, we manually occlude region in the RGB image and semantic labels of the frame #6 and generate the wrong flow maps from the occluded image by using Teed & Deng (2020). We train the model with the occluded image and wrong optical flow maps. Although Semantic Flow meets difficulty in predicting the semantics of the occluded region in the frame #6, it successfully predicts the semantics in the frames #5 and #7.
G.S COMPARED IMAGE SEGMENTATION METHOD
To compare with the image segmentation method, use the pretrained Masked R-CNN (He et al., 2017) model to predict the semantic labels in the Jumping scene. As shown Figure 8, while Masked R-CNN processes each video frame independently and hence predicts inconsistent labels of the same instances in different timestamps, our model could generate semantic labels consistent in time by learning semantics from all the video frames.
Video Frame
Ground Truth
Semantic Flow
Mask R-CNN
Frame #8
Figure 8: Comparison with Masked-Reee RCNN (He et al., 2017). Since Semantic Flow learns a semantic field that is continuous in time, it predicts consistent semantic labels at all timestamps. In contrast, image segmentation methods such as Masked-RCNN (He et al., 2017) process each video frame individually, and hence align inconsistent instance-level labels to the same instance in different frames.
17

---

Published as a conference paper at ICLR 2024
G.6
DISCUSSION ABOUT FLOW DISPLACEMENTS
Using &
Using
Figure 9: Qualitative results by using different displacements in the dynamic scene tracking setting. While Table 4 shows that our model has robustness with different displacements, find that our model could predict semantics with better visualization quality by using T&AT.
In Table 6 and Figure 4 in the main paper, demonstrate that the boundaries of predicted semantics could be more precise and the performance could be improved by using flow displacements. While in Table we show that model has robustness with different flow displacements, find that it could generate better qualitative results by using T&AT, as shown in Figure 9.
G.7 LIMITATIONS
GT Image
GT Semantic
Predicted Semantics
Predicted Flow
Figure 10: A failure case in the tracking setting. Our model fails to predict precise semantics with imperfect flow predictions.
is worth noting that training semantic fields with few semantics labels still a challenging prob- lem. For instance, in the dynamic scene tracking setting, Semantic Flow may fail to conduct precise semantic prediction due to imperfect flow predictions as shown in Figure 10.
Jumping
Jumping
FlowNet Prediction
Prediction
Ballcon1 Frame #5
Balloon1 Frame #6
FlowNet Prediction
RAFT Prediction
Figure 11: Comparison of the flow prediction between FlowNet (Dosovitskiy al., 2015) and RAFT (Teed & Deng, 2020). In the Jumping scene, the flow maps predicted by FlowNet are totally wrong and lead performance drop in Table 13. On the other hand, in the Balloon scene, while the flow maps predicted by FlowNet are less accurate than RAFT, it successfully predicts the boundary of the balloon movement. The inaccuracy in the maps can be considered which may slightly improve the performance in the Balloon scene as shown in Table 14.
small scale of noise,
18


Reviews:
[{"review_id": "FHiEumEw3D", "reviewer": "Reviewer_xtsS", "summary": "The paper introduces learning a semantic field of the dynamic scene using NeRF given a monocular video. Given a sequence of input frames from a monocular video, precomputed optical flow, their Dynamic NeRF learns a semantic field so that it can render a semantic segmentation map at novel views. The method can be used for a couple of applications that output semantic field for unseen frames given partial frames.", "questions": "- How much does the accuracy of the method depend on the off-the-shelf optical flow methods? Can it be critical?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Better accuracy over baseline methods\n\n  Compared to the two baselines, the method shows better accuracy on multiple tasks (scene completion, scene tracking, and semantic representation) in Table 1 and Table 2. Also it demonstrates better qualitative results in Fig. 3\n\n- New applications\n\n  The paper proposes interesting new applications, both dynamic scene tracking and completion that estimates semantic maps on unseen frames. (Fig. 1)", "weaknesses": "- Outdated baselines\n\n  The paper compares their method with a couple of baselines (DynNeRF and MonoNeRF) but those are a bit limited. There are many other baselines for the dynamic NeRF task such as D-NeRF, RoDynRF, NSFF (Neural Scene Flow Field), etc. It would have been great if the paper provided accuracy on more baseline methods to make the comparison much fairer. \n\n- A bit difficult to follow the equations (from Eq. (4) to Eq. (8))\n\n  I am wondering if it's possible to put the mathematic notation from Eq. (4) to Eq. (8) into Fig. 2 for better understanding.\n\n- Clarity \n\n  Some parts of the paper have lack of clarity and make it hard to understand clearly. What is the meaning of '25%/50% semantic labels' in Fig. 3? I wonder if the paper can provide more details in the figure captions. How are the 25%/50% determined? \n\n- Marginal accuracy improvement in Fig. 4\n\n  The choice of low displacement seems not so critical for the accuracy gain. Maybe it would be good to have a justification or discussion on the result.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "w7qNUjTLoQ", "reviewer": "Reviewer_5VQj", "summary": "The paper looks to solve the problem related to generating a novel view 2D semantic map, for dynamic scenes using continuous flow. Paper leverages optical flow for the foreground part of the images and uses volume density as a prior to determining flow feature contribution towards semantics. Authors evaluate this on Dynamic scene dataset.", "questions": "- Could Authors share some of the shortcomings (a few qualitative results) which may be due to imperfect flow prediction, which results in bad performance during inference?\n- In Section 3.4: while calculating Semantic Consistency Constraint; Do we generate some sort of valid mask here to enforce the semantic consistency or is it done for all pixels, irrespective of occlusion or uncertainty?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Strengths:\n-The paper is well written with the objective clearly identified. It is structured well and has logically moving sub-section-wise explanations. \n-Tackles a well-known problem in terms of generating novel view synthesis for dynamic scenes, but for semantics.\n- Proposes a novel idea, that leverages optical flow to predict semantic labels for dynamic foreground pixels/regions.\n- Evaluate and compare the model on the Dynamic Scene Dataset.", "weaknesses": "Weakness:\n- Paper leverages optical flow output as one of the intermediate steps, but fails to discuss its shortcomings and how exactly do they handle occlusion and disocclusion related to both dynamic and static regions of the frame.\n- For the most part of the paper, the authors only compare with two dynamic scene-based works, Considering other related works in dynamic scene reconstruction, Would be great to see comparative baseline results, with a few more of these models with semantic head.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Np1HZYdSOk", "reviewer": "Reviewer_QtRh", "summary": "In this paper the problem of novel view synthesis of semantic labels is studied. Rather than rendering colour, the segmentation is rendered with a NeRF model. The authors propose a model and training procedure that can learn scene flow fields and semantic renderings of a given video sequence. Furthermore, the proposed method allows for quick adaptation on novel video sequences. Since no earlier work studies this problem, the authors label the nvidia dynamic scenes dataset with pixel-wise semantic labels. The experiments show that the semantic labels can be rendered accurately, and furthermore that we do not need labels for all frames in a sequence, and that we can use the semantic labels to mask out specific parts of the video and render it without specific objects.", "questions": "See everything under weaknesses.\n- When training for semantic completion or tracking, only a subset of the frames are used for semantic supervision. Is the same true for RGB supervision or are all frames used for that? \n- In Fig. 3, what are the indices of the frames that are shown? How far are they from the frames with semantic labels?\n- For the DynNeRF and MonoNeRF baselines, what exactly is the input to the semantic heads that are learned?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- There is no existing dataset for the problem setup, so authors annotated the nvidia dynamic scenes dataset with pixel-wise semantic segmentation labels.\n- A strength of the method is that it allows for quick adaptation to new scenes, e.g. with just 500 iterations it can perform well given pre-training on other scenes. The reason is that the scene flow field is not learned from scratch, but rather from frame-wise video features, which does not need to be learned from scratch for each scene.\n- Augmenting NeRF models with semantic segmentation has been done for static scenes (e.g. Zhi et al 2021) but to the best of my knowledge not for general dynamic scenes, so the paper tackles a new problem setup.\n- The method is clearly described and ablations are provided for the main components.", "weaknesses": "- There are some baselines that would be reasonable to try that are missing from the paper. For instance, if we just render the rgb images with any NeRF method (e.g. MonoNeRF) and apply some video object segmentation algorithm (e.g. any top-performing method on the DAVIS dataset) or semantic segmentation method (trained on some dataset with overlapping labels), how well would that perform?\n- Since one of the applications mentioned in the paper is scene editing, i.e. removing some specific object, it is necessary to not just render semantics correctly but also rgb. There are no values provided for the standard novel view synthesis metrics (PSNR, SSIM, LPIPS) for rgb on the tested video sequences.\n- It would have been interesting to somehow visualise or discuss the flow fields. Since the objective is semantic rendering rather than colour rendering it is not clear if we need the scene flow to map to the same specific part of an object, or if it is sufficient or even beneficial to just map to anywhere within the same object. For instance, the consistency loss L_consis only enforces that points along flow trajectories should have the same label.\n\nMinor issues:\n- Missing related work: “Panoptic neural fields: A semantic object-aware neural scene representation” (CVPR 2022) also considers novel view synthesis for semantic segmentation from a video, although their method is limited to non-deformable dynamic objects.\n- Page 5: Does ground truth flow mean optical flow estimated from RAFT? If that is the case it should not be called ground truth.\n- Page 5: The closing parenthesis in eq. (7) is probably incorrect.\n- Page 8: “Boundray” typo\n- Page 8: Table 2 caption: “qualitative” should be “quantitative”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)