HUGS: Holistic Urban 3D Scene Understanding via Gaussian Splatting
Hongyu Zhou1 , Jiahao Shao1 , Lu Xu1 , Dongfeng Bai2 , Weichao Qiu2 , Bingbing Liu2
Yue Wang1 , Andreas Geiger3,4 , Yiyi Liao1B
1

Zhejiang University

2

Huawei Noah’s Ark Lab

3

University of Tübingen4Tübingen AI Center

Input Images➧
Predicted Flow➧

Novel View Synthesis

Novel View Semantic Synthesis

Optical Flow

Depth

Predicted Semantic

Predicted 3D Trackings

3D Semantic Reconstruction & 3D Trackings

Figure 1. Illustration. Given posed RGB images as input, our method lifts noisy 2D & 3D predictions to the 3D space via decomposed
3D Gaussians, and enables holistic scene understanding in 2D and 3D space.

Abstract
Holistic understanding of urban scenes based on RGB
images is a challenging yet important problem. It encompasses understanding both the geometry and appearance toenable novel view synthesis, parsing semantic labels, andtracking moving objects. Despite considerable progress,
existing approaches often focus on specific aspects of thistask and require additional inputs such as LiDAR scansor manually annotated 3D bounding boxes. In this paper, we introduce a novel pipeline that utilizes 3D Gaussian
Splatting for holistic urban scene understanding. Our mainidea involves the joint optimization of geometry, appearance, semantics, and motion using a combination of staticand dynamic 3D Gaussians, where moving object poses are
regularized via physical constraints. Our approach offers
the ability to render new viewpoints in real-time, yielding
2D and 3D semantic information with high accuracy, andreconstruct dynamic scenes, even in scenarios where 3D
bounding box detection are highly noisy. Experimental results on KITTI, KITTI-360, and Virtual KITTI 2 demonstrate the effectiveness of our approach. Our project pageis at https://xdimlab.github.io/hugs website.

1. Introduction
Reconstructing urban scenes is an important task in computer vision with numerous applications. Consider the creation of a photorealistic simulator for autonomous driving,
B Corresponding author.

in this context, it becomes crucial to holistically represent
all aspects of the scene relevant to driving. This entails
tasks like synthesizing images at interpolated and extrapolated viewpoints in real-time, reconstructing 2D and 3D
semantics, generating depth information, and tracking dynamic objects. To minimize sensor cost, achieving such aholistic understanding exclusively from posed RGB images
holds significant value.
With the rise of neural rendering, many approaches haveemerged to lift 2D information to 3D space, enabling scene
understanding based solely on RGB images. Several previous works focus on reconstructing static urban scenes,
achieving high-quality novel view appearance and semantic synthesis [11, 30, 51]. Another line of work addresses
dynamic scenes [19, 27, 40, 46], but most of them require
ground truth 3D bounding boxes of dynamic objects as input, which are costly to acquire. PNF [19] is the only
method that utilizes noisy bounding boxes obtained throughmonocular 3D detection and tracking, where the transformations of the bounding boxes are jointly optimized during
training. However, naı̈ve joint optimization of per-frame
pose transformations is prone to local minima and sensitive
to the initialization. Furthermore, while existing methods
are capable of rendering accurate 2D semantic labels, it is
non-trivial to extract accurate semantics in 3D due to the inaccurate (inferred) 3D geometry. In addition, most of thesemethods are unable to achieve real-time rendering.
In this paper, We leverage predicted 2D semantic labels,
optical flow, and 3D tracks, despite their inherent noise andimperfections, to achieve a holistic understanding of the dy-

namic scenes based on RGB images (see Fig. 1). Towards
this goal, we infer geometry, appearance, semantics, andmotion in 3D space using a decomposed scene representation. We leverage 3D Gaussians as the scene representation,
which have recently demonstrated superior novel view synthesis performance on static scenes with real-time rendering
capability [17]. Specifically, we propose to decompose thescene into static regions and rigidly moving dynamic objects. We model the poses of these moving objects while
adhering to the physical constraints of a unicycle model, effectively reducing the impact of noise during tracking andleading to superior performance compared to optimizing
object poses individually. This allows us to reconstruct dynamic scenes even when 3D bounding box predictions arehighly noisy. Further, we extend 3D Gaussian Splatting
to model camera exposure and explore initialization on dynamic scenes, enabling state-of-the-art novel view synthesis
performance on urban scenes. Additionally, we incorporate
semantic information into 3D Gaussians, enabling the rendering of semantic maps and the extracting of 3D semantic point clouds. Finally, we integrate the RGB, semantics
and optical flow to jointly supervise the model training, andinvestigate the interaction between these image cues to improve the performance of the scene understanding tasks.
Our main contributions are as follows: 1) Our methodaddresses the task of dynamic 3D urban scene understanding by extending Gaussian Splatting to model additional
modalities, including semantic, flow, and camera exposure,
as well as dynamic objects. 2) We achieve the decomposition of static and multiple dynamic objects from sparse
urban images and noisy labels by incorporating physicalconstraints, omitting the requirement of ground truth 3D
bounding boxes for reconstructing dynamic scenes. 3) Our
method achieves state-of-the-art performance on various
benchmarks, including novel view appearance and semantic
synthesis, as well as 3D semantic reconstruction.

2. Related Work
3D Scene Understanding: Understanding urban scenes
from various aspects has been considered essential for autonomous driving. Numerous techniques have focusedon predicting semantic labels [5, 9, 35], depth maps [10,28], and optical flows [42] solely from 2D input images.
While these methods have demonstrated impressive accuracy within the confines of the 2D space, they often fallshort of grasping a profound understanding of the underlying 3D environment. Consequently, this limitation canhinder the multi-view consistency of their predictions. Another line of approach suggests conducting semantic scene
understanding solely based on 3D input [29, 31]. This approach heavily relies on LiDAR input, which is known tobe costly and resource-intensive to collect.

More recently, a particular approach has emerged, aiming to elevate 2D information to the 3D space to facilitate scene understanding within the 2D image domain.
This advancement is made possible through the utilizationof differential neural rendering techniques, such as NeRF
(Neural Radiance Fields) [25]. Numerous NeRF-based approaches [2–4, 14, 26, 34, 38] have made significant advancements in terms of both quality and efficiency. Furthermore, some other techniques have empowered NeRF
with improved scene understanding capabilities. Semantic
NeRF [52] first proposes the lifting of noisy 2D annotations
to the 3D space based on NeRF. Significant progress has
been achieved through the efforts of the following works[37, 44, 49]. While these methods have shown promising
results, they are currently limited to dense input viewpoints
within indoor scenes and are only applicable to static environments. In this study, our focus lies in dynamic 3D
scene understanding specifically tailored to urban settings,
achieved by lifting 2D information to the 3D space.
Urban Scene Reconstruction: Numerous studies havebeen conducted to reconstruct urban scenes using various
methods. These methods can be categorized into threeclasses: point-based [1, 32], mesh-based [12, 20] and
NeRF-based [15, 22, 24, 30, 33, 39, 51]. While point-based
and mesh-based methods demonstrate faithful reconstructions, they struggle to recover all aspects of the scene, especially when it comes to high-quality appearance modeling. In contrast, NeRF-based models allow for reconstructing scene appearance and enable high-quality rendering ofnovel viewpoints. However, these approaches are primarily
designed for static scenes, lacking the ability to handle dynamic urban environments. In this study, our focus lies inaddressing the challenges of dynamic urban scenes.
Several methods have also been developed to address thereconstruction of dynamic urban scenes. Many of theseapproaches rely on the availability of accurate 3D bounding boxes for moving objects in order to separate the dynamic elements from the static components, as seen in NSG[27], MARS [40] and UniSim [46]. PNF [19] takes a different approach by leveraging monocular-based 3D bounding box predictions and proposes a joint optimization of
object poses during the reconstruction process. However,
our experimental observations indicate that the straightforward optimization of object poses yields unsatisfactory results due to the absence of physical constraints. Anothermethod, SUDS [36], avoids the use of 3D bounding boxes
by grouping the scene based on learned feature fields. However, the accuracy of this approach lags behind. In parallel,
the concurrent work EmerNeRF [45] follows a similar idea
to SUDS by decomposing the scene purely into static anddynamic components. In our research, we possess the capability to further decompose individual dynamic objects
within the scene and estimate their motion.

Figure 2. Method Overview. We decompose the scene into static regions and N rigidly moving dynamic objects. Each dynamic object isrepresented using 3D Gaussians in its canonical space and then transformed to the world coordinates based on transformations constrainedby a unicycle model. We use N unicycle models of different parameters to individually represent the motion of N dynamic objects.
Each 3D Gaussian encompasses information about appearance and semantics, whereas the optical flow can be obtained by calculating theGaussian center’s motion, enabling the rendering of RGB images, semantic maps, and optical flow within a unified model. Our method issupervised using RGB images, noisy 2D semantic labels, and noisy optical flow, denoted as LI , LS , and LF , respectively.

Gaussian Splatting: 3D Gaussians are demonstrated asa powerful scene representation for novel view synthesis.
While the original 3D Gaussian Splatting [17] primarily focuses on static scenes, subsequent research has extended
this approach to handle dynamic scenes. Dynamic 3D
Gaussians [23] necessitates a substantial number of training views accompanied by ground truth masks. Other studies [43, 47, 48, 53] have also attempted to decompose 3D
Gaussians into static and dynamic components, without further decomposing multiple dynamic objects. In our work,
we strive to achieve the decomposition of each individual
dynamic object while being capable of learning such decomposition from sparse urban images and noisy labels.

3. Method
Fig. 2 illustrates our proposed method, HUGS. Our algorithm takes as input posed images of a dynamic urban scene.
We decompose the scene into static and dynamic 3D Gaussians, with the motion of dynamic vehicles being modeled
via a unicycle model. The 3D Gaussians represent not only
appearance but also semantic and flow information, allowing for rendering the RGB images, semantic labels, as well
as optical flow through volume rendering.

3.1. Decomposed Scene Representation
We assume that the scene is composed of static regions
and a total of N dynamic vehicles exhibiting rigid motions.
Static regions are represented using static Gaussians in theworld coordinate system. Each of the N dynamic vehicles is modeled using dynamic Gaussians in a canonical coordinate system along with a set of rigid transformations{(Rnt , tnt )}Tt=1 with t denoting the timestamp.
Static and Dynamic 3D Gaussians: Following Gaussian

Splatting [17], we model both static and dynamic regions
using 3D Gaussians. Each Gaussian is defined by a 3D covariance matrix Σ ∈ R3×3 and a 3D position µ ∈ R3 , aswell as an opacity α ∈ R+ :


1
T −1
(1)
G(x) = α exp − (x − µ) Σ (x − µ)2
In addition, each Gaussian represents a color vector c ∈ R3
parameterized as SH coefficients. In this work, we propose
to additionally model semantic logits s ∈ RS of each 3D
Gaussian, allowing for rendering 2D semantic labels. Furthermore, we can naturally obtain a rendered optical flow
ft1 →t2 ∈ R2 for each 3D Gaussian by projecting the 3D position µ to the image space at two different timestamps, t1
and t2 , and calculating the motion.
Unicycle Model: We parameterize the transformations(Rt , tt ) following the unicycle model1 . The state of a unicycle model is parameterized by three elements: (xt , yt , θt ),where xt and yt represent the first two axes of t with
tt = [xt , yt , zt ], and θt is the yaw angle of Rt . To adapt
the continuous unicycle model to discrete frames, we derive
the calculus of the unicycle model for the vehicle transition
from timestamp t to t + 1 as follows:vt(sin θt+1 − sin θt )
ωtvt
yt+1 = yt − (cos θt+1 − cos θt )ωt
θt+1 = θt + ωt

xt+1 = xt +(2)Here, vt represents the forward velocity, and ωt is the angular velocity. This model integrates physical constraints
1 While it is more accurate to model vehicles using a bicycle model, weobserve that using the simpler unicycle model is sufficient for our task.

when compared to directly optimizing the transformationsof dynamic vehicles at every frame independently, thus enabling smoother motion modeling of moving objects andmaking them less prone to local minima.
While it is possible to define an initial state (x1 , y1 , θ1 )and derive the following states recursively based on velocities, vt and ωt , such a recursive parameterization is challenging to optimize. In practice, we define a set of trainable states {(xt , yt , θt )}Tt=1 along with trainable velocities−1{vt , ωt }Tt=1, and add a regularization term to ensure that thevehicle’s states adhere to the characteristics of a unicycle
model in Eq. 2. The regularization terms will be describedin Section 3.3. Additionally, we model the vertical locationsof the vehicle, {zt }Tt=1 , as optimizable parameters.

3.2. Holistic Urban Gaussian Splatting
Given the HUGS representation specified above, we areable to render images, semantic maps and optical flow tosupervise the model or make predictions at inference time.
We now elaborate on the rendering of each modality.
Novel View Synthesis: The combination of static and dynamic Gaussians can be sorted and projected onto the image
plane via α-blending:
π:

C=

X
i∈N

ci αi′i−1Y(1 − αj′ )

(3)

j=1

Here, αj′ is determined by the projected 2D Gaussian andthe 3D opacity α, see supplement for details.
In contrast to single-object scenes, urban scenes typically involve more complex lighting conditions and the images are usually captured with auto white balance and auto
exposure. NeRF-based methods [24] typically feed a perframe appearance embedding along with the 3D positions
into a neural network to compute the color, thereby compensating exposure. However, when working with 3D Gaussians, there is no neural network capable of processing appearance embeddings. Inspired by Urban Radiance Field[30], we generate an exposure affine matrix for each camera
by mapping the camera’s extrinsic parameters to an affine
matrix A ∈ R3×3 and vector b ∈ R3 via a small MLP:C̃ = A × C + b

Semantic Reconstruction: Similarly to Eq. 3, we can obtain 2D semantic labels via α-blending based on the 3D semantic logit s:
S=

X
i∈N

softmax(si )αi′

Note that we perform the softmax operation on 3D semantic logits si prior to α blending, in contrast to most existing methods that apply softmax to 2D semantic logits S̄
obtained by accumulating unnormalized 3D semantic logits si [11, 52]. As shown in Fig. 3, applying softmax in 2D
space leads to noisy 3D semantic labels. This is due to thefact that 2D space softmax can produce accurate 2D semantics by adjusting the scale of the 3D semantic logits, allowing a single sampled point with a substantial logit value tosignificantly influence the volume rendering outcome. For
example, an undesired floating point labeled with “car” may
not be penalized despite the target rendered label is “tree”,
as long as there is a 3D Gaussian providing a large logit
value of “tree” along this ray. Our solution instead removes
such floaters by normalizing logits in 3D space. See supplement for more quantitative and qualitative details.
Optical Flow: The 3D Gaussian representation also enables the rendering of optical flow. Given two timestamps t1
and t2 , we first calculate the optical flow of each 3D Gaussian’s center µ as ft1 →t2 . Specifically, we project µ to the2D image space based on the camera’s intrinsic and extrinsic parameters:
camµ′1 = K[Rcamt1 ; tt1 ]µ,camµ′2 = K[Rcamt2 ; tt2 ]µ,i−1Y(1 − αj′ )

j=1

(5)

(6)

and then calculate the motion vector as ft1 →t2 = µ′2 − µ′1 .
Next, we render the optical flow via accumulate the opticalflows via volume rendering:(4)

We demonstrate that modeling the exposure improves rendering quality in the experimental section.

π:Figure 3. 3D Semantic Reconstruction. Comparison between applying softmax to accumulated 2D semantic logits (left) and to 3D
semantic logits (right). Normalizing semantic logits in 3D spaceclearly reduces floaters and yields better 3D semantic reconstruction than the 2D normalization counterpart.

π:

F=

X
i∈N

fi αi′i−1Y(1 − αj′ )

(7)

j=1

Note that this rendering process assumes that any pixel ofa 2D Gaussian splat shares the same optical flow direction
as the corresponding Gaussian center but with scaled magnitude. While this is indeed a simplified approximation, we
observe this to work well in practice.
In our experiments, we demonstrate that supervising therendered optical flow with pseudo ground truth helps to im-prove the performance of the geometry in terms of rendered
depth maps. This is due to the fact that flow provides explicit pixel correspondences, which is inherently supervising the underlying surface location.

In addition, we regularize the acceleration of the forward
velocity vt and angular velocity ωt to be smooth:X
Lreg =∥vt+1 + vt−1 − 2vt ∥2 +t

3.3. Loss Functions

X

We leverage pre-trained recognition models to provide
noisy 2D semantic and instance predictions, noisy 2D optical flow, as well as noisy 3D tracking results. These easyto-obtain predictions are critical to enable RGB-only holistic scene understanding in both 2D and 3D space, without
relying on LiDAR input or 3D semantic supervision.
Image-based Losses: Our model is supervised with theground truth images using a combination of L1 and SSIM
losses. Let Ĩ denote the rendered image and Î the groundtruth, our rendering loss is defined as follows:
LI = (1 − λSSIM )∥Î − Ĩ∥1 + λSSIM SSIM(Î, Ĩ)(8)

We additionally apply the cross-entropy loss to the rendered semantic label wrt. pseudo-2D semantic segmentation
ground truth Ŝ:LS = −S−1
X

Ŝk log(Sk )

(9)

k=0

Similarly, we leverage pseudo optical flow ground truth F̂to supervise the rendered optical flow using:
LF = ∥F̂ − F∥1(10)While 3D Gaussians can enable the rendering of opticalflow without any supervision, we observe artifacts in therendered flow without supervision. Further, the optical flow
supervision yields an improvement in the depth maps asshown in our ablation study.
Unicycle Model Losses: We use a unicycle model to guidethe noisy 3D bounding box predictions:XX
Lt =∥xt − x̂t ∥2 +∥yt − ŷt ∥2
(11)
t

t

where x̂t and ŷt are the x and y locations of a noisy 3D
bounding box at timestamp t.
As mentioned earlier, we parameterize the vehicle’s
states (xt , yt , θt ) and the velocities vt , ωt as learnable parameters. Hence, we add the following regularization to
make the states adhere to the unicycle model as follows:
X
vt
Luni =∥xt+1 − xt − (sin θt+1 − sin θt )∥+
ω
t
t
X
vt
∥yt+1 − yt + (cos θt+1 − cos θt )∥+ωttX∥θt+1 − θt − ωt ∥(12)
t

∥θt+1 + θt−1 − 2θt ∥2

(13)

t

The total loss can be summarized as follows:
L = LI + λS LS + λF LF + λt Lt + λuni Luni + λreg Lreg
(14)

3.4. Implementation Details
Initialization: While 3D Gaussian Splatting is not highly
sensitive to the initialization, better initialization can yield
better performance. We utilize the dense point cloud obtained from COLMAP for initialization by default. Whenthe ego-vehicle is static, we use random initialization.
Pseudo-GTs:
We utilize InverseForm [5] to generate
pseudo ground truth for semantic segmentation. For initializing the unicycle model, we employ a monocular-based
method, QD-3DT [16], to acquire pseudo ground truth for
3D bounding boxes and tracking IDs at each training view.
For optical flow, we use Unimatch [41] to obtain pseudo
ground truth.
Training: We train the model for 30,000 iterations on
dynamic scenes. For the KITTI-360 leaderboard, we perform early stopping at 15,000 iterations. Following [17], we
adopt the approach of setting the weight parameter λSSIM
to 0.2. Furthermore, we assign weights λS and λF as 0.01,
while λt , λuni and λreg are set as 0.1. The learning rateof the unicycle model parameters progressively decreases
during training.
Time Consuming: Our approach can converge within 30
minutes and achieve inference at a speed of approximately
93 fps on a single NVIDIA RTX 4090. While NSG and
MARS inference at a speed of less than 1 fps. A speed
breakdown of our method is provided in the supplement.

4. Experiments
Datasets: We perform a range of experiments to assess
the performance of our model across various tasks, suchas novel view synthesis, novel semantic synthesis, and 3D
semantic reconstruction. These experiments are conductedusing the KITTI [13], Virtual KITTI 2 (vKITTI) [7], and
KITTI-360 datasets [21]. We apply 50% dropout rate following existing evaluation protocols [21, 40] on all of thesedatasets.
Baselines: We evaluate the dynamic scene novel view synthesis task by comparing our method with NSG [27] andKITTIvKITTI

NSG

MARS

Ours

GT

Figure 4. Qualitative Comparison on KITTI and vKITTI. We use monocular-based 3D bounding box predictions for KITTI, and manually
jittered 3D bounding boxes for vKITTI. We zoom in on a patch of a dynamic object for each KITTI scene.

NSG [27]MARS [40]Ours

KITTI Scene02
PSNR↑ SSIM↑ LPIPS↓
23.00
0.664
0.373
23.30
0.731
0.139
25.42
0.821
0.092

KITTI Scene06
PSNR↑ SSIM↑ LPIPS↓
23.78
0.717
0.234
25.09
0.856
0.083
28.20
0.919
0.027

vKITTI Scene02
PSNR↑ SSIM↑ LPIPS↓
21.40
0.689
0.376
22.67
0.882
0.128
26.21
0.911
0.040

vKITTI Scene06
PSNR↑ SSIM↑ LPIPS↓
20.60
0.719
0.255
21.67
0.856
0.134
26.65
0.921
0.030

Table 1. Novel View Synthesis on Dynamic Scenes with predicted or noisy 3D trackings.

MARS [40], which are two open-source methods for dynamic urban scenes. Additionally, we compare the static
novel view appearance and semantic synthesis task with
mip-NeRF [2], PNF [19], and MARS [40]. Furthermore,
we assess the quality of 3D semantic scene reconstruction
by comparing it with Semantic Nerfacto [34].
Evaluation Metrics: For novel view synthesis, we adopt
the default setting for quantitative assessments, including
the evaluation of PSNR, SSIM and LPIPS [50]. Regarding
novel view semantic synthesis, we follow KITTI-360 [21],which reports the mean Intersection over Union on class(mIoUcls ) and category (mIoUcat ), respectively. Further,
we evaluate our performance on 3D Semantic Segmentation
against a ground truth semantic LiDAR point cloud, measuring both geometric reconstruction quality and semantic
accuracy. The geometric quality is evaluated as the chamferdistance between two point clouds, including completenessand accuracy, whereas the semantic accuracy is also measured using mIoUcls . In our ablation study, we evaluate 3D
tracking performance by measuring the rotation and translation error eR and et of our optimized 3D bounding boxes
wrt. the ground truth.

4.1. Novel View Synthesis
We first evaluate HUGS for novel view synthesis on various
datasets including dynamic and static scenes. For dynamic
scenes, we leverage noisy 3D bounding box predictions as
input, instead of using the ground truth. Despite not being

our main focus, we include a comparison of using groundtruth 3D bounding boxes in the supplement.
Dynamic Scene with Noisy 3D Bounding Boxes: Following [27, 40], we evaluate our performance on dynamic
scenes of the KITTI and vKITTI datasets. In contrast tothese methods that leverage ground truth poses, we investigate a more practical scenario where the bounding boxes
are generated by a monocular-based 3D tracking algorithm,
QD-3DT [16], in Table 1. Here, the predicted 3D bounding boxes are only provided for training views, as testing
views should not be used as inputs for the tracking model.
In experiments where the unicycle model is not utilized, thebounding boxes of testing views are obtained through linear
interpolation from neighbour training views. Where the unicycle model is used, the bounding boxes of testing views arecomputed using Eq. 2. For vKITTI, there is no pre-trained
monocular tracking algorithm. We hence jitter the groundtruth poses to simulate noisy monocular predictions, withan average noise of 0.5 meters in translation and 5 degrees
in rotation. Our model’s robustness wrt. various levels ofnoise will be analyzed in the ablation study.
Table 1 demonstrate that our method consistently outperforms against the baselines. Note, that QD-3DT yields reasonable predictions on the KITTI dataset2 . Hence, NSG and
MARS reconstruct the dynamic objects reasonably well, butwith more blurriness and artifacts (see Fig. 4), as they do
2 In fact, following the evaluation protocol of MARS, the sequences weevaluate on are used as training sequences for QD-3DT.

MARS
Ours
MARS
Ours
Figure 5. Details Qualitative Comparison with MARS on
KITTI-360 Leaderboard.

mip-NeRF [2]
PNF [19]
MARS [40]Ours

PSNR↑ SSIM↑ LPIPS↓ mIoUcls ↑ mIoUcat ↑
21.54
0.778
0.365
48.25
67.47
22.07
0.820
0.221
73.06
84.97
23.09
0.857
0.174
23.38
0.870
0.121
72.65
85.64

Table 2. Novel View Semantic and Appearance Synthesis on
KITTI-360.

not model the optimization of the object poses. In contrast,
our method allows for reconstructing dynamic objects withsharp details, not only in cases of minor pose error on the
KITTI dataset but also on the vKITTI dataset with more severe noise.
Static Scene Leaderboard: We further evaluate our performance on the KITTI-360 leaderboard, which contains 5
static sequences. Our method achieves state-of-the-art performance on the leaderboard as in Table 2 (left), demonstrating the effectiveness of the 3D Gaussian representation in modeling complex urban scenes. As we will discussin the ablation study, incorporating the affine transform tomodel camera exposure is important for reaching high fidelity. Fig. 5 shows the qualitative comparison of our proposed method to another top-ranking method, MARS, on
the leaderboard.

4.2. Semantic and Geometric Scene Understanding
Next, we evaluate our model on various semantic and geometric scene understanding tasks on the KITTI-360 dataset.
Novel View Semantic Synthesis: Our holistic representation also enables novel view semantic synthesis. Hence,
we submit our novel view semantic synthesis performanceto the KITTI-360 leaderboard for comparison as well, see
Table 2 (right). Despite not leveraging category-level prioras done in previous work [19], our approach achieves comparable performance to the SOTA [19] as shown in Fig. 6.
3D Semantic Scene Reconstruction: While existing 2Dto-3D semantic lifting methods solely evaluate their performance in the 2D image space, we further evaluate our per-

PNF
Ours
Figure 6. Qualitative Comparison with PNF on KITTI-360
Leaderboard.

Semantic Nerfacto
Ours

acc.↓
1.508
0.233

comp.↓
24.28
0.214

mIoUcls ↑
0.055
0.505

Table 3. 3D Semantic Reconstruction on KITTI-360. Note that
all metrics are calculated in 3D space.

formance in the 3D space to examine the underlying 3D geometry. To this goal, we leverage the ground truth LiDAR
points provided by the KITTI-360 dataset for evaluation.
With each Gaussian possessing semantic information, we
can obtain a semantic point cloud by extracting the Gaussian’s center µ and its semantic label. We evaluate the geometric quality and semantic accuracy of this semantic point
cloud in Table 3. We compare our method with Semantic Nerfacto [34], a Semantic NeRF implemented using amore advanced backbone, as the state-of-the-art novel view
semantic synthesis method, PNF, in Table 2 is not opensource. For this baseline, we extract a semantic point cloudby specifying a threshold to the density field. While Semantic Nerfacto enables rendering faithful 2D semantic labels as shown in the supplement, the underlying 3D semantic point cloud is significantly worse in comparison. The
Gaussian based representation instead allows for extracting
a much more accurate semantic point cloud in comparison.

4.3. Scene Editing
Our decomposed scene representation enables various
downstream applications. Our method allows for decomposing foreground moving objects from the background as
shown in Fig. 7. Further, we can edit the scene by swapping
dynamic objects, or manipulating their rotation and translations, see Fig. 8.

4.4. Ablation Study
We conduct ablation studies on dynamic and static scenes,
respectively.
Dynamic Scene: As KITTI provides accurate 3D bounding box ground truth, we ablate the effectiveness of our unicycle model on KITTI by manually adding noise to the 3D

KITTI (5% noise)
KITTI (10% noise)
KITTI (20% noise)
PSNR↑ SSIM↑ LPIPS↓ eR ↓ et ↓ PSNR↑ SSIM↑ LPIPS↓ eR ↓ et ↓ PSNR↑ SSIM↑ LPIPS↓ eR ↓ et ↓w/o opt., w/o uni.
23.83 0.878 0.062 0.031 0.027 22.16 0.861 0.079 0.063 0.106 20.28 0.835 0.101 0.125 0.425
w/ opt., w/o uni.
24.80 0.897 0.038 0.022 0.051 22.75 0.879 0.056 0.054 0.130 20.56 0.855 0.081 0.135 0.612
w/ opt., w/ uni. (Ours) 28.78 0.928 0.023 0.017 0.022 26.66 0.908 0.032 0.037 0.035 23.59 0.875 0.061 0.081 0.176
Table 4. Ablation Study on Dynamic Scenes of KITTI.

w/o Affine transform
w/o LS
w/o LF
Ours

PSNR↑ SSIM↑ LPIPS↓ Depth ↓24.18 0.827 0.083–
24.47 0.831 0.081
0.892
24.45 0.831 0.080
1.031
24.52 0.833 0.081
0.872

Table 5. Ablation Study on Static Scenes on KITTI-360.
Background
ForegroundFigure 7. Scene Decomposition on KITTI. Our approach enables
clear decomposition of foreground and background.

Original
EditedFigure 8. Scene Editing on KITTI. Our decomposed scene representation enables replacing dynamic objects (1st row) and moving
dynamic objects around (2nd & 3rd rows).

bounding boxes and evaluate both the novel view synthesis
results and the tracking performance, see Table 4. In thisexperiment, we compare our full model to two variants, i.e.,
using the noises without optimization (w/o opt., w/o uni.),and performing naı̈ve per-frame optimization without using
the unicycle model (w/ opt., w/o uni.). The results validate
the effectiveness of the unicycle model, which obviously
improves the rendering quality and 3D tracking accuracy.
Qualitative results in Fig. 9 further verify the effectivenessof our unicycle model in enabling accuracy object reconstruction given noisy 3D bounding boxes.
Static Scene: We further study the effect of different components on three static scenes of KITTI-360 in Table 5. This
allows us to ablate design choices without mixing up the
impact of dynamic objects. The results indicate the significance of exposure modeling, which is particularly important for scenes with strong exposure variance. The semantic
and flow losses have little contribution in improving novel
view synthesis. It is rational as imposing a constraint on thesemantic or flow does not necessarily contribute to appear-

w/o opt., w/o uni.
w/ opt., w/o uni.
Ours
Figure 9. Detail Qualitative Comparison on KITTI with Noisy
Bounding Boxes.

ance. However, note that incorporating the flow supervision clearly improves the underlying geometry, since optical flow provides explicit correspondence. See supplementfor qualitative comparison.

5. Conclusion
In this paper, we present HUGS, a holistic scene representation that jointly optimizes appearance, geometry, and motion for urban scenes. This leads to state-of-the-art performance on various tasks. Our method has several limitations.
Firstly, the reconstructed dynamic objects can only rotate toa certain degree. Future work may explore category-level
prior, to enable accurate reconstruction of the full object.
Further, our model lacks control of more degrees of freedom, e.g., light editing, which could be a promising direction to explore based on the Gaussian representation.
Acknowledgements: This work is supported by NSFC under
grant 62202418, U21B2004 and the National Key R&D Programof China under Grant 2021ZD0114501. Yiyi Liao is with the Zhejiang Provincial Key Laboratory of Information Processing, Communication and Networking (IPCAN). Andreas Geiger was supported by the ERC Starting Grant LEGO-3D (850533) and the
DFG EXC number 2064/1 - project number 390727645.

References[1] Sameer Agarwal, Yasutaka Furukawa, Noah Snavely, Ian Simon, Brian Curless, Steven M. Seitz, and Richard Szeliski.
Building Rome in a day. Communications of the ACM, 54(10):105–112, 2011. 2[2] Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter
Hedman, Ricardo Martin-Brualla, and Pratul P. Srinivasan.
Mip-NeRF: A Multiscale Representation for Anti-Aliasing
Neural Radiance Fields, 2021. arXiv:2103.13415 [cs]. 2, 6,
7[3] Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P.
Srinivasan, and Peter Hedman. Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields, 2022.
arXiv:2111.12077 [cs].
[4] Jonathan T. Barron, Ben Mildenhall, Dor Verbin,
Pratul P. Srinivasan, and Peter Hedman.
Zip-NeRF:
Anti-Aliased Grid-Based Neural Radiance Fields, 2023.
arXiv:2304.06706 [cs]. 2[5] Shubhankar Borse, Ying Wang, Yizhe Zhang, and Fatih
Porikli. InverseForm: A Loss Function for Structured
Boundary-Aware Segmentation, 2021. arXiv:2104.02745[cs]. 2, 5
[6] Shubhankar Borse, Ying Wang, Yizhe Zhang, and Fatih
Porikli. InverseForm: A Loss Function for Structured
Boundary-Aware Segmentation, 2021. arXiv:2104.02745[cs]. 14[7] Yohann Cabon, Naila Murray, and Martin Humenberger. Virtual KITTI 2, 2020. arXiv:2001.10773 [cs, eess]. 5, 12[8] Xu Chen, Zijian Dong, Jie Song, Andreas Geiger, and Otmar
Hilliges. Category Level Object Pose Estimation via Neural
Analysis-by-Synthesis, 2020. arXiv:2008.08145 [cs]. 12[9] Bowen Cheng, Maxwell D. Collins, Yukun Zhu, Ting Liu,
Thomas S. Huang, Hartwig Adam, and Liang-Chieh Chen.
Panoptic-DeepLab: A Simple, Strong, and Fast Baseline for
Bottom-Up Panoptic Segmentation, 2020. arXiv:1911.10194[cs]. 2[10] Ainaz Eftekhar, Alexander Sax, Roman Bachmann, Jitendra Malik, and Amir Zamir. Omnidata: A Scalable Pipeline
for Making Multi-Task Mid-Level Vision Datasets from 3D
Scans, 2021. 2[11] Xiao Fu, Shangzhan Zhang, Tianrun Chen, Yichong Lu,
Lanyun Zhu, Xiaowei Zhou, Andreas Geiger, and Yiyi Liao.
Panoptic NeRF: 3D-to-2D Label Transfer for Panoptic Urban Scene Segmentation, 2022. arXiv:2203.15224 [cs]. 1, 4,
12[12] David Gallup, Jan-Michael Frahm, and Marc Pollefeys.
Piecewise planar and non-planar stereo for urban scene reconstruction. In 2010 IEEE Computer Society Conference
on Computer Vision and Pattern Recognition, pages 1418–
1425, San Francisco, CA, USA, 2010. IEEE. 2[13] A. Geiger, P. Lenz, and R. Urtasun. Are we ready forautonomous driving? The KITTI vision benchmark suite.
In 2012 IEEE Conference on Computer Vision and Pattern
Recognition, pages 3354–3361, Providence, RI, 2012. IEEE.
5, 12[14] Lily Goli, Cody Reading, Silvia Sellán, Alec Jacobson, andAndrea Tagliasacchi. Bayes’ Rays: Uncertainty Quantifica-tion for Neural Radiance Fields, 2023. arXiv:2309.03185[cs]. 2[15] Jianfei Guo, Nianchen Deng, Xinyang Li, Yeqi Bai,
Botian Shi, Chiyu Wang, Chenjing Ding, Dongliang
Wang, and Yikang Li.
StreetSurf: Extending Multiview Implicit Surface Reconstruction to Street Views, 2023.
arXiv:2306.04988 [cs]. 2[16] Hou-Ning Hu, Yung-Hsu Yang, Tobias Fischer, Trevor Darrell, Fisher Yu, and Min Sun. Monocular Quasi-Dense 3D
Object Tracking, 2021. arXiv:2103.07351 [cs]. 5, 6, 15[17] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler,
and George Drettakis. 3D Gaussian Splatting for Real-TimeRadiance Field Rendering. 42(4). 2, 3, 5[18] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkuehler,
and George Drettakis. 3D Gaussian Splatting for Real-TimeRadiance Field Rendering. ACM Transactions on Graphics,
42(4):1–14, 2023. 11[19] Abhijit Kundu, Kyle Genova, Xiaoqi Yin, Alireza Fathi,
Caroline Pantofaru, Leonidas Guibas, Andrea Tagliasacchi,
Frank Dellaert, and Thomas Funkhouser. Panoptic Neural
Fields: A Semantic Object-Aware Neural Scene Representation, 2022. arXiv:2205.04334 [cs]. 1, 2, 6, 7, 13[20] Florent Lafarge, Renaud Keriven, Mathieu Bredif, andHoang-Hiep Vu. A Hybrid Multiview Stereo Algorithm
for Modeling Urban Scenes. IEEE Transactions on PatternAnalysis and Machine Intelligence, 35(1):5–17, 2013. 2[21] Yiyi Liao, Jun Xie, and Andreas Geiger. KITTI-360: A
Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3D. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1–1, 2022. 5, 6, 12[22] Fan Lu, Yan Xu, Guang Chen, Hongsheng Li, Kwan-Yee
Lin, and Changjun Jiang. Urban Radiance Field Representation with Deformable Neural Mesh Primitives, 2023.
arXiv:2307.10776 [cs]. 2[23] Jonathon Luiten, Georgios Kopanas, Bastian Leibe, andDeva Ramanan. Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis, 2023. arXiv:2308.09713[cs]. 3[24] Ricardo Martin-Brualla, Noha Radwan, Mehdi S. M. Sajjadi,
Jonathan T. Barron, Alexey Dosovitskiy, and Daniel Duckworth. NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections, 2021. arXiv:2008.02268 [cs].
2, 4[25] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik,
Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF:
Representing Scenes as Neural Radiance Fields for View
Synthesis, 2020. arXiv:2003.08934 [cs]. 2[26] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Transactions on Graphics, 41(4):1–15, 2022. 2[27] Julian Ost, Fahim Mannan, Nils Thuerey, Julian Knodt, andFelix Heide. Neural Scene Graphs for Dynamic Scenes,
2021. arXiv:2011.10379 [cs]. 1, 2, 5, 6, 13, 14[28] Luigi Piccinelli, Christos Sakaridis, and Fisher Yu. iDisc: Internal Discretization for Monocular Depth Estimation, 2023.
arXiv:2304.06334 [cs]. 2

[29] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J
Guibas. PointNet++: Deep Hierarchical Feature Learning on
Point Sets in a Metric Space. In Advances in Neural Information Processing Systems. Curran Associates, Inc., 2017.
2[30] Konstantinos Rematas, Andrew Liu, Pratul P. Srinivasan, Jonathan T. Barron, Andrea Tagliasacchi, Thomas
Funkhouser, and Vittorio Ferrari. Urban Radiance Fields,
2021. arXiv:2111.14643 [cs]. 1, 2, 4[31] Damien Robert, Bruno Vallet, and Loic Landrieu. Learning Multi-View Aggregation In the Wild for Large-Scale 3D
Semantic Segmentation. In 2022 IEEE/CVF Conference on
Computer Vision and Pattern Recognition (CVPR), pages
5565–5574, New Orleans, LA, USA, 2022. IEEE. 2[32] Johannes L. Schonberger and Jan-Michael Frahm. Structurefrom-Motion Revisited. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4104–
4113, Las Vegas, NV, USA, 2016. IEEE. 2[33] Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul P. Srinivasan, Jonathan T. Barron, and Henrik Kretzschmar. Block-NeRF: Scalable Large
Scene Neural View Synthesis, 2022. arXiv:2202.05263 [cs].
2[34] Matthew Tancik, Ethan Weber, Evonne Ng, Ruilong Li,
Brent Yi, Justin Kerr, Terrance Wang, Alexander Kristoffersen, Jake Austin, Kamyar Salahi, Abhik Ahuja, David
McAllister, and Angjoo Kanazawa. Nerfstudio: A Modular Framework for Neural Radiance Field Development. In
Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Proceedings, pages
1–12, 2023. arXiv:2302.04264 [cs]. 2, 6, 7, 13, 15[35] Andrew Tao, Karan Sapra, and Bryan Catanzaro. Hierarchical Multi-Scale Attention for Semantic Segmentation, 2020.
arXiv:2005.10821 [cs]. 2[36] Haithem Turki, Jason Y. Zhang, Francesco Ferroni, and Deva
Ramanan. SUDS: Scalable Urban Dynamic Scenes, 2023.
arXiv:2303.14536 [cs]. 2[37] Suhani Vora, Noha Radwan, Klaus Greff, Henning Meyer,
Kyle Genova, Mehdi S. M. Sajjadi, Etienne Pot, Andrea
Tagliasacchi, and Daniel Duckworth. NeSF: Neural Semantic Fields for Generalizable Semantic Segmentation of 3D
Scenes, 2021. arXiv:2111.13260 [cs]. 2[38] Peng Wang, Yuan Liu, Zhaoxi Chen, Lingjie Liu, Ziwei
Liu, Taku Komura, Christian Theobalt, and Wenping Wang.
F$ˆ{2}$-NeRF: Fast Neural Radiance Field Training withFree Camera Trajectories, 2023. arXiv:2303.15951 [cs]. 2[39] Felix Wimbauer, Nan Yang, Christian Rupprecht, and DanielCremers. Behind the Scenes: Density Fields for Single View
Reconstruction, 2023. arXiv:2301.07668 [cs]. 2[40] Zirui Wu, Tianyu Liu, Liyi Luo, Zhide Zhong, Jianteng
Chen, Hongmin Xiao, Chao Hou, Haozhe Lou, Yuantao
Chen, Runyi Yang, Yuxin Huang, Xiaoyu Ye, Zike Yan,
Yongliang Shi, Yiyi Liao, and Hao Zhao. MARS: An
Instance-aware, Modular and Realistic Simulator for Autonomous Driving, 2023. arXiv:2307.15058 [cs]. 1, 2, 5,
6, 7, 13, 14[41] Haofei Xu, Jing Zhang, Jianfei Cai, Hamid Rezatofighi,
Fisher Yu, Dacheng Tao, and Andreas Geiger. Unifying

Flow, Stereo and Depth Estimation, 2023. arXiv:2211.05783[cs]. 5[42] Haofei Xu, Jing Zhang, Jianfei Cai, Hamid Rezatofighi,
Fisher Yu, Dacheng Tao, and Andreas Geiger. Unifying
Flow, Stereo and Depth Estimation, 2023. arXiv:2211.05783[cs]. 2[43] Zhen Xu, Sida Peng, Haotong Lin, Guangzhao He, Jiaming Sun, Yujun Shen, Hujun Bao, and Xiaowei Zhou.
4K4D: Real-Time 4D View Synthesis at 4K Resolution,
2023. arXiv:2310.11448 [cs]. 3[44] Bangbang Yang, Yinda Zhang, Yinghao Xu, Yijin Li, Han
Zhou, Hujun Bao, Guofeng Zhang, and Zhaopeng Cui.
Learning Object-Compositional Neural Radiance Field for
Editable Scene Rendering. In 2021 IEEE/CVF International
Conference on Computer Vision (ICCV), pages 13759–
13768, Montreal, QC, Canada, 2021. IEEE. 2[45] Jiawei Yang, Boris Ivanovic, Or Litany, Xinshuo Weng,
Seung Wook Kim, Boyi Li, Tong Che, Danfei Xu, Sanja
Fidler, Marco Pavone, and Yue Wang.
EmerNeRF:
Emergent Spatial-Temporal Scene Decomposition via SelfSupervision, 2023. arXiv:2311.02077 [cs]. 2[46] Ze Yang, Yun Chen, Jingkang Wang, Sivabalan Manivasagam, Wei-Chiu Ma, Anqi Joyce Yang, and Raquel Urtasun. UniSim: A Neural Closed-Loop Sensor Simulator. 1,
2[47] Ziyi Yang, Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing
Zhang, and Xiaogang Jin. Deformable 3D Gaussians forHigh-Fidelity Monocular Dynamic Scene Reconstruction,
2023. arXiv:2309.13101 [cs]. 3[48] Zeyu Yang, Hongye Yang, Zijie Pan, Xiatian Zhu, and Li
Zhang. Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting, 2023.
arXiv:2310.10642 [cs]. 3[49] Zehao Yu, Songyou Peng, Michael Niemeyer, Torsten Sattler, and Andreas Geiger. MonoSDF: Exploring Monocular
Geometric Cues for Neural Implicit Surface Reconstruction,
2022. arXiv:2206.00665 [cs]. 2[50] Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The Unreasonable Effectiveness of
Deep Features as a Perceptual Metric. In 2018 IEEE/CVF
Conference on Computer Vision and Pattern Recognition,
pages 586–595, Salt Lake City, UT, 2018. IEEE. 6, 12[51] Xiaoshuai Zhang, Abhijit Kundu, Thomas Funkhouser,
Leonidas Guibas, Hao Su, and Kyle Genova.
Nerflets: Local Radiance Fields for Efficient Structure-Aware
3D Scene Representation from 2D Supervisio, 2023.
arXiv:2303.03361 [cs]. 1, 2[52] Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, andAndrew J. Davison. In-Place Scene Labelling and Understanding with Implicit Scene Representation, 2021.
arXiv:2103.15875 [cs]. 2, 4, 12[53] Wojciech Zielonka, Timur Bagautdinov, Shunsuke Saito,
Michael Zollhöfer, Justus Thies, and Javier Romero. Drivable 3D Gaussian Avatars, 2023. arXiv:2311.08581 [cs]. 3[54] M. Zwicker, H. Pfister, J. Van Baar, and M. Gross. EWA
splatting. IEEE Transactions on Visualization and Computer
Graphics, 8(3):223–238, 2002. 11

Appendix
In this appendix, we begin by discussing implementation details in Appendix A, which includes information about our
3D Gaussian, metrics, and the training and inference processes. We then describe the datasets used in our experiments inAppendix B. Appendix C provides information about the baselines we compare with. Finally, Appendix D contains additional
experiment results.

A. Implementation
In this section, we begin by discussing our 3D Gaussian details, encompassing semantic, opacity and depth implementation(Appendix A.1). Subsequently, we discuss the difference between 3D softmax and 2D softmax in 3D Semantic Scene
Reconstruction (Appendix A.2). Finally, we elucidate the evaluation metrics we utilize (Appendix A.3). Our source code
will be released.

A.1. 3D Gaussian Details
Following [18], each Gaussian has the following attributes: rotation (Rg ∈ R3×3 ), scale (Sg ∈ R3×1 ), opacity (α) andspherical harmonics (SH). The corresponding 3D covariance matrix Σ ∈ R3×3 can be calculated using the following
formula:Σ = Rg Sg STg RTg(15)
When provided with a viewing transformation W ∈ R3×3 and the Jacobian of the affine approximation of the projective
transformation J ∈ R3×3 , the covariance matrix Σ′ ∈ R3×3 in camera coordinates can be expressed as:
Σ′ = JWΣWT JT(16)

Following EWA splatting [54], we can skip the third row and column of Σ′ to obtain a 2 × 2 covariance matrix with the samestructure and properties. For brevity, we still use the notation Σ′ ∈ R2×2 to denote the 2D covariance matrix.
By considering the projected 3D Gaussian center µ ∈ R2×1 and an arbitrary point x ∈ R2×1 on camera coordinates, theopacity α′ of x contributed by this 3D Gaussian can be computed as follows:


1
′ −1
T
′
(17)
α = α exp − (x − µ) (Σ ) (x − µ)2
The color c of each Gaussian can be computed based on the view direction and its corresponding spherical harmonics (SH).
Given a set of sorted 3D Gaussians N along the ray, we obtain the accumulated color via volume rendering:
π:

C=

X

ci αi′i−1Y(1 − αj′ )

(18)

j=1

i∈N

The same volume rendering technique can be applied to obtain semantic S, depth D and optical flow F. With the given
semantic feature si , depth value di , and Gaussian motion fi relative to the camera pose, we can define the semantic rendering,
depth rendering, and flow rendering as follows:S=

X

softmax(si )αi′

Xdi αi′

X
i∈N

(19)

i−1
Y

(1 − αj′ )(20)(1 − αj′ )

(21)

j=1

i∈N

F=

(1 − αj′ )

j=1

i∈N

D=

i−1
Y

fi αi′i−1
Y

j=1

Note that all the projections and volume rendering techniques mentioned are implemented in CUDA. Calculating the projected 2D opacity α′ on each pixel and sorting Gaussians based on their distances from the camera takes the majority ofcomputations in the rendering process. These computations need to be performed only once for rendering all modalities, thus
maintaining the real-time rendering property of the original 3D Gaussian Splatting.

A.2. 3D Semantic Scene Reconstruction
We utilize Eq. (19), referred to as 3D softmax, to render semantic maps. This is in contrast to most existing NeRF-based
semantic reconstruction methods that perform softmax to the accumulated 2D logits [11, 52], described in Eq. (22), referred
to as 2D softmax. The fundamental difference between these two rendering techniques lies in the fact that 3D softmax
normalizes the logits of each 3D point. This normalization process helps prevent a single point with a significantly high logit
value from imposing an overwhelming influence on the overall volume rendering outcome. On the other hand, it also prevents
placing 3D points of low logit values in empty space. As a result, 3D softmax is effective in reducing floaters and enhancing
the geometry of the reconstruction results. In Appendix D.3, we present a comprehensive analysis of the qualitative andquantitative comparison results between these two rendering methods.


i−1
X
Y
si αi′(1 − αj′ )
(22)
S2D norm = softmax 
i∈N

j=1

In the following sections, we refer to our default setting obtained by Eq. (19) as S3D norm .

A.3. Metrics
Novel View Appearance Synthesis: To assess the quality of novel view appearance synthesis, we utilize the Peak Signalto-Noise Ratio (PSNR), Structural Similarity Index (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS) [50]following the common practice.
Novel View Semantic Synthesis: Following KITTI-360 [21], we evaluate the quality of novel view semantic synthesis via
the mean Intersection over Union (mIoU) metric.
3D Semantic Reconstruction: We evaluate 3D semantic reconstruction quality by extracting a 3D semantic point cloudand comparing it with the ground truth LiDAR points. We evaluate both geometric and semantic metrics in the 3D space.
Specifically, we evaluate geometric reconstruction quality by measuring the accuracy (acc.) and completeness (comp.). Accuracy measures the average distance from reconstructed points to the nearest LiDAR point, while completeness measures
the average distance from LiDAR points to the nearest reconstructed points. In order to measure the semantic quality of
the reconstructed point cloud, we map the predicted 3D semantics to the LiDAR points. Concretely, for each point in the
LiDAR point cloud, we identify its closest counterpart in the predicted semantic point cloud and allocate a semantic label
based on this nearest neighbor. The assigned semantic labels of all LiDAR points are then compared with the 3D semantic
segmentation ground truth provided by KITTI-360, evaluated via the mIoU metric. Note that we only use the LiDAR point
clouds for evaluation.
3D Tracking: To demonstrate the effectiveness of our model in rectifying noisy 3D tracking results, we evaluate theaccuracy of predicted poses compared to ground truth poses in our ablation study. Considering the rotation and translation
parameters of a ground truth bounding box denoted as R̂ and t̂, respectively, and the corresponding parameters of predicted
poses, represented as R and t, we employ two metrics for this evaluation following [8]: eR quantifies the rotation accuracy,
while et assesses the translation accuracy as follows
T r(R̂ · R−1 ) − 1
2
et = ∥t̂ − t∥2eR = arccos

(23)
(24)

where T r represents the trace of a matrix.
Depth Estimation: In our ablation study, we evaluate the depth estimation quality of our different variants. This is achieved
by first projecting the LiDAR points acquired at the same frame to the 2D image space, followed by measuring the L2 distance
between the projected LiDAR depth and our method. Considering the projected LiDAR depth is sparse, our assessment
focuses solely on pixels with valid LiDAR projections when calculating the L2 distance.

B. DataIn this section, we present details of datasets on which we conducted our experiments, including KITTI [13], Virtual KITTI
2 (vKITTI) [7] and KITTI-360 [21].

Speed (ms)

Pre.
6.25

+ π RGB8.13 (+1.88)+ Affine
8.54 (+0.41)+ π Semantic
9.70 (+1.16)+ π Flow
10.17 (+0.47)

Table 6. Time consumption breakdown of our method.

KITTI: Following NSG [27] and MARS [40], we select frames 140 to 224 from Scene02 and frames 65 to 120 from Scene06
on KITTI for conducting our experiments.
vKITTI: Virtual KITTI 2 is a synthetic dataset that closely resembles the scenes present in KITTI. In line with the settings
outlined in NSG and MARS, we conduct experiments on exactly the same frames from Scene02 and Scene06.
KITTI-360: In addition, we perform experiments on KITTI-360, encompassing both static and dynamic scenes. For thetasks of novel view synthesis and novel semantic synthesis on the leaderboard, we conduct experiments on the sequences
provided by the official dataset. Furthermore, we explore dynamic scenes, such as frames 11322 to 11381 from sequence 00,
as showcased in our teaser.

C. Baselines
In this section, we discuss the baselines against which we compare our approach, including NSG [27], MARS [40], PNF [19],and Semantic Nerfacto [34].
NSG: NSG is the pioneering method that introduces the decomposition of dynamic scenes into static background anddynamic foreground components. They propose a learned scene graph representation that enables efficient rendering ofnovel scene arrangements and viewpoints. However, the official source code provided by NSG often encounters issues whentraining on KITTI Scene02. Therefore, we utilize the version implemented by the authors of MARS, which is more stable
and yields slightly improved results compared to the original version.
MARS: We utilize the latest version of the code provided by the official MARS repository. This latest version incorporates
bug fixes and includes additional training iterations, resulting in improved performance. In fact, the updated version achieves
a notable improvement of 3 to 4 dB on PSNR compared to the numbers reported in the original paper.
PNF: Since PNF is not open-source, we directly compare our method to their submission on the KITTI-360 leaderboard
regarding novel view appearance & semantic synthesis. To the best of our knowledge, PNF is the only work that considers
the optimization of noisy 3D bounding boxes of dynamic objects. In our ablation study, we conduct a naı̈ve baseline that
optimizes the 3D bounding boxes of each frame independently, which can be considered as a re-implementation of PNF’s
bounding box optimization in our framework.
Semantic Nerfacto: For the evaluation of 3D semantic point cloud geometry, we compare our results with Semantic Nerfacto [34] as an alternative to PNF [19]. Nerfacto [34] is an integration of several successful methods that demonstrate strong
performance on real data. It incorporates camera pose refinement, per-image appearance embedding, proposal sampling,
scene contraction, and hash encoding within its pipeline. Additionally, Nerfacto includes a semantic head in its framework,
enabling the generation of meaningful semantic maps, as demonstrated in Appendix D.2.

D. Additional Experiment Results
D.1. Time Consumption Breakdown
Tab. 6 shows our detailed runtime breakdown as various components are incrementally enabled. Preparation (Pre.) contains
operations like tile partition and Gaussian sorting. π denotes volume rendering, and affine denotes affine transform. Other
components like unicycle model, dynamic decomposition, and depth rendering are excluded as they hardly consume any
additional time.

D.2. Additional Comparison Experiments
Dynamic Scene with GT 3D Bounding Boxes: Despite not being our primary focus, we additionally provide a comparison
with NSG and MARS using ground truth 3D trackings. In this setting, our approach demonstrates superior performanceacross all test scenes, see Tab. 7.

NSG [27]MARS [40]Ours

KITTI Scene02
PSNR↑ SSIM↑ LPIPS↓
22.51
0.653
0.397
22.95
0.728
0.145
25.89
0.829
0.092

KITTI Scene06
PSNR↑ SSIM↑ LPIPS↓
23.38
0.717
0.243
27.01
0.883
0.062
28.90
0.925
0.016

vKITTI Scene02
PSNR↑ SSIM↑ LPIPS↓
23.50
0.718
0.352
29.80
0.950
0.034
30.73
0.955
0.018

vKITTI Scene06
PSNR↑ SSIM↑ LPIPS↓
26.42
0.811
0.170
32.71
0.959
0.023
33.31
0.963
0.010

Table 7. Novel View Appearance on Dynamic Scenes with ground truth 3D trackings.

Semantic Nerfacto

Ours

Pseudo GT

Figure 10. Qualitative Comparison with Nerfacto on 2D space. The Pseudo GT column represents the semantic maps that are predictedby [6] on GT RGB images.

Semantic Nerfacto

Ours

Figure 11. Qualitative Comparison with Nerfacto on 3D space. The semantic point cloud extracted from Semantic Nerfacto struggles tofaithfully represent the geometry.

KITTI 02
KITTI 06

QD-3DT
Ours
QD-3DT
Ours

eR ↓
0.027
0.018
0.017
0.012

et ↓
0.215
0.108
0.046
0.033

Table 8. Qualitative Comparison with a tracking method,
QD-3DT [16], on two sequences.

Ours w/ S2D norm
Ours w/ S3D norm

Seq01 mIoUcls ↑
0.427
0.544

Seq02 mIoUcls ↑
0.363
0.452

Figure 12. Pose comparison with QD-3DT.
Seq03 mIoUcls ↑
0.416
0.520

Average mIoUcls ↑
0.402
0.505

Table 9. Comparison on 3D and 2D Semantic Softmax on KITTI-360.
KITTI-360 Scene00
KITTI-360 Scene01
KITTI-360 Scene02
Average
PSNR↑ SSIM↑ LPIPS↓ PSNR↑ SSIM↑ LPIPS↓ PSNR↑ SSIM↑ LPIPS↓ PSNR↑ SSIM↑ LPIPS↓
Random
20.84 0.784 0.150
19.40 0.705 0.171
22.55 0.800 0.136
20.93 0.763 0.457
LiDAR
25.64 0.856 0.070
22.88 0.784 0.089
24.04 0.836 0.080
24.19 0.825 0.080
COLMAP 26.23 0.863 0.069
22.94 0.794 0.096
24.38 0.843 0.077
24.52 0.833 0.081
Table 10. Quantitative Comparison with different initialization.

Details of Comparison with Semantic Nerfacto: While Semantic Nerfacto excels at rendering meaningful novel view
semantic images (as seen in Fig. 10), Fig. 11 shows it struggling to accurately reconstruct correct geometry. Following thecommon practice of NeRF-based semantic reconstruction methods [34], we apply 2D softmax to Semantic Nerfacto. when
we attempted to apply the 3D Softmax technique to Nerfacto, it did not yield better results compared to using 2D softmax.
The results can be attributed to the incorrect of Nerfacto’s 3D geometry. Instead of adjusting 2D logits with large-scale
logits in 3D, the use of 3D softmax prevents the “cheating” approach by normalizing logits in 3D space. However, thisnormalization requirement necessitates sufficiently accurate geometry for satisfactory results.
Comparisons with Tracking Methods: To further compare with off-the-shelf tracking methods, we show the performanceof QD-3DT [16] and our optimized pose initialized with [16] in Tab. 8 and qualitatively illustrate the poses of one vehicle inFig. 12. Our method consistently improves [16] across two KITTI scenes.

D.3. Additional Ablation Experiments
3D and 2D Semantic Softmax: We provide more 3D and 2D semantic logits softmax comparison in Fig. 13 and Tab. 9.
As can be seen, normalizing semantic logits in 3D space leads to notable qualitative and quantitative improvement comparedto 2D space normalization.
Improvements on Geometry: We now qualitatively examine how the optical flow loss LF and the semantic loss LS impact
the geometry, as shown in Fig. 14 and Fig. 15. Both figures reveal that incorporating either the semantic loss or the opticalflow loss improves the underlying geometry. While the impact of the semantic loss on geometry may be less evident, theoptical flow clearly enhances geometric accuracy. This improvement is rationalized by the fact that optical flow guides
correspondences across neighboring frames. It’s important to note that when the semantic loss LS is active, the sky region of
the depth maps in Fig. 14 is set to infinite.
Effects of Initialization: We conduct a thorough comparison of the results obtained through different initialization strategies. In particular, we consider random initialization and COLMAP-based initialization. To further investigate whether
adopting LiDAR point cloud for initialization is helpful in urban scenes, we further consider LiDAR point clouds as initialization. We report the quantitative and qualitative comparison in Tab. 10 and Fig. 16, respectively. We observe thatboth LiDAR and COLMAP initialization outperform random initialization. Interestingly, the COLMAP-based initializationeven shows a slight advantage over the LiDAR-based one. This could be attributed to the presence of points in the LiDAR

Ours w/ S2D norm

Ours w/ S3D norm

Figure 13. Qualitative Comparison of 3D and 2D softmax results. Note that normalizing semantic logits in 3D space (Ours w/ S3D norm )clearly reduces floaters and yields better 3D semantic reconstruction than the 2D normalization counterpart (Ours w/ S2D norm ).

point clouds that remain unobserved in any training views, leading to artifacts in test viewpoints. Furthermore, COLMAP
improves the quality of objects located at far distances, which cannot be accurately captured by LiDAR. These findings underscore the potential for achieving high-fidelity novel view synthesis in urban scenes based solely on RGB images. In our
main experiments, we adopt the COLMAP-based initialization by default.

D.4. Visualization of Optimization Progress
We present the visualization of the optimization progress for both the noisy bounding boxes and the background semantic
point cloud in Fig. 17. Using noisy 3D bounding boxes as input, our approach optimizes both the background and the posesof the bounding boxes simultaneously. As evident, the application of physical constraints derived from the unicycle model
results in a smooth trajectory for the bounding boxes.

w/o LS , w/o LF

w/ LS , w/o LF

w/o LS , w/ LF

w/ LS , w/ LF

Figure 14. Qualitative Comparison on depth. In the presence of the semantic loss LS (2nd and 4th columns), we set the sky region’s
depth infinite based on its semantic label. Note that the activation of either the semantic loss LS (2nd column) or the optical loss LF(3rd column) yields enhancements in geometry, e.g., the left car in the bottom row, with the improvement in optical flow loss being moreevident.

w/o LS , w/o LF

w/ LS , w/o LF

w/o LS , w/ LF

w/ LS , w/ LF

Figure 15. Qualitative Comparison on optical flow. While 3D Gaussians can enable the rendering of optical flow without additional
supervision on semantic or optical flow, the rendered flow maps exhibit clear artifacts (1st column). These artifacts are particularly
noticeable on the cars and the ground. Interestingly, the incorporation of semantic supervision LS mitigates the artifacts to some extent(2nd column). Additionally, introducing pseudo-optical flow supervision LF contributes to further improvement in the optical flow results(3rd and 4th columns).

Random

LiDAR

COLMAP

Figure 16. Qualitative Comparison with different initialization strategies. The superiority of both LiDAR-based and COLMAP-based
initialization over random initialization is evident. Random initialization occasionally results in significant artifacts, as illustrated by theright building in the 1st row. LiDAR-based initialization, while generally effective, introduces artifacts in areas very close to the ego car,such as the bottom right corner of the 4th-6th rows. These regions typically encompass LiDAR points unseen by any training views. The
COLMAP-based initialization further demonstrates an improvement over the LiDAR-based approach in distant regions, exemplified by thetrees in the 1st row.

10 steps
2000 steps
5000 steps
Figure 17. Visualization of Optimization Progress. Our method jointly optimizes the static background and the trajectory of the dynamic
foreground objects. By integrating physical constraints using the unicycle model, our method allows for recovering a smooth trajectoryfrom noisy 3D bounding boxes. To prevent visual clutter, we exclude point clouds of the dynamic object and only visualize the boundingboxes.