3D StreetUnveiler with Semantic-aware 2DGS - a simple baseline

## Abstract

Abstract Unveiling an empty street from crowded observations captured by in-car cameras is crucial for autonomous driving. However, removing all temporarily static objects, such as stopped vehicles and standing pedestrians, presents a significant challenge. Unlike object-centric 3D inpainting, which relies on thorough observation in a small scene, street scene cases involve long trajectories that differ from previous 3D inpainting tasks. The camera-centric moving environment of captured videos further complicates the task due to the limited degree and time duration of object observation.
To address these obstacles, we introduce StreetUnveiler to reconstruct an empty street.
StreetUnveiler learns a 3D representation of the empty street from crowded observations.
Our representation is based on the hard-label semantic 2D Gaussian Splatting (2DGS) for its scalability and ability to identify Gaussians to be removed.
We inpaint rendered image after removing unwanted Gaussians to provide pseudo-labels and subsequently re-optimize the 2DGS.
Given its temporal continuous movement, we divide the empty street scene into observed, partial-observed, and unobserved regions, which we propose to locate through a rendered alpha map.
This decomposition helps us to minimize the regions that need to be inpainted.
To enhance the temporal consistency of the inpainting, we introduce a novel time-reversal framework to inpaint frames in reverse order and use later frames as references for earlier frames to fully utilize the long-trajectory observations.
Our experiments conducted on the street scene dataset successfully reconstructed a 3D representation of the empty street. The mesh representation of the empty street can be extracted for further applications. The project page and more visualizations can be found at: https://streetunveiler.github.io

## 1 Introduction

Accurate 3D reconstruction of an empty street scene from an in-car camera video is crucial for autonomous driving. It provides reliable digital environments that simulate real-world street scenarios.
Although this is an important task, it is seldomly studied in previous works because of its challenging nature in the following aspects:
(1) Lack of ground truth data for pre-training inpainting models specialized for street scenes;
(2) The camera-centric moving captures objects from limited angles and for brief periods;
(3) The long trajectory of in-car camera videos leads to objects appearing and disappearing at different time points, complicating object removal.

But there still exists a blessing we can take from the long trajectory moving-forward nature. As the car moves forward, objects that disappear from the later frame will only be visible in previous video frames. This gives a hint about maintaining the temporal consistency of the same regions.

To address the challenge of reconstructing an empty street, we introduce StreetUnveiler,
a reconstruction method targeting unveiling the empty representation of long-trajectory street scenes.
StreetUnveiler involves several key steps. First, it reconstructs the observed 3D representation and identifies unobserved regions occluded by objects. Then, it uses a time-reversal inpainting framework to consistently inpaint these unobserved regions as pseudo labels. Finally, it re-optimizes the 3D representation based on these pseudo labels.

Figure: Figure 1: We achieve accurate reconstruction from in-car camera videos. With the aid of the proposed hard-label semantic 2D Gaussian Splatting and time-reversal inpainting framework, we remove the unwanted objects with satisfactory appearance and geometry of occluded regions.
Refer to caption: x1.png

Figure: Figure 2: (a) The mask of the whole unwanted object; (b) Inpainting with (a) mask; (c) Generate the inpainting mask through a rendered alpha map. The pixel with a low alpha value is selected as an inpainted pixel; (d) Inpainting with the generated (c) mask.
Refer to caption: x2.png

StreetUnveiler first reconstructs the original parked-up street with Gaussian Splatting (GS) due to its scalability and editability. However, as is illustrated in Fig. 2, inpainting with the naïve object mask (orange mask) often results in blurring and loss of details in large inpainted regions, which is a common issue in the previous works [, , , , ]. Generating masks for completely unobservable regions (blue mask) that are invisible from any viewpoint remains a challenge. Recent work [] requires user-provided masks, which is impractical for long trajectories. Moreover, the messy appearance of these regions after removing the Gaussians makes it difficult to use methods like SAM [].
To address the difficulty of finding an ideal inpainting mask, we propose to generate the mask through the rendered alpha map and reconstruct the scene using a hard-label semantic 2DGS [] instead of 3DGS []
. 2DGS has a high opacity value for Gaussians, resulting in low alpha values in completely unobservable regions. A semantic distortion loss and a shrinking loss are employed to further reduce the rendered alpha values of the completely unobservable regions.
This approach automatically generates masks for unobservable regions without user input, leading to better inpainting results.

Furthermore, we propose a time-reversal inpainting framework to enhance the temporal consistency of inpainting results in completely unobservable regions. By inpainting the video frames in reverse order, we use the later frame as a reference to inpaint the earlier frame. When the video is played in reverse, the object in the later frame will transition only from near to far in the camera view as the camera moves away from the object in a reversed time-space. This method uses a high-to-low-resolution guiding approach instead of filling an area larger than the reference region, as in the low-to-high-resolution approach, which results in more consistent inpainting.
Finally, the inpainted pixels are used as pseudo labels to guide the re-optimization of 2DGS. This enables our method to learn a scalable 2DGS model that represents an empty street while preserving the appearance integrity of regions visible in other views.

Our contribution can be summarized as follows:

- •
We propose representing the street as hard-label semantic 2DGS, optimizing the 3D scene with semantic guidance for scalable representation and improved instance decoupling.
- •
We use a rendered alpha map to locate completely unobservable regions and apply a semantic distortion loss and a shrinking loss to create a reasonable inpainting mask for these regions.
- •
We introduce a novel time-reversal inpainting framework for long-trajectory scenes, enhancing the temporal consistency of inpainting results for re-optimization. Experiments show that our method can reconstruct an empty street from in-car camera video containing obstructive elements.

## 2 Related work

Neural scene representation and reconstruction. The use of neural radiance fields (NeRF)[] to represent 3D scenes inspired a lot of follow-up work based on the original approach. Some works[, , , ] explore explicit representations such as low-rank matrices, hash grids, or voxel grids to increase the model capacity of original MLPs. Some works explored multiple separate MLPs [, , ] to represent instances and backgrounds separately. However, these scale-up strategies are complicated to implement at the scale of street scenes. Existing works [, , , , , , , , , , , , ] explored mesh-based, primitive-based, or grid-based representations for large-scale street scenes. However, both grid-based representation [] and mesh-based representation [] may be constrained by their limited topology, making it hard to decouple the scene into separate instances. Recent advances in point-based rendering techniques [, , , ] can achieve both high-quality and fast rendering speed. The point-based nature of Gaussian Splatting enables scalability for street scenes. While recent works [, , , , ] have explored the reconstruction of large-scale scenes using Gaussian Splatting, our work focuses on the unveiling stage of a street scene, which is more important for autonomous driving and more challenging.

3D scene manipulation and inpainting. Early works [, , , , , , , , , ] explored street scene editing by leveraging single-view or multi-view image inpainting networks. With the rapid development of Neural Scene Representation, editing a 3D scene has been explored by lots of works [, , , , , , , ]. Edit-NeRF [] pioneered shape and color editing of neural fields using latent codes. Subsequent works [, , , ] utilized CLIP models to provide editing guidance from text prompts or reference images. Recent works [, , , , , , , , , ] also explored 2D stylization and inpainting techniques, utilizing pretrained Diffusion Priors [] for editing 3D scenes. Specifically, [, , , ] investigated these approaches in collaboration with Gaussian Splatting. Unlike them, our work focuses on street scene object removal and empty street reconstruction, which is more challenging.

Image and video inpainting. Image inpainting []
aims to fulfill the missing region within an image.
Standard approaches included
GAN-based methods [, ], attention-based methods [, ], transformer-based methods [, ], and more recently, diffusion-based methods [].
Control-Net [] enabled generating images with additional conditions on the frozen diffusion models.
Recently, LeftRefill [] learned to guide the frozen diffusion inpainting models with extra conditions of the reference image, enabling multi-view inpainting on the frozen diffusion model.
However, these image inpainting methods mainly focused on the static scenario.
Video inpainting considers the temporal consistent inpainting in the continuous image sequence, utilizing approaches like 3D CNN [, ], temporal shifting [], flow guidance [, , ], temporal attentions [], to name a few.
However, these video inpainting methods hardly considered the long trajectory movement of cameras.
In contrast, in our paper, we focus on the inpainting of large-scale street scenes.
Furthermore, the 2DGS representation used in our paper enables the free-view rendering of the inpainted video.

## 3 Problem formulation

Given in-car camera videos and the Lidar data of a parked-up street, our goal is to remove all temporarily static objects in the street, like stopping vehicles and standing pedestrians, and finally reconstruct an empty street. This task, named as Street Unveiling, is
to reconstruct scenes devoid of these static obstacles, providing an empty representation of the street environment.
Such representations are mainly represented by 3D models for free-view rendering.
This task holds significant implications for autonomous driving systems, urban planning, and scene understanding applications.

Street Unveiling shares some similarities with related tasks but cannot be addressed using existing approaches.
(1) 3D reconstruction primarily involves modeling a primary image or scene with an object-centric camera. In contrast, Street Unveiling focuses on the background, aiming to remove foreground objects to reveal an empty street. The absence of ground truth further differentiates it from standard 3D reconstruction tasks.
(2) Video inpainting typically deals with videos captured by fixed or minimally moving cameras, featuring one or a few central objects. Conversely, Street Unveiling involves long camera trajectories without central objects.
These distinctions require different capabilities and novel methods to address the unique challenges of Street Unveiling.

## 4 Semantic street reconstruction

We opt for 2D Gaussian Splatting  [] (2DGS) as our scene representation for its rendering speed and editability. We first introduce the 2DGS in Sec. 4.1. Subsequently, we elaborate our algorithm tailored for street unveiling using 2DGS in Sec. 4.2 and Sec. 4.3.

### 4.1 Preliminary: 2D Gaussian Splatting

Our reconstruction stage builds upon the state-of-the-art point-based renderer with the splendid geometry performance, 2DGS []. 2DGS is defined by several key components: the central point $\mathbf{p}_{k}$, two principal tangential vectors $\mathbf{t}_{u}$ and $\mathbf{t}_{v}$ that determine its orientation, and a scaling vector $\mathbf{S}=(s_{u},s_{v})$ controlling the variances of the 2D Gaussian distribution.

2D Gaussian Splatting represents the scene’s geometry as a set of 2D Gaussians. A 2D Gaussian is defined in a local tangent plane in world space, parameterized as follows:

$$ $\displaystyle P(u,v)$ $\displaystyle=\mathbf{p}_{k}+s_{u}\mathbf{t}_{u}u+s_{v}\mathbf{t}_{v}v.$ (1) $$

For the point $\mathbf{u}=(u,v)$ in $uv$ space, its 2D Gaussian value can then be evaluated using the standard Gaussian function:

$$ $\mathcal{G}(\mathbf{u})=\exp\left(-\frac{u^{2}+v^{2}}{2}\right).$ (2) $$

The center $\mathbf{p}_{k}$, scaling $(s_{u},s_{v})$, and the rotation $(\mathbf{t}_{u},\mathbf{t}_{v})$ are learnable parameters.
Each 2D Gaussian primitive has opacity $\alpha$ and view-dependent appearance $\mathbf{c}$ with spherical harmonics.
For volume rendering, Gaussians are sorted according to their depth value and composed into an image with front-to-back alpha blending:

$$ $\mathbf{c}(\mathbf{x})=\sum_{i=1}\mathbf{c}_{i}\alpha_{i}\mathcal{G}_{i}( \mathbf{u}(\mathbf{x}))\prod_{j=1}^{i-1}(1-\alpha_{j}\mathcal{G}_{j}(\mathbf{u }(\mathbf{x}))).$ (3) $$

where $\mathbf{x}$ represents a homogeneous ray emitted from the camera and passing through $uv$ space.

### 4.2 2DGS for street scene reconstruction

2DGS [] features for its accurate geometry reconstruction of the object surface.
However, the application of 2DGS to reconstruct objects devoid of surfaces, such as the sky in an open-air street scene, remains unexplored. We aim to reconstruct the street scene as a radiance field and semantic field using 2DGS. More details about radiance field reconstruction are included in the supplementary.

Learning 2D Gaussians with semantic guidance.
We aim to augment the radiance field of street scenes with editability. Inspired from [, , , ], we harness the power of 2D semantic segmentation and distill such knowledge back to 2D Gaussians. To do so, we inject each 2D Gaussian with a ‘hard’ semantic label. The ‘hard’ means that the semantic label is non-trainable, which differs from the learnable ‘soft’ label used in recent works [, , ]. Note that although our ‘hard’ semantic label is not trainable, it allows for rendering correct 2D semantic maps by altering its opacity, rotation, scaling, and position. This encourages the points with the same semantic labels to gather closer, facilitating accurate object removal in 3D space.
Assume that each 2D Gaussian associated with a one-hot encoded semantic label $\mathrm{s}$, we render the 2D semantic map as:

$$ $\hat{S}(\mathbf{x})=\sum_{i=1}\mathrm{s}_{i}\alpha_{i}\mathcal{G}_{i}(\mathbf{ u}(\mathbf{x}))\prod_{j=1}^{i-1}(1-\alpha_{j}\mathcal{G}_{j}(\mathbf{u}( \mathbf{x}))).$ (4) $$

### 4.3 Optimization of 2DGS for Street Unveiling

In this part, we first introduce standard objectives used by previous approaches to optimize 2DGS [].
Then we discuss the inferiority of these objectives in the street scene and propose the newly introduced objectives tailored for Street Unveiling.
In summary, our objectives consist of photo-metric loss, semantic loss, normal consistency loss, two different depth distortion losses, and shrinking loss.

Standard approach:
As in 3DGS [], we use $\mathcal{L}_{1}$ loss and D-SSIM loss for supervising RGB color, with $\lambda=0.2$:

$$ $\mathcal{L}_{\text{rgb}}=(1-\lambda)\mathcal{L}_{1}+\lambda\mathcal{L}_{\text{ D-SSIM}}.$ (5) $$

Following 2DGS [], depth distortion loss and normal consistency loss are adopted to refine the geometry property of the 2DGS representation of the street scene.

$$ $\mathcal{L}_{\text{d}}=\sum_{i,j}\omega_{i}\omega_{j}|z_{i}-z_{j}|\hskip 28.45 274pt\mathcal{L}_{n}=\sum_{i}\omega_{i}(1-\mathbf{n}_{i}^{\top}\mathbf{N})$ (6) $$

Here, $\omega_{i}$ represents the blending weight of the $i-$th intersection. $z_{i}$ denotes the depth of the intersection points.
$\mathbf{n}_{i}$ is the normal of the splat facing the camera. $\mathbf{N}$ is the estimated normal at nearby depth point $\mathbf{p}$.

We employ Cross-Entropy (CE) loss to supervise semantic labels:

$$ $\mathcal{L}_{s}(\mathbf{x})=\text{CE}(\hat{S}(\mathbf{x}),S(\mathbf{x}))$ (7) $$

where $S$ is a pseudo semantic map extracted from a pre-trained detector [].

Inferiority of standard objectives.
In Street Unveiling, the scene semantics are expected to be maintained in a less messy and more consistent way to better recognize the Gaussians of objects to remove.
However, solely naïve depth distortion won’t hinder the merging of the 2DGS [] with different semantic labels, leading to noisy semantic information about the 3D world. Meanwhile, the noisy Gaussians in the unseen region will still exist if we don’t find a way to eliminate them. These problems will further harm the generation of an ideal inpainting mask.

Clean up objectives.
To reduce the noise in the semantic fields, we propose a semantic depth distortion loss $\mathcal{L}_{\text{ds}}$
and a shrinking loss $\mathcal{L}_{\alpha}$ on opacity $\alpha$:

$$ $\mathcal{L}_{\text{ds}}=\sum_{k}\mathcal{L}_{\text{d}}^{k}\hskip 28.45274pt \mathcal{L}_{\alpha}=\cfrac{1}{N}\sum_{p}\alpha_{p}$ (8) $$

where $k$ iterates over each semantic label and $\mathcal{L}_{\text{d}}^{k}$ denotes the distortion loss of 2DGS [] with same semantic labels. This semantic depth distortion loss is exerted on the rendered result of the Gaussians with the same semantic label. Intuitively, it will encourage the 2DGS with the same label to have a more consistent depth at the pixel level.
Shrinking loss will further eliminate the Gaussians that are actually unseen by any viewpoint. $\alpha_{p}$ represents the opacity value $\alpha$ of each Gaussian. $N$ is the total number of Gaussians.

The total loss is given as

$$ $\mathcal{L}=\mathcal{L}_{\text{rgb}}+\lambda_{d}\mathcal{L}_{\text{d}}+\lambda _{n}\mathcal{L}_{\text{n}}+\lambda_{ds}\mathcal{L}_{\text{ds}}+\lambda_{s} \mathcal{L}_{\text{s}}+\lambda_{\alpha}\mathcal{L}_{\alpha}$ (9) $$

We empirically set $\lambda_{d}=100$, $\lambda_{n}=0.05$, $\lambda_{ds}=100$, $\lambda_{s}=0.1$, and $\lambda_{\alpha}=0.001$.

## 5 Empty street reconstruction

A common strategy [, ] to inpaint within small scenes is utilizing 2D inpainting methods to inpaint removed objects in the image space for re-optimization.
However, lots of problems arise when it comes to the street scene.
(1) Some views result in over-blurry inpainting results due to the huge size of the inpainting mask, as is illustrated in Fig. 2(b);
(2) Some occluded regions of the street struggle to maintain consistency because they are exposed to a large number of views over the long trajectory.
These challenges will make it more vulnerable to inconsistent inpainting.

In the context of point-based scene representation, eliminating the object involves deleting Gaussians.
However, a naïve removal often yields unsatisfactory results, particularly in the completely unobservable regions beneath the object.
In this section, we first propose how to generate the ideal mask for inpainting as in Fig. 2(c). Then, we propose our time-reversal inpainting framework and how to use the inpainting results to re-optimize the 2DGS.

### 5.1 Generation of ideal inpainting mask

In the street video captured by a moving car, we can divide the pixel space into three categories:
(1) The observable regions, where the regions are not occluded by any objects;
(2) The partially observable regions, where the regions are occluded in some views but are observable in other views;
(3) The completely unobservable regions, where the regions are unobservable in all recording views.
For regions in the second case, we can utilize information from other views to preserve more information about the appearance of the street scene.
As illustrated in Fig. 2, naïvely inpainting with the object mask will cause the unexpected blurry inpainting result at the partially observable region, which can be viewed from other viewpoints but is occluded from the current viewpoint.

To distinguish partially observable regions from completely unobservable regions and improve the inpainting quality, we propose using the rendered alpha map to generate the mask for completely unobservable regions.
For a given viewpoint, we first remove the Gaussians of unwanted objects and their neighbor Gaussians. Then we render the alpha map of the remaining scene.
We identify the completely unobservable region via pixels with low alpha values.
The pixels with alpha values lower than a threshold are selected as inpainting masks.
The threshold is set as 0.99 in our implementation.

### 5.2 Time-reversal inpainting

The core challenge in reconstructing the empty street scene is ensuring consistency between different views over the long trajectory.
However, current video inpainting methods cannot generalize to our long trajectory and complex scenarios, which can be validated from Tab. 2, Fig. 5, and supplementary video comparison. This usually lags behind the scale-up speed of image inpainting models.
To this end, we propose using a reference-based image inpainting method that is trained to ensure consistency between the inpainted region and the reference-based image.
Particularly, we adopt the LeftRefill [] for its stable diffusion-based backbone and the matching-based training strategy.
The stable diffusion backbone leads to a more powerful inpainting model with a strong generation capacity in open-world scenarios, which fits the requirement of Street Unveiling.
Furthermore, the matching-based training strategy ensures that the inpainting model correctly fulfills the masked region based on the observation in the reference image, which encourages consistency between different views.

Figure: Figure 3: Illustration of reference-based inpainting of two views. Left: When we inpaint the near view with the far view as a reference, the consistency of the inpainting result degenerates. There are fewer matching pixels between the reference far-view image and near-view inpainting result; Right: Inpainting the far view using the near view as a reference results in better quality and more accurate pixel matching. It’s easier to generate the low-resolution content with the high-resolution image as a reference.
Refer to caption: x3.png

However, time-forward inpainting sequences usually lead to the failure of consistent inpainting.
Given the moving-forward nature of data-collecting vehicles, objects to be removed transit from far to near in the camera view.
(1) As is illustrated in Fig. 3, when we use the far-view image as a reference to inpaint the same region in the near-view image, the models may not correctly capture the matching relationships and thus causing inconsistent inpainting.
Conversely, setting the near-view image as the reference image leads to a more precise matching result and naturally better inpainting results.
(2) The near-view image can capture more fine-grained information and a larger receptive field, thus making the inpainting easier to inpaint in a high-to-low resolution instead of low-to-high which requires extra super-resolution capacity for the inpainting model.
Besides, the objects removed in the final frame are consistently observed in the earlier frames.

Based on the above analysis, we propose the time-reversal inpainting framework.
If we reverse the time, we can turn the moving-forward nature into a moving-backward nature. When the time is reversed, objects to be removed will instead transition from near to far in the camera view because the camera will be away from the removed object in reversed time-space.

We target to unconditionally inpaint a 3D region only once and then transmit the inpainted region’s pixels to other views with reference-based inpainting. As is illustrated in Fig. 4, we first unconditionally inpaint both frame $T_{n}$ and $T_{n+1}$ with []. However, for frame $T_{n}$, there are some regions that can be seen in $T_{n+1}$. We expect they would share more matching pixels by utilizing the implicit pixel-matching ability of reference-based inpainting model []. Then we use frame $T_{n+1}$ as a reference to inpaint frame $T_{n}$, masking only the regions visible in $T_{n+1}$.

Figure: Figure 4: Illustration of time-reversal inpainting. After we remove the Gaussians of the objects, we first unconditionally inpaint both frame $T_{n}$ and $T_{n+1}$ with []. Then we transmit the pixels from frame $T_{n+1}$ to frame $T_{n}$ in the form of reference-based inpainting []. From a high-level understanding, we inpaint the earlier frame $T_{n}$ with the later frame $T_{n+1}$ as a reference condition.
Refer to caption: x4.png

### 5.3 Re-optimization of the 2D Gaussians

Once we finish time reversal inpainting, we use our inpainting results as pseudo labels to guide the re-optimization of 2DGS [] representation.
We use the following loss to refine 2DGS:

$$ $\mathcal{L}_{\text{retrain}}=\mathcal{L}_{1}+\lambda_{d}\mathcal{L}_{\text{d}} +\lambda_{n}\mathcal{L}_{\text{n}}.$ (10) $$

## 6 Experiments

Dataset. For the evaluation of our approach from the reconstruction aspect and the object removal aspect, we adopt real-world street scenes from Waymo Open Perception Dataset [].
The Waymo dataset collects data from 5 camera perspectives, encompassing roughly 230 degrees in field of view (FOV). We downscale the origin image to $484\times 320$ for efficiency and a fair comparison.

Metrics.
To evaluate the effectiveness of object removal, we approach it from a multi-view inpainting perspective. Following previous works [, , ], we calculate the LPIPS [] and Fréchet Inception Distance (FID) [] scores to quantify the discrepancies between the ground-truth views and removed results. These metrics are computed for each frame and then averaged.

Baselines. We compare our approach to 3D inpainting method SPIn-NeRF [] and a recent Gaussian Splatting based inpainting method Infusion [].
As the original MLP implementation of SPIn-NeRF [] works poorly in the large-scale street scene, we re-implement SPIn-NeRF [] based on 2DGS [], clarifying that our superiority not only from 2DGS but also the proposed time reversal inpainting. Infusion [] is evaluated with the official implementation. But Infusion [] only conducts GS removal and projection once for the whole scene. This doesn’t match our long-trajectory tasks. Instead, we conduct every 10 frames to fit our setting.

Figure: Figure 5: Qualitative comparison results of our methods. We can observe that our methods achieve clearer results than temporarily consistent inpainting baselines. Video comparisons will be placed in the supplementary.
Refer to caption: x5.png

### 6.1 Comparison

**Table 1: Comparison with state-of-the-art 2D/3D inpainting methods. Our FID is only lower than SDXL, yet SDXL doesn’t maintain consistency between different video frames.**
| Method | LPIPS$\downarrow$ | FID $\downarrow$ |
| --- | --- | --- |
| Single Image Inpainting |  |  |
| LaMa(2D) [] | 0.251 | 164.247 |
| SDXL [] | 0.269 | 149.222 |
| Video Inpainting |  |  |
| ProPainter [] | 0.257 | 162.584 |
| 3D Inpainting |  |  |
| SPIn-NeRF [] | 0.252 | 165.792 |
| (in 2DGS) |  |  |
| Infusion [] | 0.330 | 189.586 |
| Ours | 0.241 | 157.970 |

Peak GPU memory usage in our experiments is 16GB. The quantitative comparison results are shown in Tab. 2, and the qualitative comparison of 3D inpainting methods are shown in Fig. 5. Noticed that SPIn-NeRF [] utilizes LaMa [] and Infusion [] utilizes SDXL [] for inpainting. We can observe that 3D inpainting baseline methods lead to worse results, especially when the case is challenging. The results demonstrate that our proposed method achieves better 3D inpainting results from the appearance aspect. The geometry property of the removed region will be discussed in the supplementary. Video comparisons will also be included in the supplementary. In Tab. 2, our proposed method outperforms all the baselines in LPIPS.
It only achieves a lower FID compared to SDXL, yet SDXL doesn’t maintain consistency between different video frames. This can be easily observed from supplementary videos.

### 6.2 Further analysis

Figure: Figure 6: Ablation for different inpainting methods as pseudo labels. From the top two rows, we can observe that time-reversal inpainting is able to achieve more consistent inpainting results than other methods. The bottom row shows that our method can achieve better 3D inpainting results.
Refer to caption: x6.png

Ablation of different inpainting methods as pseudo labels.
We compare the reconstruction results with pseudo labels from different inpainting methods.
From Fig. 6, we can observe that time reversal will maintain the consistency between View $1$ and View $2$. Current single image inpainting models, like LaMa [] and SDXL [], fail to maintain the consistency over the video frames. Although the video inpainting models [] can be temporarily consistent at near frames, the whole inpainting region will be blurred since it can not guarantee 3D consistency.

Ablation of 3D representation.
We ablate through the 3D representation by comparing the results obtained with 3DGS [] and 2DGS []. From Fig. 7, we can observe that after we remove the Gaussians, the rendered alpha map with 3DGS fails to generate an ideal inpainting mask. The quantitative results given in Tab. 2(b) verify the necessity of 2DGS representation.

Figure: Figure 7: Ablation for 3D representation. When we use 3DGS, it’s hard to generate an ideal inpainting mask with a rendered alpha map. Good inpainting results are hard to achieve.
Refer to caption: x7.png

## 7 Conclusion

We propose StreetUnveiler, a comprehensive pipeline for reconstructing empty streets from in-car camera videos. Our method represents the street scene using a hard-label semantic-aware 2D Gaussian Splatting [], allowing us to remove each instance from the scene seamlessly. To create an ideal inpainting mask, we utilize the rendered alpha map after removing unwanted 2DGS. Additionally, we introduce a novel time-reversal inpainting framework that enhances consistency across different viewpoints, facilitating the reconstruction of empty streets. Extensive experiments demonstrate that our method effectively reconstructs empty street scenes and supports free-viewpoint rendering.

## 8 Acknowledgement

The work was supported by NSFC #62172279, #61932020, Program of Shanghai Academic Research Leader. The authors would like to thank Shuo Wang, Binbin Huang, and Zibo Zhao for their valuable feedback and proofreading.

## Appendix A Implementation details

### A.1 Details of hard-label semantic 2DGS reconstruction

Initialization with Lidar points.
High-quality appearance and semantic reconstruction of the whole street scene are hard to reach, with barely SFM points [, ] as initialization for street scenes.
Lidar points are leveraged to better reconstruct the street scene like in  [, , ].
We use an off-the-self 2D semantic segmenter [] to process the 2D images and back-project the hard semantic labels to 2D Gaussians [].

Environment map for street reconstruction.
We empirically find that
most 2D Gaussians’ opacity will be larger than $0.9$ or lower than $0.1$, leading to the imperfect reconstruction quality of the background environment, *i.e.*, sky.
To better model the environment in the street scene, we employ a tiny MLP $f$ to query the color of the environment map, which is similar to  [, ]. The queried environment color at $\mathbf{x}$ is denoted as $\mathbf{c}_{\text{env}}$. The final color of the ray is obtained by blending the color of 2DGS projection and the environment map as follows:

$$ $\mathbf{c}_{\text{env}}(\mathbf{x})=f(\mathbf{M,\mathbf{x}})\hskip 28.45274pt \mathbf{c}_{\text{final}}(\mathbf{x})=\mathbf{c}(\mathbf{x})+(1-\mathbf{\alpha }(\mathbf{x}))\mathbf{c}_{\text{env}}(\mathbf{x})$ (11) $$

where $\mathbf{M}$ denotes the projection matrix from world coordinates to pixel coordinates.
$\mathbf{\alpha}(\mathbf{x})$ is the rendered alpha map of 2DGS rendering.

Details of two-stage reconstruction training

The optimization of our designed 2DGS [] reconstruction for street scenes contains two stages.
(1) In the first stage, we employ adaptive density control of 3DGS []. And $\mathcal{L}_{\text{d}}$, $\mathcal{L}_{\text{n}}$ and $\mathcal{L}_{\text{ds}}$ will be deactivated to reach a more stable initialization of 2DGS reconstruction.
(2) In the second stage, $\mathcal{L}_{\text{d}}$, $\mathcal{L}_{\text{n}}$ and $\mathcal{L}_{\text{ds}}$ is activated. As empirically, most 2D Gaussians’ opacity will be larger than $0.9$ or lower than $0.1$. The noisy 2DGS with the wrong semantic label will be optimized as low opacity through $\mathcal{L}_{\text{ds}}$. We prune the Gaussians with opacity lower than a threshold $\epsilon$ to further eliminate the noisy semantics in the 3D world, with $\epsilon$ set as $0.3$ in our experiments.

### A.2 Details of time-reversal inpainting framework

As is mentioned in  [], when we are using a latent-diffusion-based inpainting model, there will be non-ignorable shifts in low-frequency fields if we use images decoded by KL-VAE [, ] repeatedly for different times. Given that our method can be summarised as inpainting frame $T_{i}$ with $T_{i+1}$ as a reference through LeftRefill [], which is latent-diffusion-based. For a whole sequence of video, if we simply iteratively inpaint every $T_{i}$ with $T_{i+1}$ as a reference, the shifts in low-frequency fields will be badly augmented by KL-VAE, which will severely harm the quality of our 2D inpainting guidance. To alleviate this inevitable shift from the KL-VAE of the latent diffusion model []. We first select some keyframes in the video. Then we use time-reversal inpainting to inpaint the selected keyframes iteratively in the reversed time sequence.

We firstly time-reversal inpaint the keyframes of timestamps $\{T_{k_{1}},\ldots,T_{k_{n}}\}$.
After inpainting all the keyframes in the reversed time sequence, we generate the middle frames between keyframe $T_{k_{i+1}}$ and keyframe $T_{k_{i}}$ with keyframe $T_{k_{i}}$ as reference image. When we have inpainted the street scene in the image sequences, we will use these results as pseudo-labeled data to further re-optimize the 2DGS of the empty street scene.

## Appendix B More experiments

### B.1 Ablation of time-forward inpainting and time-reversal inpainting

To further validate the effectiveness of time-reversal inpainting, we do an additional ablation here with time-forward inpainting, which is the reverse version of our proposed time-reversal inpainting. In Tab. 3, our time-reversal inpainting achieves better quantitative results than time-forward inpainting.

For our time-reversal inpainting, we inpaint frame $T_{n}$ with $T_{n+1}$ as reference. For time-forward inpainting, frame $T_{n+1}$ is inpainted with frame $T_{n}$ as reference. The Fig. 8 elaborates the details about the process of these two methods. The qualitative comparison in Fig.9 showcases the high-to-low-resolution nature of time-reversal inpainting, which will enhance the quality of the results.

Figure: Figure 8: Illustration of the difference between time-reversal inpainting and time-forward inpainting on inpainting strategy. We first use unconditional inpainting to inpaint the frame $T_{n+1}$. For our time-reversal inpainting, we inpaint frame $T_{n}$ with $T_{n+1}$ as reference. For time-forward inpainting, frame $T_{n+1}$ is inpainted with frame $T_{n}$ as reference.
Refer to caption: x8.png

Figure: Figure 9: Illustration of the qualitative comparison between time-reversal inpainting and time-forward inpainting. We can observe that the quality of time-forward inpainting results degenerates because time-forward inpainting uses a low-to-high-resolution approach. This requires extra super-resolution capacity for the inpainting model to get a better result. However, our time-reversal inpainting uses a high-to-low-resolution approach. High-resolution content will better guide the low-resolution content.
Refer to caption: x9.png

### B.2 Ablation of hard semantic label

We additionally ablate the effectiveness of the hard semantic label. From Fig. 10, we can observe that both 2DGS representation and hard semantic label contribute to a more stable reconstruction of the semantic field.

The comparison between (a) and (b) demonstrates that the use of hard semantic labels effectively reduces noise within the semantic fields. In addition, the comparison between (a) and (c) indicates that the 2DGS representation leads to more stable semantic fields. Finally, (d) illustrates the clean and stable semantic field achieved by employing hard-label semantic 2DGS in our method.

**Table 3: Quantitative ablation of time-reversal inpainting and time-forward inpainting. The result validates the effectiveness of our method.**
| Method | LPIPS$\downarrow$ | FID $\downarrow$ |
| --- | --- | --- |
| Time-Forward Inpainting | 0.249 | 160.666 |
| Time-Reversal Inpainting(Ours) | 0.241 | 157.970 |

By reconstructing a clean and stable semantic field of the street scene, we can more accurately identify the Gaussians that need to be removed. This facilitates obtaining a high-quality 2D inpainting result, which serves as effective guidance for re-optimization.

Figure: Figure 10: Illustration of hard semantic label ablation. Black rectangles in the figures include the noise in the semantic fields. The comparison between (a) and (b) demonstrates that hard semantic labels effectively reduce noise within semantic fields. Similarly, the comparison between (a) and (c) indicates that the 2DGS representation contributes to more stable semantic fields. Finally, (d) illustrates the clean and stable semantic field achieved by employing hard-label semantic 2DGS in our method.
Refer to caption: x10.png

### B.3 Comparison of geometry performance

Since we want to reconstruct the empty street, we also want to compare the geometry property of our method other than just appearance. From Fig. 11, Fig. 12, Fig. 13, Fig. 14, we can observe that our method produces both better appearance quality and geometry quality from rendered RGB and rendered normal images.

Figure: Figure 11: Illustration of geometry performance comparison.
Refer to caption: x11.png

Figure: Figure 12: Illustration of geometry performance comparison.
Refer to caption: x12.png

Figure: Figure 13: Illustration of geometry performance comparison.
Refer to caption: x13.png

Figure: Figure 14: Illustration of geometry performance comparison.
Refer to caption: x14.png

Figure: Figure 15: Illustration of geometry performance comparison.
Refer to caption: x15.png

## Appendix C Empty street scene mesh extraction

We can further extract the mesh for our reconstructed empty street scene using TSDF fusion following 2DGS [] with Open3D []. In Fig. 16 and Fig. 17, we compare the extracted colored mesh before and after our unveiling. Our inpainting framework can successfully remove unwanted cars from the street and finally reconstruct an empty street.

Figure: Figure 16: Illustration of colored mesh comparison between before and after the unveiling.
Refer to caption: x16.png

Figure: Figure 17: Illustration of colored mesh comparison between before and after the unveiling.
Refer to caption: x17.png

## Appendix D Limitations

Although our proposed method can achieve scene reconstruction without unwanted static objects, our method is not without limitations.
(1) Our method depends on the precision of the 2D segmentation model to reconstruct a reliable semantic field to identify the Gaussians to remove. Failure of the 2D semantic segmentation will lead to low-quality Gaussians removal results. Street Unveiling without pseudo-semantic guidance would be an important step towards a more robust solution.
(2) Since the consistency of our inpainting assumes the sufficient ability of reference-based inpainting to be aware of matching, the bottleneck of the pre-trained reference-based inpainting model may also be a bottleneck of our method.
(3) Our method’s computational costs grow linearly with the video’s frame number because every frame needs to be inpainted.

## Appendix E Societal impacts

The technology can potentially distort the representation of public spaces in urban planning, leading to flawed decision-making. At the same time, it may also be misused to alter the representation of important archaeological sites in digital reconstructions, leading to misinformation and misunderstanding of historical facts.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: I think the paper is well-written with a good application as it proposed an inpainting method that seems to work well to maintain temporal consistency.
soundness: 3
strengths: - I particularly like the idea of time-traversal inpainting. While single-image inpainting has been extensively explored, video-based inpainting (where temporal consistency is required) remains relatively understudied. Although the overall process isn't completely novel, I appreciate that the method is specifically tailored for driving videos with predominantly forward motion. Reference-based inpainting shows great promise for many applications and seems like a valuable direction to pursue.

- The paper is well-written, with clear motivation. The process, insights, and technical details are thorough and convincing. I especially appreciate the comprehensive related work section, which is very helpful for readers who don't necessarily work in inpainting or neural scene representation.

- The results are impressive, with both quantitative and qualitative improvements over baseline inpainting methods. The method appears to handle challenging cases well, particularly in maintaining consistency across long sequences.
summary: This paper introduces StreetUnveiler, a method to reconstruct empty street scenes from crowded video captured by moving vehicles. The key challenge is removing temporarily static objects (like parked cars and pedestrians) while maintaining temporal consistency over long trajectories. The main contributions are: 1) adapting 2DGS with hard-label semantic for scalable scene representation with explicit object removal, 2) using rendered alpha map to classify areas as fully observed, partially observed, or unobserved for creating inpainting masks, and 3) a time-reversal inpainting framework that maintains temporal consistency by using future frames to condition earlier frames. The paper shows better inpainting results compared to other off-the-shelf methods.
weaknesses: - My only concern is that while the ultimate goal is temporally-consistent inpainting, the authors chose a more complex approach going through neural scene representation (2D/3DGS) and using rendered alpha masks to identify different region types (observable/unobservable) for generating inpainting masks. While this is interesting in principle, I wonder if a simpler pipeline could achieve similar goals - for example, establishing object correspondences across the video frames and then applying LeftRefill. I also note that LeftRefill is missing from the quantitative comparisons.

---

confidence: 4
contribution: 1
initial_rating: 8
presentation: 3
questions: please address the concern in the weakness session
soundness: 3
strengths: 1. Clear writing and presentation, with a well-designed supplementary website
2. Two novel and effective design elements:
   * An improved method for generating inpainting masks that incorporates observations from other frames
   * Time-reversal inpainting that utilizes future frames as references
summary: This work aims to remove static movable objects (e.g., parked vehicles) in street view data for 3D reconstruction (e.g., 3DGS). The authors propose two main improvement to achieve better inpainting results: better inpainting mask, better reference for image inpainting.

Experiments conducted on Waymo and PandaSet datasets demonstrate better performance compared to image/video inpainting methods and other 3D inpainting approaches.
weaknesses: 1. The contribution seems somewhat limited. While the two major contributions are effective, they appear to be incremental rather than groundbreaking. The work could serve as a baseline, but not much more that.

2. The performance metrics are not entirely convincing. Table 1 doesn't demonstrate significant advantages over other baselines. The video in the supplementary materials appears noticeably inferior to other NVS methods on Waymo (e.g., EmerNeRF, StreetSurf, StreetGaussian).

3. The benchmark methodology raises some questions (please correct me if I'm mistaken, as I haven't encountered this benchmark setting before). What are the ground truth images used for LPIPS? If all rendered images lack vehicles while the ground truth images typically include them, is this an appropriate measurement for LPIPS/FID?

4. I have reservations about the task itself: Is it truly necessary to remove static components from the scene? Static elements (e.g., trees, traffic signs, rubbish bin) are integral parts of the environment. Even if removal is desired, wouldn't it be more efficient to accomplish this through multi-pass data collection?




## updated response to the authors after rebuttal
I want to clarify that my use of 'this issue will kill the paper' is a common academic shorthand expressing concern about a critical technical issue that needs to be addressed.  My colleagues and I often use this phrase in academic discussions to highlight fundamental issues that could seriously affect a paper's contribution. If you're unfamiliar with this expression and find it harsh, I sincerely apologize – that wasn't my intention.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: In fact, there are many methods that directly predict the masks of dynamic objects, and these are usually more reliable than the masks obtained by optical flow. Additionally, in many complex situations (such as vehicles driving sideways at an intersection), can the prior advantage of time-reversal inpainting still be maintained?
soundness: 3
strengths: - The use of 2D Gaussian combined with semantic information is reasonable. It allows for better identification and removal of objects by injecting semantic labels into 2D Gaussians.
- Experiments on street scene datasets demonstrate the effectiveness of the method.
- The proposed time-reversal framework inpaints frames in reverse order and uses later frames as references for earlier frames. This enhances the temporal consistency of the inpainting results, making the reconstructed street more realistic and consistent across different views.
summary: The paper presents StreetUnveiler for reconstructing an empty street from in-car camera videos. It uses hard-label semantic 2DGS for representation, divides the scene into regions located by an alpha map, and introduces a time-reversal inpainting framework. Experiments show its effectiveness in reconstructing the empty street for autonomous driving applications.
weaknesses: - The 2DGS reconstruction and optimization processes might be computationally complex and costly, especially for large-scale street scenes, leading to long training and inference times. The author may consider other faster reconstruction baselines.
- The method has limited ability to handle dynamic objects. Although some approaches for simple dynamic cases are proposed, more improvements are needed for complex dynamic scenes.

---

confidence: 5
contribution: 4
initial_rating: 8
presentation: 4
questions: 1. Applications of Unveiled Street Scene Reconstruction: The task of reconstructing an unveiled street scene is indeed challenging and engaging. However, real-world street environments include more than just vehicles, and vehicles can often obscure parts of other objects, such as the lower half of a road sign. Would it be more practical to first apply 2DGS to separate scenes based on specific requirements, then proceed with targeted inpainting? This approach could provide more utility by selectively reconstructing essential scene elements.
2. Potential to Build on Existing 3DGS Street Scene Reconstruction Methods: Current methods, such as Street Gaussians[1] (using 3D box-based scene decomposition) and S3Gaussian[2] (self-supervised scene decomposition), aim to separate scenes into dynamic and static components. Could integrating these approaches enhance StreetUnveiler’s ability to more effectively distinguish between removable objects and essential scene elements? This could potentially improve the precision in identifying and preserving critical components.

[1]Yan Y, Lin H, Zhou C, et al. Street gaussians for modeling dynamic urban scenes[J]. arXiv preprint arXiv:2401.01339, 2024.
[2]Huang N, Wei X, Zheng W, et al. $\textit {S}^ 3$ Gaussian: Self-Supervised Street Gaussians for Autonomous Driving[J]. arXiv preprint arXiv:2405.20323, 2024.
soundness: 4
strengths: 1. The writing of the paper is clear and easy to follow.
2. The task of empty street scene reconstruction is interesting. The limitations discussed in the introduction are well studied in the following section. 
3. The proposed method is evaluated on two datasets. Many quantitative and qualitative experiments are conducted to demonstrate the effectiveness of this method.
summary: The paper presents StreetUnveiler, a method based on semantic 2DGS to achieve the empty street scene reconstruction. 
StreetUnveiler first utilizes the rendered alpha map to locate unobservable regions with the aid of distortion loss and shrinking loss.
Then a time-reversal inpainting framework is introduced to optimize temporal consistency by leveraging later frames as references for earlier ones. 
Experiments are conducted on Waymo and Pandaset to demonstrate the effectiveness of StreetUnveiler for the empty street reconstruction.
weaknesses: 1. Lack of Quantitative Metrics for Mesh: While the authors discuss the differences in problem formulation between StreetSurf and StreetUnveiler, a quantitative comparison of mesh reconstruction is still lacking. StreetSurf uses point clouds as ground truth to calculate Chamfer Distance as a metric. Could a point cloud derived from Structure from Motion (SfM) be used as a ground truth for evaluating StreetUnveiler’s performance? Even relative values across different baselines could provide meaningful insight.
2. Limitations in Semantic Label Supervision: Current semantic supervision seems limited to common objects like vehicles, yet street environments also include pedestrians and less common obstacles. Could expanding the range of semantic (instance) labels improve the model’s robustness across diverse urban scenes?

Meta-review: In this paper, the authors have proposed 3D StreetUnveiler to reconstruct an empty street from crowded observations based on semantic 2DGS. It first uses a rendered alpha map to locate unobserved regions, and creates a inpainting mask. Then, a time-reversal inpainting framework is proposed to enhance the temporal consistency. The paper is well-written and the contributions are recognized by most reviewers. There are some concerns raised by Reviewer roan which should be considered from my point of view. (1) A comprehensive comparison with StreetGaussian, e.g. the example in https://annoy449.github.io/output.mp4. (2) Some technical contributions of the paper are engineering and the novelty of the design should be highlighted. I agree that despite of these concerns, this work is still inspiring and will encourage more works in this topic. I would recommend a decision of accept as poster.

Decision: Accept (Poster)