3D StreetUnveiler with Semantic-aware 2DGS - a simple baseline
Authors: Jingwei Xu
Yikai Wang
Yiqun Zhao
Yanwei Fu
Shenghua Gao
Venue: ICLR 2025 Poster (2025)
Keywords: 3D inpainting
Empty Street Reconstruction
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Unveiling an empty street from crowded observations captured by in-car cameras is crucial for autonomous driving. However, removing all temporarily static objects, such as stopped vehicles and standing pedestrians, presents a significant challenge. Unlike object-centric 3D inpainting, which relies on thorough observation in a small scene, street scene cases involve long trajectories that differ from previous 3D inpainting tasks. The camera-centric moving environment of captured videos further complicates the task due to the limited degree and time duration of object observation. To address these obstacles, we introduce StreetUnveiler to reconstruct an empty street. StreetUnveiler learns a 3D representation of the empty street from crowded observations. Our representation is based on the hard-label semantic 2D Gaussian Splatting (2DGS) for its scalability and ability to identify Gaussians to be removed. We inpaint rendered image after removing unwanted Gaussians to provide pseudo-labels and subsequently re-optimize the 2DGS. Given its temporal continuous movement, we divide the empty street scene into observed, partial-observed, and unobserved regions, which we propose to locate through a rendered alpha map. This decomposition helps us to minimize the regions that need to be inpainted. To enhance the temporal consistency of the inpainting, we introduce a novel time-reversal framework to inpaint frames in reverse order and use later frames as references for earlier frames to fully utilize the long-trajectory observations. Our experiments conducted on the street scene dataset successfully reconstructed a 3D representation of the empty street. The mesh representation of the empty street can be extracted for further applications.

Published as a conference paper at ICLR 2025
3D STREETUNVEILER WITH SEMANTIC-AWARE 2DGS
A SIMPLE BASELINE
Jingwei Xu', Yikai Wang2, Yiqun Zhao Yanwei Fu Shenghua Gao
1 ShanghaiTTech University 2 Nanyang Technological University 3 The University of Hong Kong
HKU Shanghai Intelligent Computing Research Center Fudan University TTrnnccnggamm
xu jw2023e shangha tech. edu. cn
yikai wangentu. vangentu.edu.ss hku. hk yanwei fuê fudan. edu. on
y yiqun. zha ochao0comm connect.
gaosh@hku .k
ABSTRACT
Unveiling empty street from crowded observations captured by in-car cameras is crucial for autonomous driving. However, removing all temporarily static objects, such as stopped vehicles and standing pedestrians, presents a significant challenge. Unlike object-centric 3D inpainting, which relies on thorough observation in a small scene, street scene cases involve long trajectories that differ from previous 3D inpainting tasks. The camera-centric moving environment of captured videos further complicates the task due the limited degree and time duration of object observation. To address these obstacles, we introduce StreetUnveiler to recon- struct an empty street. StreetUnveiler learns 3D representation of the empty street from crowded observations. Our representation is based on the hard-label semantic 2D Gaussian Splatting (2DGS) for its scalability and ability identify Gaussians to be removed. We inpaint rendered image after removing unwanted Gaussians to provide pseudo-labels and subsequently re-optimize the 2DGS. Given its temporal continuous movement, we divide the empty street scene into observed, partial- -obeeed- and unobserved regions, which we propose locate through a rendered alpha map. This decomposition helps us to minimize the regions that need to be inpainted. To enhance the temporal consistency of the inpainting, we introduce novel time-reversal framework to inpaint frames in reverse order and use later frames as references for earlier frames to fully utilize the unngtrraecctory observations. Our experiments conducted the street scene dataset successfully reconstructed a 3D representation of the empty street. The mesh representation of the empty street be extracted for further applications.
INTRODUCTION
Accurate 3D reconstruction of an empty street scene from an in-car camera video is crucial for autonomous driving. provides reliable digital environments that simulate real-world street scenarios. Although this is an important task, it is seldomly studied in previous works because of its challenging nature in the following aspects: (1) Lack of ground truth data for pre-training inpainting models specialized for street scenes; (2) The camera-centric moving captures objects from limited angles and for brief periods; (3) The long trajectory of in-car camera videos leads objects appearing and disappearing at different time points, complicating object removal.
But there still exists a blessing we can take from the long trajectory moving- forward nature. As the car moves forward, objects that disappear from the later frame will only be visible in previous video frames. This gives a hint about maintaining the temporal consistency of the same regions.
To address the challenge of reconstructing empty street, we introduce Street re- construction method targeting unveiling the empty representation of long-trajectory street scenes.
Yikai Wang was affiliated with Fudan University during the development of this paper.
1 Prof. Yanwei Fu is also with Fudan ISTBI-ZINU Algorithm Centre for Brain-inspired Intelligence, Zhejjang Normal University, Jinhua, China.
1 *Correpooning author

---

Published as a conference paper at ICLR 2025
Semart
stetic
Impainting
Recopimized IDCS
Re Ba-optimization
Bander
opdiniaed 2005
Clean Street Resuls
Novel View Synthess
Emply Street faech Estraccion
Input
Output
Application
We achieve accurate empty street reconstruction from in-car camera videos. With the aid of the proposed hard-label semantic 2D Gaussian Splatting and time-reversal inpainting framework, remove the unwanted objects with satisfactory appearance and geometry of occluded regions.
Figure 1:
StreetUnveiler involves several key steps. First, it reconstructs the observed 3D representation and identifies unobserved regions occluded by objects. Then, it uses time-reversal inpainting framework to consistently inpaint these unobserved regions as pseudo labels. Finally, it re-optimizes the 3D representation based on these pseudo labels. The overall pipeline is illustrated in Fig.
StreetUnveiler first reconstructs the original parked-up street with Gaussian Splatting (GS) due to its scalability and editability. However, as illustrated in Fig. 2 inpainting with the naïve object mask (orange mask) often results in blurring and loss of details in large inpainted regions, which common issue the previous works Mirzaei et al. (2023); Weder al. (2023); Wang al. (2023a): Weber et al. (2024): Liu et al. (2024). Generating masks for completely unobservable regions (blue mask) that are invisible from any viewpoint remains a challenge. Recent work Liu al. (2024) requires user providdo masks, which is impractical for long trajectories. Moreover, the messy appearance these regions after removing the Gaussians makes it difficult to use methods like SAM et al. (023). To address the difficulty of finding ideal inpainting mask, propose to generate the mask through the rendered alpha map and reconstruct the scene using hard- label semantic 2DGS Huang et al. (2024) instead of 3DGS Kerbl al. 2023)) 2DGS has high opacity value for Gaussians, resulting in low alpha values in completely unobservable regions. A semantic distortion loss and shrinking loss are employed to further reduce the rendered alpha values of the complctely unobservable regions. This approach automatically generates masks for unobservable regions without user input, leading to better inpainting results.
Furthermore, we propose time-reversal in- painting framework enhance the tempo- ral consistency of inpainting results in com- pletely unobservable regions. By inpainting the video frames in reverse order, we use the later frame as a reference to inpaint the earlier frame. When the video is played in reverse, the object in the later frame will transition only from near far in the era view as the camera moves away from the object in reversed time-space. This method uses a gigg-to-low-resolutin ing approach instead of filling an area larger than the reference region, as in the low-to- high-resolution approach, which results in more consistent inpainting. Finally, the in- painted pixels are used as pseudo labels to guide the re-optimization of 2DGS. This en- ables our method to learn scalable 2DGS model that represents empty street while preserving the appearance integrity of re- gions visible in other views.
Vehicle Mask
Naive Inpaint
Inpaint
bb)
Remove Gaussians
Generated
Remove 2DGS and Inpaint
guid-
Inpaint
(c)
(d)
Figure 2: (a) The mask of the whole unwanted ob- ject; (b) Inpainting with (a) mask; (c) Generate the inpainting mask through rendered alpha map. The pixel with low alpha value is selected as an inpainted pixel; (d) Inpainting with the generated (c) mask.
2

---

Published as a conference paper at ICLR 2025
Our contribution can be summarized as follows:
We propose representing the street as hard-label semantic 2DGS, optimizing the 3D scene with semantic guidance for scalable representation and improved instance decoupling.
We use a rendered alpha map to locate completely unobservable regions and apply a semantic distortion loss and a shrinking loss to create reasonable inpainting mask for these regions.
We introduce a novel time-reversal inpainting framework for long- cong-trajectory scenes, en- hancing the temporal consistency of inpainting results for re-optimization. Experiments show that our method can reconstruct an empty street from in-car camera video containing obstructive elements.
RELATED WORK
Neural scene representation and reconstruction. The use of neural radiance fields Mildehall et al.] (2020) to represent 3D scenes inspired a lot of follow-up work based on the original approach. Some Molke al. (2022); ; Chen al. (2022); (Sun al.] (2022):
Sara Fridovich- and Alex Yu et al. (2022); Yu et al. (2022)] Xiao et al. (2024) explore explicit
representations such as low-rank matrices, hash grids, or voxel grids to increase the model capacity of original MLPs. Some works explored multiple separate MLPs Reiser et al. (021): Kundu et al. (2022); Fu al. (2022) represent instances and backgrounds separately. However, these scale- up strategies are complicated to implement at the scale of street scenes. Existing works Xu et al.
(2023); ; Tancik et al. (2022); Turki et al. (2022);] [u et al.] (2023): Rematas al. (2022); Yang et al.
(2023b): Wang al. (2023c); : Turki et al. (2023); Meuleman et al. (2023); Wang et al. (202bb); Guo ct al. (2023); Zhang et al. (2023b); Siddiqui et al. (2023) explored mesh-based, primitive-
or grid-based representations for large-scale street scenes. However, both grid-based representation Guo ctt al. (2023) and mesh-based representation Wang et al. (2023c) may be constrained by their limited topology, making it hard to decouple the scene into separate instances. Recent advances in point- based rendering techniques Kerbl et al. (2023); Lassner & Zollhofer (2021): |
Xu et al. (2022); Huang et al.
(2024a) can achieve both high-quality and fast rendering speed. The paint-based nature of Gaussian Splatting enables scalability for street scenes. While recent works Chen et al. (003b)); Yan et al. (2024); Lin et al. (2024b); Ren et al. (2024); Cheng et al. (2024) have explored the reconstruction of large-scale sige--ccle scenes using Gaussian Splatting. our work focuses on the unveiling stage of a street scene, which is more important for autonomous driving and more challenging.
3D scene manipulation and inpainting. Early works Wang et al. (2021) Yuan et al. 2 Philip
Drettakis 2018); Thonat et al. (2016); Anguelov et al. (2010): Liu et al. (2018): | Yu et al. (2018)
2019): Yi et al. (2020); /Zha et al. (2021); Mirzaei et al. (2024) explored street scene editing by
leveraging single-view view or multi view image inpainting networks. With the rapid development of
Neural Scene Representation, editing 3D scene has been explored by lots of works Chong Bao and Bangbang Yang et al. (2022); Zhao et al. (2024b); Yang et al. (2023a): Yuan et al. (2022) ; Bao et al. (203); Kobayasih et al. (2022); Kerr et al. (2023); Peng et al. (2023); Zhao al. (202aa).
Edit-NeRF et al. (2021) pioneered shape and color editing of neural fields using latent codes. Subsequent works Bao al. (2023): : Kobayashi et al. (2022: Kerr et al. (202): Peng et al. (023)
utilized CLIP models to provide editing guidance from text prompts or reference images. Recent
works Weder et al. (2023); Zhang al. (2022); Mirzaei et (023); Xiang et al. (2023) Chen et al.
(2023a) et al. (2023) Ye et al. (023)): Weber et al. (2024) >: Waan et al. (002cc); 1; Lin et al. (2024a);
Mirzaei et al. (2024); Prabhu al. (2023) also explored 2D stylization and inpainting techniques, utilizing pretrained Diffusion Priors Rombach et al. (022) for editing 3D scenes. Specifically, Chen
al. (202a): /Fng et al. (2023): Ye et al. (2023); Wang et al. (2024c) investigated these approaches in collaboration with Gaussian Splatting. Unlike them, our work focuses on street scene object removal and empty street reconstruction, which more challenging.
Image and video inpainting. Image inpainting Bertalmio et al. (2000) aims to fulfill the missing region within an image. Standard approaches included GAN-based methods Pathak et al. (2016); Zhao et al. (2020), attention
methods Yu et al. (2018): ; Liu et al. (2019). transformer- methods Wan et al. (222) Liu et al. (2022). ). and more recently, diffusion-
methods Rombach et al. 2022);; Wang et al. (2024a). Control-No Net Zhang et al. (2023a) enabled generating images with addifional conditions on the frozen diffusion models. Recently, LeftRefill Cao et al.) (2024) learned to
guide the frozen diffusion inpainting models with extra conditions of the reference image, enabling
3

---

Published as a conference paper at ICLR 2025
multi-view inpainting the frozen diffusion model. However, these image inpainting methods mainly focused on the static scenario. Video inpainting considers the temporal consistent inpainting
in the continuous image sequence, utilizing approaches like 3D CNN Wang al. (2019); Hu et al. (2020), temporal shifting Zou et al. (2021). flow guidance Kim et al. 2019); Xu et al. (2019): Li al. (2022), temporal attentions Ren al. 2022), to name few. However, these video inpainting methods hardly considered the long trajectory movement of cameras. In contrast, in our paper, we focus on the inpainting of large-scale street scenes. Furthermore, the 2DGS representation used in our paper enables the free-view rendering of the inpainted video.
3 PROBLEM FORMULATION
Given in-car camera videos and the Lidar data of parked-up street, our goal is to remove all temporar- ily static objects in the street, like stopping vehicles and standing pedestrians, and finally reconstruct empty street. This task, named Street Unveiling, is reconstruct obstacles, providing an empty representation of the street environment. Such representations are mainly represented by 3D models for free-view rendering. This task holds significant implications for autonomous driving systems, urban planning, and scene understanding applications.
devoid of these static
Street Unveiling shares some similarities with related tasks but cannot be addressed using existing approaches. (1) 3D reconstruction primarily involves modeling a primary image or scene with an object-centric camera. In contrast, Street Unveiling focuses on the background, aiming remove foreground objects to reveal an empty street. The absence of ground truth further differentiates it from standard 3D reconstruction tasks. (2) Video inpainting typically deals with videos captured by fixed or minimally moving cameras, featuring one or a few central objects. Conversely, Street Unveiling involves long camera trajectories without central objects. These distinctions require different capabilities and novel methods to address the unique challenges of Street Unveiling.
SEMANTIC STREET RECONSTRUCTION
We opt for 2D Gaussian Splatting Huang al. (002a)) (2DGS) as scene representation for its rendering speed and editability. We first introduce the 2DGS in Sec. 4.1 Subsequently, we elaborate
our algorithm tailored for street unveiling using 2DGS Sec. 4.2 and Sec. 4.3
4.1 PRELIMINARY: 2D GAUSSIAN SPLATTING
Our reconstruction stage builds upon the state-of-the-art point-based renderer with the splendid geometry performance, 2DGS Huang et al. (2024a). 2DGS is defined by several key components: the central point Рę. two principal tangential vectors t, and t, that determine its orientation, and scaling vector S = (sa, 8.) controlling the variances of the 2D Gaussian distribution.
2D Gaussian Splatting represents the scene's geometry set of 2D Gaussians. A 2D Gaussian is defined in local tangent plane in world space, parameterized as follows:
P(u, v) Pı + su,t, + s,t,
(I)
For the point u = (u, v) in ue space, its 2D Gaussian value can then be evaluated using the standard Gaussian function:
u +
((u) = exp
2
The center P&, scaling (su: Sw), and the rotation (tw, t.) are learnable parameters. Each 2D Gaussian primitive has opacity a and view-dependent appearance e with spherical harmonies. For volume rendering, Gaussians are sorted according to their depth value and composed into an image with front-to-back alpha blending:
c(x) c.., II - [[=xa)((
(3)
1=1
where x represents homogeneous ray emitted from the camera and passing through space.
4

---

Published as a conference paper at ICLR 2025
4.2 2DGS FOR STREET SCENE RECONSTRUCTION
2DGS Huang et al. 2024a) features for its accurate geometry reconstruction of the object surface. However, the application of 2DGS to reconstruct objects devoid of surfaces, such the sky in an open-air street scene, remains unexplored. We aim to reconstruct the street scene as a radiance field and semantic field using 2DGS. More details about radiance field reconstruction are included in the supplementary.
Learning 2D Gaussians with semantic guidance. We aim augment the radiance field of street scenes with editability. Inspired from Guo et al. (2022); Yan et al. (2024); Chen et al. (2023b); Zhou al.] (204)), we harness the power of 2D semantic segmentation and distill such knowledge back to 2D Gaussians. To do so, we inject each 2D Gaussian with a 'hard' semantic label. The 'hard' means that the semantic label is non-trainable, which differs from the learnable "soft' label used in recent works Zhou al. )202);; ; Yan al. (2024); Zhou et al. (2023b). Note that although our 'hard' semantic label is not trainable, it allows for rendering correct 2D semantic maps by altering its opacity, rotation, scaling, and position. This encourages the points with the same semantic labels to gather closer, facilitating accurate object removal in 3D space. Assume that each 2D Gaussian associated with one-hot encoded semantic label s, render the 2D semantic map as:
i-1
5(x) Σ G II (1-,,,;(u(x))),
j=1
During our densification, the newly generated splats will inherit the original hard semantic labels.
4.3 OPTIMIZATION OF 2DGS FOR STREET UNVEILING
In this part, we first introduce standard objectives used by previous approaches to optimize 2DGS 2DGS|Huang al.) (2024a). Then we discuss the inferiority of these objectives in the street scene and propose the newly introduced objectives tailored for Street Unveiling. In summary, our objectives consist of photo-metric loss, semantic loss, normal consistency loss, different depth distortion losses, and shrinking loss.
Standard approach: As in 3DGS Kerbl et al. 2023) we use L1 loss and D-SSIM loss for supervising RGB color, with À = 0.2:
Ligh (1 A)C1 + A
(5)
Following 2DGS Huang et al. (2024a), depth distortion loss and normal consistency loss are adopted to refine the geometry property of the 2DGS representation of the street scene.
La Σ wawl zzl
L. -== (1-,NN N,,
(6)
1,7
Here, wi represents the blending weight of the i-th intersection. 2; denotes the depth of the intersection points. n, is the normal of the splat facing the camera. N is the estimated normal at nearby depth point p.
We employ Cross-Entropy (CE) loss supervise semantic labels:
L. (x) = CB(3(x), ((()))
where S' is pseudo semantic map extracted from a pre-trained segmentation model Xie et al. (2021).
Inferiority of standard objectives. In Street Unveiling, the scene semantics are expected to be maintained in a less messy and more consistent way to better recognize the Gaussians of objects to remove. However, solely naïve depth distortion won't hinder the merging of the 2DGS Huang et al. (2024a) with different semantic labels, leading to noisy semantic information about the 3D world. Meanwhile, the noisy Gaussians in the unseen region will still exist if we don't find a way to eliminate them. These problems will further harm the generation of an ideal inpainting mask.
Clean up objectives. To reduce the noise in the semantic fields, we propose semantic depth distortion loss Las and shrinking loss L. opacity
Cr. Σ ch
La
ap
(8)
p
5

---

Published as a conference paper at ICLR 2025
where & iterates over each semantic label and La denotes the distortion loss of 2DGS Huang et al. (2024a) with same semantic labels. This semantic depth distortion loss exerted on the rendered result of the Gaussians with the same semantic label. Intuitively, it will encourage the 2DGS with the same label to have a more consistent depth at the pixel level. Shrinking loss will further eliminate the Gaussians that are actually unseen by any viewpoint. Op represents the opacity value of each Gaussian. N is the total number of Gaussians.
The total loss given
== Ligh + Xaha Anhin Ads Ash, + Aafa
We empirically set Aa 100, A. = 0.05, Ads 100, X, 0.1, and Aa 0.001.
EMPTY STREET RECONSTRUCTION
common strategy Mirzaei al. (023): Weder al. (2023) inpaint within small scenes is utilizing 2D inpainting methods to inpaint removed objects in the image space for re-optimization.
However, lots of problems arise when it comes the street scene. (1) Some views result in over- blurry inpainting results due to the huge size of the inpainting mask, as is illustrated in Fig, bb;; (2) Some occluded regions of the street struggle to maintain consistency because they are exposed to a large number of views over the long trajectory. These challenges will make it more vulnerable to inconsistent inpainting.
In the context of point-based scene representation, eliminating the object involves deleting Gaussians. However, a na've removal often yields unsatisfactory results, particularly the completely unobserv- able regions beneath the object. In this section, we first propose how to generate the ideal mask for inpainting as in Fig. C(.. Then, we propose our time-reversal inpainting framework and how to use the inpainting results re-optimize the 2DGS.
5.1 GENERATION OF IDEAL INPAINTING MASK
In the street video captured by moving car, we can divide the pixel space into three categories: (I) The observable regions, where the regions are not occluded by any objects; (2) The partially observable regions, where the regions are occluded in some views but observable in other views; (3) The completely unobservable regions, where the regions are unobservable in all recording views. For regions in the second case, information about the appearance of the street scene. As illustrated in Fig. 2 naively inpainting with the object mask will cause the unexpected blurry inpainting result at the partially observable region, which can be viewed from other viewpoints but is occluded from the current viewpoint.
utilize information from other views to preserve more
To distinguish partially observable regions from completely unobservable regions and improve the inpainting quality, we propose using the rendered alpha map to generate the mask for completely unobservable regions. For given viewpoint, we first remove the Gaussians of unwanted objects. For robustness, we remove other Gaussians that are "too close" to the previously removed Gaussians. Then we render the alpha map of the remaining scene. We identify the completely unobservable region via pixels with low alpha values. The pixels with alpha values lower than threshold are selected as inpainting masks. The threshold set as 0.99 in our implementation.
5.2 TIME-REVERSAL INPAINTING
The core challenge in reconstructing the empty street scene is ensuring consistency between different views over the long trajectory. However, current video inpainting methods cannot generalize to our long trajectory and complex scenarios, which can be validated from Tab. [ Fig. 5 and supplementary video comparison. This usually lags behind the scale-up speed of image inpainting models. To this end, we propose using a reference-based image inpainting method that is trained to ensure consistency between the inpainted region and the reference-bact image. Particularly, we adopt the LeftRefill Cao et al. 2024) for its stable diffusion-
backbone and the matching based training strategy. The stable diffusion backbone leads to more powerful inpainting model with strong generation capacity in open-world scenarios, which fits the requirement of Street Unveiling. Furthermore, the matching-based training strategy ensures that the inpainting model correctly fulfills

---

Published as a conference paper at ICLR 2025
the masked region based on the observation in the reference image, which encourages consistency between different views.
Concatenate
Concatenate
Reference
Masied
Reference
Reference Based Inpainting
Based Incainting
Fewer Matches
More Matches
Figure 3: Illustration of reference- based inpainting of two views. Left: When inpaint the near view with the far view as a reference, the consistency of the inpainting result degenerates. There are fewer matching pixels between the reference far-view image and near-view inpainting result; Right: Inpainting the far view using the near view as reference results in better quality and more accurate pixel matching. It's easier generate the low-resolution content with the high-resolution image as reference.
However, fime-forward inpainting sequences usually lead to the failure of consistent inpainting. Given the moving-forware nature of data-collecting vehicles, objects to be removed transit from far to near in the camera view. (1) As is illustrated in Fig. B] when we use the far-view image reference to inpaint the same region the near-vies image, the models may not correctly capture the matching relationships and thus causing inconsistent inpainting. Conversely, setting the near-view image as the reference image leads to a more precise matching result and naturally better inpainting results. (2) The near-view image capture more fine-grained information and larger receptive field, thus making the inpainting easier to inpaint in high-to-low resolution instead of low-to-high which requires extra super-resolution capacity for the inpainting model. Besides, the objects removed in the final frame are consistently observed in the earlier frames.
Based on the above analysis. we propose the reversal inpainting framework. If we reverse the time, we can turn the moving-forwe forward nature into moving-backwart backward nature. When the time is reversed, objects to be removed will instead transition from near to far in the camera view because the camera will be away from the removed object in reversed time-space.
We target to unconditionally inpaint 3D region only once and then transmit the inpainted region's pixels to other views with reference-based inpainting. As is illustrated in Fig. 4 we first uncondition- ally inpaint both frame T, and T with Cao et al. (2023). However, for frame T,, there are some regions that can be seen in T,-1. We expect they would share more matching pixels by utilizing inpainting model Cao et al. (2024). Then we
the implicit pixel- matching ability of reference-base frame 1 reference inpaint frame T,, masking only the regions visible in T, +1 More implementation details are elaborated in Sec. A.2 the supplementary.
Unconditional Inpainting
Matching-aware
Time-Reversal Inpainting
Gaussians
Frome T,
Remove
Unconditional
Mask seen Frame Tres
Reference
Inpainting
Inpainting
Figure 4: Illustration of time-reversal inpainting. After remove the Gaussians of the objects, first unconditionally inpaint both frame T, and Ta+1 with with[Cao et al. (2023). Then we transmit the pixels from frame T,+1 frame T, in the form of reference- sierence-bbaee inpainting Cao et al. (2024). From a high-level understanding, we inpaint the earlier frame T, with the later frame T,+1 as reference condition.
7

---

Published as a conference paper at ICLR 2025
5.3 RE-OPTIMIZATION OF THE 2D GAUSSIANS
Once we finish time reversal inpainting, we use our inpainting results as pseudo labels to guide the re-optimization of 2DGS Huang et al. (2024a). We use the following loss for re-optimization:
Letrrinn Li + daCa + An
(10)
EXPERIMENTS
Our experiments were conducted on single NVIDIA A40 GPU with peak memory usage of 16GB.
Dataset. For the evaluation of our approach from the reconstruction aspect and the object removal aspect, we adopt real-workd street scenes from Waymo Open Perception Dataset [Sun et al.) (2020) and Pandaset Xıao et al. (202I). The Waymo dataset collects data from 5 camera perspectives.
encompassing roughly 230 degrees in field of view (FOV). We downscale the resolution to 484 320. The Pandaset collects data from 6 camera perspectives, encompassing 360 degrees in FOV. We downscale the resolution to 480 270, We select fiew video sequences as the same experimental setup in Yan et al. (2024); Chen al. (2023b); Zhou et al. (2024), using 24 scenes from Waymo and 9 scenes from Pandaset for our experiments.
Metrics. To evaluate the effectiveness of object removal, approach it from multi-view inpainting perspective. We follow the well-established previous works Mirzaei et al. (2023); Weder et al. (2023); Liu et al. 2024); [Ln et al. (2024a), calculate the LPIPS Zhang et al. (2018) and Fréchet Inception Distance (FID) Heusel et al. (2017) scores to quantify the discrepancies between the ground-truth views and removed results. Each output video frame is paired with the corresponding frame from the original training video to compute the LPIPS. We use the image collections of the output video and original training video compute FID.
Baselines. We compare our approach to 3D inpainting method SPIn-NeRF Mirzaei et al. (023) and a recent Gaussian Splatting based inpainting method Infusion Liu et al.) (2024). As the original MLP implementation of SPIn-NeRF Mirzaci et al. (2023) works poorly in the large-scale street scene, we re-implement SPIn-NeRF Mirzaei et al. (2023) based on 2DGS [Huang et al. (2024a), clarifying that our superiority not only from 2DGS but also the proposed time reversal inpainting. Infusion Liu et al.) (2024) is evaluated with the official implementation. Since Infusion Liu et al. (2024) is designed for small scenes, it only conducts GS removal and projection once for the whole scene. Its original setting doesn't match our long-trajectory tasks. Instead, we conduct every 10 frames to our setting.
Video Inpaint
3D Inpaint
Spin-NeRF (in 2DGS)
Training Data
Removed Object
ProPainter
Infusion
2LS
Figure 5: Qualitative comparison results of our methods. Our methods achieve clearer results than temporarily consistent inpainting baselines. Video comparisons will be placed in the supplementary.
8

---

Published as a conference paper at ICLR 2025
Waymo
Pandaset
LPIPS
FID +
LPIPS
FID
Single Image Inpainting
LaMa(2D) Suvorov al. SDXL Podell et al. (2023)
2021
0.228
138.089
0.276
160.895
0.231
116.634
0.276
133.042
Video Inpainting
ProPainter Zhou et al. 2023a)
0.233
141.906
0.286
178.135
3D Inpainting
SPIn-NeRF Mirzaei et al.
(0233
0.221
140.831
0.266
174.223
(in 2DGS)
Infusion Liu et al. (2024)
0.307
176.882
0.325
176.882
Ours
0.216
127.581
0.261
155.527
Table 1:
Comparison with state-of-the-art 2D/33D inpainting methods on both two datasets. Our FID only lower than SDXL, yet SDXL doesn't maintain consistency between different video frames.
Waymo
Pandaset
LPIPS
FID +
LPIPS
FID L
Ablation of different pseudo labels
w/LaMa Suvorov [202]
0,226
137.753
0,281
169.836
w/SDXL Podel et al. (2023)
0.222
139.716
0,279
169.805
w/ProPainter Zhou et (2023a
0.224
138.944
0.281
(69.848
Ablation of 3D representation
w/3DGS Kerbl et al. (023))
0.219
140.749
0.280
161.203
Time-Forward Inpainting
0.220
136.858
0.270
158.166
Ours
0.216
127.581
0.261
155.527
Table 2: Quantitative ablation study both two datasets over different 2D inpainting methods for 3D inpainting. And the ablation study over different 3D representations. The comparison verifies the effectiveness of the time-reversal inpainting pipeline and the necessity of the 2DGS representation.
6.1 COMPARISON
The quantitative comparison results are shown in Tab. 1 and the qualitative comparison of 3D inpainting methods are shown in Fig. [] Noticed that SPIn-NeRF Mirzaei al. (2023) utilizes LaMa (Suvorov et al. (2021) and Infusion Liu et al. (2024) utilizes SDXL et al. (2023) for inpainting. We can observe that 3D inpainting baseline methods lead to worse results, especially when the case is challenging. The results demonstrate that our proposed method achieves better 3D inpainting results from the appearance aspect. The geometry property of the removed region will be discussed in the supplementary. Video comparisons will also be included in the supplementary. In Tab. D. our proposed method outperforms all the baselines in I.PIS. It only achieves a lower FID) compared SDXXL, yet SIXL. doesn't maintain consistency between different video frames. This can be easily observed from supplementary videos.
6.2 FURTHER ANALYSIS
Ablation of different inpainting methods pseudo labels. We compare the reconstruction results with pseudo labels from different inpainting methods. From Fig. we can observe that time reversal will maintain the consistency between View 1 and View 2. Current single image inpainting models, like LaMa Suvorov et al. (2021) and SDXL Podell et al. 2023)). fail to maintain the consistency over the video frames. Although the video inpainting models Zhou et al. (2023a) can be temporarily consistent at near frames, the whole inpainting region will be blurred since it can not guarantee 3D consistency. The quantitative results in Tab. verify the effectiveness of our time-reversal pipeline.

---

Published as a conference paper at ICLR 2025
Training Data
Removed Object
LaMa
SDXL
ProPainter
Ours
2D Inpaint Guidance
ViewI
inconsistent
inconsistent
inconsistent
consistent
View
3D Inpaint Results
Figure 6: Ablation for different inpainting methods as pseudo labels. From the top rows, can observe that time-reversal inpainting is able to achieve more consistent inpainting results than other methods. The bottom row shows that our method can achieve better 3D inpainting results.
S9 /M
Gaussians
Remove
Gaussland
Remove
Gaussians
Ours
Remove
Gaussians
Rendered Image
Rendered Image
Generated
2D Inpaint
3D Inpaint
Inpainting Mask
Guidance
Result
Figure 7: Ablation for 3D representation. When we use 3DGS, generating an ideal inpainting mask with rendered alpha map hard. Good inpainting results are hard to achieve.
Ablation of 3D representation. We ablate through the 3D representation by comparing the results obtained with 3DGS Kerbl et al. (2023) and 2DGS Huang al. (2024a). From Fig. that after we remove the Gaussians, the rendered alpha map with 3DGS fails to generate an ideal inpainting mask. The quantitative results given in Tab. 2 verify the necessity of 2DGS representation.
observe
Ablation of time-reversal inpainting. For our time-reversal inpainting, we conduct inpainting on frame T, with T reference. We additionally ablate the time order in inpainting progress. For time-forward inpainting, frame T++1 is inpainted with frame Tn as reference. Tab, quantitatively demonstrates the necessity of the time-reversal order. We provide a more comprehensive discussion and qualitative illustration in Sec. B.1 of the supplementary.
CONCLUSION
We propose StreetUnveiler, a pipeline for reconstructing empty streets from in-car camera videos. Our method represents the street scene using hard-label semantic-aware 2D Gaussian Splatting Huang al. (2024a), allowing us to remove each instance from the scene seamlessly. To create an ideal inpainting mask, we utilize the rendered alpha map after removing unwanted 2DGS. Additionally, we introduce novel time-reversal inpainting framework that enhances consistency across different viewpoints, facilitating the reconstruction of empty streets. Extensive experiments demonstrate that our method effectively reconstructs empty street scenes and supports free-viewpoint rendering.
8 ACKNOWLEDGEMEN
The work supported NSFC #62172279, #6 #61932020, Program of Shanghai Academic Research Leader. Shuo Wang, Binbin Huang, Zibo Zhao are greatly appreciated for valuable proofreading.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Dragomir Anguelov, Carole Dulong, Daniel Filip, Christian Frueh, Stephane Lafon, Richard Lyon, Abhijit Ogale, Luc Vincent, and Josh Weaver. Google street view: Capturing the world street level. Computer, 43, 2010.
Chong Bao, Yinda Zhang, and Bangbang et al. Yang. Sine: Semantic-driven image-based nerf editing with prior-guided editing field. In CVPR, pp. 20919-20929, 2023.
Marcelo Bertalmio, Guillermo Sapiro, Vincent Caselles, and Coloma Ballester. Image inpainting. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques, pp- 417-424, 2000.
Chenjie Cao, Qiaole Dong, and Yanwei Fu. Zits++: Image inpainting by improving the incremental transformer on structural priors. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
Chenjie Cao, Yunuo Cai, Qiaole Dong, Yikai Wang, and Yanwei Fu. Leftrefill: Filling right canvas based on left reference through generalized text-to-image diffusion model. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2024.
Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In European Conference on Computer Vision (ECCV), 2022.
Yiwen Chen, Zilong Chen, Chi Zhang, Feng Wang. Xiaofeng Yang, Yikai Wang, Zhongang Cai, Lei Yang. Huaping Liu, and Guosheng Lin. Gaussianeditor: Swift and controllable 3d editing with gaussian splatting, 2023a.
Yurui Chen, Chun Gu, Junzhe Jiang, Xiatian Zhu, and Li Zhang. Periodic vibration gaussian: Dynamic urban scene reconstruction and real-time rendering. arXiv:2311.1856 2023b.
Kai Cheng, Xiaoxiao Long, Kaizhi Yang, Yao Yao, Wei Yin, Yuexin Ma, Wenping Wang, and Xuejin Chen. Gaussianpro: 3d gaussian splatting with progressive propagation. arXiv preprint arXiv:2402. 14650, 2024.
Chong Bao and Bangbang Yang, Zeng Junyi, Bao Hujun, Zhang Yinda, Cui Zhaopeng, and Zhang Learning disentangled neural mesh-based implicit field for geometry and texture editing. In European Conference on Computer Vision (ECCV), 2022.
Guofeng. Neumesh:
Zhiwen Fan, Wenyan Cong, Kairun Wen, Kevin Wang, Jian Zhang, Xinghao Ding, Danfci Xu, Boris Ivanovic, Marco Pavone, Georgios Pavlakos, Zhangyang Wang, and Yue Wang. Instantsplat: Unbounded sparse-view pose-free gaussian splatting in 40 seconds, 2024.
Jiemin Fang, Junjie Wang, Xiaopeng Zhang, Lingxi Xie, and Qi Tian. Gaussianeditor: Editing 3d
gaussians delicately with text instructions, 2023.
Xiao Fu, Shangzhan Zhang, Tianrun Chen, Yichong Lu, Lanyun Zhu, Xiaowei Zhou, Andreas Geiger, and Yiyi Liao. Panoptic nerf: 3d-to-2d label transfer for panoptic urban scene segmentation. In Proceedings of the International Conference on 3D Vision (3DV), 2022.
Haoyu Guo, Sida Peng, Haotong Lin, Qianqian Wang, Guofeng Zhang, Hujun Bao, and Xiaowei world
Zhou. Neural 3d scene reconstruction with the manhattan world assumption. In CVPR, 2022.
Jianfci Guo, Nianchen Deng, Xinyang Li, Yeqi Bai, Botian Shi, Chiyu Wang, Chenjing Ding, Dongliang Wang, and Yikang Li. Streetsurf: Extending multi-view implicit surface reconstruction to street views. arXiv preprint 236.:23066 2023.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. In Proceedings of the 3 st International Conference on Neural Information Processing Systems, NIPS' 17, pp. 6629-6640, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 97815 1188660699
Yuan-Ting Ting Hu, Heng Wang. Nicolas Ballas, Kristen Grauman, and Alexander G Schwing, Proposal- based video completion. In Computer Eision-ECCT 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXVII 16, pp. 38-54. Springer, 2020.
11

---

Published as a conference paper at ICLR 2025
Binbin Huang. Zehao Yu, Anpei Chen, Andreas Geiger, and Shenghua Gao. 2d gaussian splatting for geometrically accurate radiance fields. In SIGGRAPH 2024 Conference Papers. Association for Computing Machinery, 2024a. doi: 00.1145/364151933657722
Nan Huang, Xiaobao Wei, Wenzhao Zheng, Pengju An, Ming Lu, Wei Zhan, Masayoshi Tomizuka, Kurt Keutzer, and Shanghang Zhang. S3gaussian: Self-supervised street gaussians for autonomous driving. arXiv preprint arXiv:2405.20323. 2024b.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühllr, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4), July 2023. URL
https://repo- sam inria. fr/ fungraph haangggapph/dd--asssia-------- swwann-sppanttnngn
Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik. Lerf: Language embedded radiance fields. In International Conference on Computer Vision (ICCV), 2023.
Dahun Kim, Sanghyun Woo, Joon- Young Lee, and In So Kweon. Deep video inpainting. In Proceedings the IEEF/CVF conference on computer vision and pattern recognition, pp. 5792- 5801, 2019.
Diederik P Kingma and Max Welling, Auto-encoding variational bayes, 2022.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan- Yen Lo, Piotr Dollar, and Ross Girshick. Segment anything. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp. 4015-4026, October 2023.
Sosuke Kobayashi, Eicch Matsumoto, and Vincent Sitzmann. Decomposing nerf for editing via feature field distillation. Advances in Neural Information Processing Systems, volume 35, 2022. URL https: http://axxvvv.oooo www. ptppd///255. 15585... .ddf
Abhijit Kundu, Kyle Genova, Xiaoqi Yin, Alireza Fathi, Caroline Pantofaru, Leonidas Guibas, Andrea Tagliasacchi, Frank Dellaert, and Thomas Funkhouser. Panoptic neural fields: A semantic object- aware neural scene representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Christoph Lassner and Michael Zollhofer. Pulsar: Efficient sphere-based neural rendering. In CVPR, 2021.
Zhen Li, Cheng-Ze Lu, Jianhua Qin, Chun-Le Guo, and Ming-Ming Cheng. Towards an end-to-end framework for flow-guided video inpainting. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 17562-1757)) 2022.
Chieh Hubert Lin, Changil Kim, Jia-Bin Huang, Qinbo Li, Chih- Yao Ma, Johannes Kopf, Ming Hssan Yang, and Hung-Yo Yu Tseng. Taming latent diffusion model for neural radiance field inpainting. In European Conference on Computer Vision (ECCV), 2024a.
Jiaqi Lin, Zhihao Li, Xiao Tang, Jianzhuang Liu, Shiyong Liu, Jiayue Liu, Yangdi La, Xiaofei Wu, Songcen Xu, Youliang Yan, and Wenming Yang. Vastgaussian: Vast 3d gaussians for large scene reconstruction. In CVPR, 2024b.
Guilin Liu, Kevin Shih, Ting-Chun Wang, Fitsum A Reda, Karan Sapra, Zhiding Yu, Andrew Tao, and Bryan Catanzaro. Partial convolution based padding. Arxiv, 2018
Hongyu Liu, Bin Jiang, Yi Xiao, and Chao Yang. Coherent semantic attention for image inpainting. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4170-4179, 2019.
Qiankun Liu, Zhentao Tan, Dongdong Chen, Qi Chu, Xiyang Dai, Yinpeng Chen, Mengchen Liu, Lu Yuan, and Nenghai Yu. Reduce information loss transformers for pluralistic image impainting. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11347-11357, 2022.
Steven Liu, Xiuming Zhang, Zhoutong Zhang, Richard Zhang, Jun Yan Zhu, and Bryan Russell. Editing conditional radiance fields, 2021.
12

---

Published as a conference paper at ICLR 2025
Zhiheng Liu, Hao Ouyang, Qiuyu Wang, Ka Leong Cheng, Jie Xiao, Kai Zhu, Nan Xue, Yu Liu, Yujun Shen, and Yang Cao. Infusion: Inpainting 3d gaussians via learning depth completion from diffusion prior arXiv preprint 2401 2024.
Fan Lu, Yan Xu, Guang Chen, Hongsheng Li, Kwan- Yee Lin, and Changjun Jiang. Urban radiance field representation with deformable neural mesh primitives. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
Andreas Meuleman, Lu-Lun Liu, Chen Gao, Jia-Bin Huang, Changil Kim, Min H Kim, and Johannes Kopf. Progressively optimized local radiance fields for robust view synthesis. In Proc. IEEE/CVF 2023.
Conf. Computer Vision and Pattern Recognition, pp.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision (ECCV), 2020.
Ashkan Mirzaei, Tristan A Konstantinos G. Derpanis, Jonathan Kelly, Mar- cus A. Brubaker, Igor Gilitschenski, and Alex Levinshtein. SPIn-NeRF: Multiview segmentation and perceptual inpainting with neural radiance fields. In CVPR, 2023.
Ashkan Mirzaei, Riccardo De Lutio, Seung Wook Kim, David Acuna, Jonathan Kelly, Sanja Fidler, Igor Gilitschenski, and Zan Gojeic. Reffusion: Reference adapted diffusion models for 3d scene inpainting, 2024. URL https:://rxxv...ro www.:/i..ovorraaaaoo awtaab//22442 10765
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primi- July 2022. tives with multiresolution hash encoding. ACM Trans. Graph., (1(4) 353011 doi: 10. (0.1145/3522223.35520
URL
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2536-2544, 2016.
Songyou Peng, Kyle Genova, Chiyu "Max" Jiang, Andrea Tagliasacchi, Mare Pollefeys, and Thomas Funkhouser. Openscene: 3d scene understanding with open vocabularies. 2023.
Julien Philip and George Drettakis. Plane-based multi-view inpainting for image- rendering in large scenes. In Proceedings of the ACM SIGGRAPH Symposium Interactive 3D Graphics and Games (I3D), 2018.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis, 2023.
Kira Prabhu, Jane Wu, Lynn Tsai, Peter Hedman, Dan B Goldman, Ben Poole, and Michael Broxton. Inpaint3d: 3d scene content generation using 2d inpainting diffusion, 2023. URL https
//arxiv.org/abs/2331 03869
Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger. Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps. In International Conference on Computer Vision (ICCV), 2021.
Konstantinos Rematas, Andrew Liu, Pratul P. Srinivasan, Jonathan T. Barron, Andrea Tagliasacchi, Tom Funkhouser, and Vittorio Ferrari. Urban radiance fields. CVPR, 2022.
Jingjing Ren, Qingqing Zheng, Yuanyuan Zhao Xuemiao Xu, and Chen Li. Diformer: Discrete latent transformer for video inpainting. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 3511-3520, 2022.
Kerui Ren, Lihan Jiang, Tao Lu, Mulin Yu, Linning Xu, Zhangkai Ni, and Bo Dai. Octree-gs: Towards consistent real-time rendering with lod-structured 3d gaussians, 2024.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), 2022.
13

---

Published as a conference paper at ICLR 2025
Sara Fridovich-Kan Keil and Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In CVPR, 2022.
Johannes Lutz Schönberger and Jan-Michael Frahm. Structurefrrommmttion revisited. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Johannes Lutz Schönberger, Enliang Zheng, Mare Pollefeys, and Jan-Michael Frahm. Pixelwise view selection for unstructured multi-view stereo. In European Conference Computer Vision (ECCV), 2016.
Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Bulò, Norman Müller, Matthias NieBner, Angela Dai, and Peter Kontschieder. Panoptic lifting for 3d scene understanding with neural fields. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),
pp. 9043 00022905,,440 June 2023.
Cheng Sun, Min Sun, and Hwann-Tzong Chen. Direct voxel grid optimization: Super-fast conver- gence for radiance fields reconstruction. CVPR, 2022.
Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, Vijay Vasudevan, Wei Han, Jiquan Ngiam, Hang Zhao, Aleksei Timofeev, Scott Ettinger, Maxim Krivokon, Amy Gao, Aditya Joshi, Yu Zhang, Jonathon Shlens, Zhifeng Chen, and Dragomir Anguelov. Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksei Silvestrov, Naejin Kong, Harshith Goka, Kiwoong Park, and Victor S. Lempitsky. Resolution-robust large mask inpainting with fourier convolutions. 2022 IEEE/CVF Win- ter Conference on Applications of Computer Vision (WACV), pp. 3172-3182, 2021. URL
https: //aps www.eessseaaticccch..oooo www.hhooarrrrg/ooppouuuoe wtg/CorpusIi:: ::: 227113333333
Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul Srinivasan, Jonathan T. Barron, and Henrik Kretzschmar. Block-NeRF: Scalable large scene neural view synthesis. arXiv, 2022.
Theo Thonat, Eli Shechtman, Sylvain Paris, and George Drettakis. Multi-view inpainting for image- based scene editing and rendering. In Proceedings of the International Conference on 3D Vision (3DV), 2016.
Haithem Turki, Deva Ramanan, and Mahadev Satyanarayanan. Mega-nerf: Scalable construction of large-scale nerfs for virtual thyouhhs In CVPR, pp.
June 2022.
Haithem Turki, Jason Y Zhang, Francesco Ferroni, and Deva Ramanan. Suds: Scalable urban dynamic scenes. In Computer Vision and Pattern Recognition (CVPR), 2023.
Ziyu Wan, Jingbo Zhang, Dongdong Chen, and Jing Liao. High-fidelity pluralistic image completion with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4692-4701 2021.
Chuan Wang, Haibin Huang, Xiaoguang Han, and Jue Wang. Video inpainting by jointly learn- ing temporal structure and spatial details. Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 5222-5239, 2019.
Dongqing Wang, Tong Zhang, Alaa Abboud, and Sabine Susstrunk. Inpaintnerf360: Text-guided 3d inpainting on unbounded neural radiance fields. arXiv, 2023a.
Peng Wang, Yuan Liu, Zhaoxi Chen, Lingjie Liu, Ziwei Liu, Taku Komura, Christian Theobalt, and Wenping Wang. F2-nerf: Fast neural radiance field training with free camera trajectories. CVPR. 2023b.
Yifan Wang, Andrew Liu, Richard Tucker, Jiajun Wu, Brian L. Curless, Steven M. Seitz, and Noah Snavely. Repopulating street scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
14

---

Published as a conference paper at ICLR 2025
Yikai Wang, Chenjie Cao, Ke Fan, Qiaole Dong, Yifan Li, Xiangyang Xue, and Yanwei Fu. Repo- sitioning the subject within image. Transactions on Machine Learning Research, 2024a. ISSN 2835-8856.
Yikai Wang. Chenjie Cao, and Ke Fan Xiangyang Xue Yanwei Fu. Towards context-stable and visual-consistent image inpainting, 2024b.
Yuxin Wang, Qianyi Wu, Guofeng Zhang, and Dan Xu. Gseream: Learning 3d geometry and feature consistent gaussian splatting for object removal. In ECCV, 2024c.
Zian Wang, Tianchang Shen, Jun Gao, Shengyu Huang, Jacob Munkberg, Jon Hasselgren, Zan Gojeic, Wenzheng Chen, and Sanja Fidler. Neural fields meet explicit geometric representations for inverse rendering of urban scenes. In The IEEE Conference Computer Vision and Puttern Recognition (CVPR), June 2023c.
Ethan Weber, Aleksander Holynski, Varun Jampani, Saurabh Saxena, Noah Snavely, Abhishek Kar, and Angjoo Kanazawa. Nerfiller: Completing scenes via generative 3d inpainting. In CVPR, 2024.
Silvan Weder, Guillermo Harnia- Aron Monszpart, Marc Pollefeys, Gabriel Brostow, Michael Firman, and Sara Vicente. Removing objects from neural radiance fields. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Tiange Xiang, Adam Sun, Jiajun Wu, Ehsan Adeli, and Fei-Fei Li. Rendering humans from object-occluded monocular videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2023.
Pengchuan Xiao, Zhenlei Shao, Steven Hao, Zishuo Zhang, Xiaolin Chai, Judy Jiao. Zesong Li, Jian Wu, Kai Sun, Kun Jiang, Yunlong Wang, and Diange Yang. Pandaset: Advanced sensor suite dataset for autonomous driving. In 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), pp. 3095-33000 1095-3101. IEEE Press, 2021. doi: 10. 000.9//TSSC89772222226666665 2021. 9565009
URL httph: org/ 10. 111.9/1T14489999
Yuting Xiao, Jingwei Xu, Zehao Yu, and Shenghua Gao. Debsdf: Delving into the details and bias of neural indoor scene reconstruction. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2024.
Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. In Neural Information Processing Systems (NeurIPS), 2021.
Linning Xu, Yuanbo Xiangli, Sida Peng, Xingang Pan, Nanxuan Zhao, Christian Theobalt, Bo Dai, and Dahua Lin. Grid-guided neural radiance fields for large urban scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Qiangeng Xu, Zexiang Xu, Julien Philip, Sai Bi, Zhixin Shu, Kalyan Sunkavalli, and Ulrich Neumann. Point-nerf: Point-based neural radiance fields. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4488-548,, 2022.
Rui Xu, Xiaoxiao Li, Bolei Zhou, and Chen Change Loy. Deep flow-guided video inpainting. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Yunzhi Yan, Haotong Lin, Chenxu Zhou, Weijie Wang, Haiyang Sun, Kun Zhan, Xianpeng Lang, Xiaowei Zhou, and Sida Peng. Street gaussians: Modeling dynamic urban scenes with gaussian splatting. In ECCV, 2024.
Bangbang Yang, Wenqi Dong, Lin Ma. Wenbo Hu, Xiao Liu, Zhaopeng Cui, and Yuewen Ma. Dreamspace: Dreaming your room space with text-driven panoramic texture propagation. 2023a.
Ze Yang, Yun Chen, Jingkang Wang, Sivabalan Manivasagam, Wei-Chiu Ma, Anqi Joyce Yang, and Raquel Urtasun. Unisim: neural closed-loop sensor simulator. In CVPR, 2023b.
Mingqiao Ye, Martin Danelljan, Fisher Yu, and Lei Ke. Gaussian grouping: Segment and edit anything in 3d scenes, 2023.
15

---

Published as a conference paper at ICLR 2025
Zili Yi, Qiang Tang, Shekoofeh Azizi, Daesik Jang. and Zhan Xu. Contextual residual aggregation for ultra righuuesotttion image inpainting. In IEEE Conf. Comput. Vis. Pattern Recog. (CVPR), pp. 7508 7000--5177211 2020.
Jiahui Yu, Zhe Lin, Jimei Yang. Xiaohui Shen, Xin Lu, and Thomas S Huang. Generative image inpainting with contextual attention. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5505-5514, 2018.
Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Free-form image inpainting with gated convolution. In Proceedings of the IEEE International Conference on Computer Vision (ICCV). pp. 4471 1470-, 2019.
Zehao Yu, Songyou Peng, Michael Niemeyer, Torsten Sattler, and Andreas Geiger. Monosdf: Exploring monocular geometric cues for neural implicit surface reconstruction. Advances in Neural Information Processing Systems (NeurIPS), 2022.
Yu-Jie Yuan, Yang-Tian Sun, Yu-Kun Lai, Yuewen Ma, Rongfei Jia, and Lin Gao. Nerf-editing: Geometry editing of neural radiance fields. In CVPR, 2022.
Zefeng Yuan, Hengyu Li, Jingyi Liu, and Jun Luo. Multiview scene image inpainting based on conditional generative adversarial networks. IEEE Transactions on Intelligent Vehicles,502 June 2020.
Kai Zhang, Nick Kolkin, Sai Bi, Fujun Luan, Zexiang Xu, Eli Shechtman, and Noah Snavely. Arf: Artistic radiance fields, 2022.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3836-3847, 2023a.
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In 2018 IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 586-595, 2018.
doi: 10. 20Wp/CCPR.2018.0000000
Xiaoshuai Zhang, Abhijit Kundu, Thomas Funkhouser, Leonidas Guibas, Hao Su, and Kyle Genova. Nerflets: Local radiance fields for efficient structure-aware 3d scene representation from 2d supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023b.
Shengyu Zhao, Jonathan Cui, Yilun Sheng, Yue Dong. Xiao Liang. Eric, Chao Chang, and Yan Xu. Large scale image completion via co-modulated generative adversarial networks. In International Conference on Learning Representations, 2020.
Yiqun Zhao, Chenming Wu, Binbin Huang, Yihao Zhi, Chen Zhao, Jingdong Wang, and Shenghua Gao. Surfel-based gaussian inverse rendering for fast and relightable dynamic human reconstruction from monocular video. arXiv preprint arXiv: 1 152 12, 2024a.
Yiqun Zhao, Zibo Zhao, Jing Li, Sixun Dong, and Shenghua Gao. Roomdesigner: Encoding anchor- latents for conssssstttntt and chape-compatible indoor scene generation. In Proceedings of the International Conference on 3D Vision (3DV), 2024b.
Zibo Zhao, Wen Liu, Yanyu Xu, Xianing Chen, Weixin Luo, Lei Jin, Bohui Zhu, Tong Liu, Binqiang Zhao, and Shenghua Gao. Prior based human completion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7951-7961, 2021.
Hongyu Zhou, Jiahao Shao, Lu Xu, Dongfeng Bai, Weichao Qiu, Bingbing Liu, Yue Wang, Andreas Geiger, and Yiyi Liao. Hugs: Holistic urban 3d scene understanding via gaussian splatting, 2024.
Qian- Yi Zhou, Jacsik Park, and Vladlen Koltun. Open3D: A modern library for 3D data processing. arXiv:1801.09847, 2018.
Shangchen Zhou, Chongyi Li, Kelvin C.K Chan, and Chen Change Loy. ProPainter: Improving propagation and transformer for video inpainting. In Proceedings of IEEE International Conference Computer Vision (ICCV) 2023a.
16

---

Published as a conference paper at ICLR 2025
Shijie Zhou, Haoran Chang, Sicheng Jiang, Zhiwen Fan, Zehao Zhu, Dejia Xu, Pradyumna Chari, Suya You, Zhangyang Wang, and Achuta Kadambi. Feature 3dgs: Supercharging 3d gaussian splatting to enable distilled feature fields. arXiv preprint arXiw:2512.05200, 2023b.
Xueyan Zou, Linjie Yang. Ding Liu, and Yong Jae Lee. Progressive temporal feature alignment network for video inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16448-164577 2021.
17

---

Published as a conference paper at ICLR 2025
A IMPLEMENTATION DETAILS
A.1 DETAILS OF HARD-LABEL SEMANTIC 2DGS RECONSTRUCTION
Initialization with Lidar points. High-quality appearance and semantic reconstruction of the whole street scene are hard to reach, with barely SFM points SScööneergrr & Frahm| (2016); [Schönberger al. (2016) as initialization for street scenes. Lidar points are leveraged better reconstruct the street scene like in Yan et al. (2024); ; Chen et al. (023b); Zhou et al. (2024). We use an off-the-self 2D semantic segmenter Xie et al. (2021 to process the 2D images and back maok-project the hard semantic labels to 2D Gaussians Huang et al. (2024a).
Environment map for street reconstruction. We empirically find that most 2D Gaussians' opacity will be larger than 0.9 or lower than 0.1, leading to the imperfect reconstruction quality of the background environment, i.e., sky. To better model the environment in the street scene, we employ a tiny MLP f to query the color of the environment map, which is similar to Guo et al. (2023); Turki cct al. (2023). The queried environment color at x is denoted as Cenv- The final color of the ray is obtained by blending the color of 2DGS projection and the environment map follows:
Cenv (x) ((M x)
(x)aa()) c(x) + (1 a (em)) (x)
(11)
where M denotes the projection matrix from world coordinates to pixel coordinates. a(x) is the rendered alpha map of 2DGS rendering.
Details of two-stage reconstruction training
The optimization of our designed 2DGS Huang et al. (2024a) reconstruction for street scenes contains two stages. (1) In the first stage, employ adaptive density control of 3DGS Kerbl et al. (2023). And La. C, and Las will be deactivated to reach a more stable initialization of 2DGS reconstruction. (2) In the second stage, La Lu and Las is activated. As empirically, most 2D Gaussians' opacity will be larger than 0.9 or lower than 0.1. The noisy 2DGS with the wrong semantic label will be optimized low opacity through Las- We prune the Gaussians with opacity lower than a threshold to further eliminate the noisy semantics in the 3D world, with set as 0.3 in our experiments.
A.2 DETAILS OF REME-REVERSAL INPAINTING FRAMEWORK
As is mentioned in Wang al. 2024b). when we are using latent-diffussonn-baae inpainting model, there will be non- non-ignorable shifts in low- fre.ueeegeny fields if we use images decoded by KL- ViE|Kingmaa Welling (2022: Rombach al. (2022) repeatedly for different times. Given that our method can be summarised as inpainting frame T, with Ti+1 as reference through LeftRefil Cao al. (022)), which is aatent-diffusonn-aaee based. For whole sequence of video, if we simply iteratively inpaint every T; with T.+1 reference, the shifts in low-frequency fields will be badly augmented by KL-VAE, which will severely harm the quality of our 2D inpainting guidance. To alleviate this inevitable shift from the KL-VAE of the latent diffusion model Rombach al. (2022). We first select some keyframes in the video. Then we use time-reversal inpainting to inpaint the selected keyframes iteratively in the reversed time sequence.
We firstly select the keyframes of timestamps .T....... TR, }. and we start to inpainting all the keyframes in the reversed time sequence. After inpaint the keyframe TK, we generate the middle frames between keyframe Tk... and keyframe Tr, with keyframe Tk, as reference image. Per-image processing follows Fig. Finally, we will use these results as pseudo-labeled data to further re-optimize the 2DGS of the empty street scene.
To achieve more precise scene optimization, we first identify the 2DGS for removal using hard semantic labels. After removing these 2DGS, we restrict the re-optimization stage to only update Gaussians that are spatially close to the removed ones, ensuring targeted refinement.
B MORE EXPERIMENTS
B.1 ABLATION OF FIRE-DORWARD INPAINTING AND REVERSEVRRALL INPAINTING
To further validate the effectiveness of reverselersal inpainting, we do an additional ablation in Sec. 6.2 with time-forward inpainting, which is the reverse version of our proposed time-reversal
18

---

Published as a conference paper at ICLR 2025
inpainting. In Tab. B and Tab. our time-reversal inpainting achieves better quantitative results than fime-forward inpainting.
For time-reversal impainting, inpaint frame Tn with Tn+1 reference. For time-forward inpainting, frame T is inpainted with frame T, as reference. The Fig. claborates the details about the process of these two methods. The qualitative comparison in showcases the high-to- low-resolution nature of rime-reversal inpainting, which will enhance the quality of the results.
Frame Tixt
Te-d
Same
Inpainting
Frame 1
Frame Tx
Reference Based Inpainting
Reference Inpainting
Ta-1
Reference
Frame
Reference
Reference
Based Inpsinting
Rased
Time- Rene-RRvarssll Inpaintinn
Time-Forward Inpainting
Figure 8: Illustration of the difference between time-reversal inpainting and time-forward inpainting on inpainting strategy. We first use unconditional inpainting to inpaint the frame T.+1- For time-reversal inpainting, inpaint frame Tn with T,+1 reference. For time-forward inpainting, frame T1 is inpainted with frame T2 as reference.
Ty-1
Frame T
Same
Tn
Frame T
Better
Frame Tn+1
Frame Tam
Better
Time Reversal Inpainting
Time-Forward Inpainting
Figure 9: Illustration of the qualitative comparison between time-reversal inpainting and time- forward impainting. We can observe that the quality of time-forward inpainting results degenerates because time-forward inpainting uses -ii-too-hig--rsolttionn approach. This requires extra super- resolution capacity for the inpainting model to get better result. However, our time-reversal inpainting uses a high-to-low-resolution approach. High-resolution content will better guide the low-resolution content.
B.2 ABLATION OF HARD SEMANTIC LABEL
We additionally ablate the effectiveness of the hard semantic label. From Fig. both 2DGS representation and hard semantic label contribute to a more stable reconstruction of the semantic field.
observe that
The comparison between (a) and (b) demonstrates that the use of hard semantic labels effectively reduces noise within the semantic fields. In addition, the comparison between (a) and (c) indicates
19

---

Published as a conference paper at ICLR 2025
Waymo
Pandaset
LPIPS4
FID
LPIPS
FID +
Time-Forward Inpainting
0.220
136.858 127.581
0.270 0.261
158.166
ReverRe Inpainting(Ours)
0.216
155.527
Table 3: Quantitative ablation of time-reversal inpainting and time-forward inpainting. The result validates the effectiveness of our method.
Waymo
Pandaset
LPIPS|
FID 4
LPIPS
FID +
LeftRefill Cao et al. (0024)
0.227
135,421 127.581
0.288 0.261
168.112 155.527
Ours
0.216
Table 4: Quantitative comparison with LeftRefill Cao et al. (2024).
The result validates the effective-
ness of our method.
that the 2DGS representation leads to more stable semantic fields. Finally, (d) illustrates the clean and stable semantic field achieved by employing hard-label semantic 2DGS in our method.
By reconstructing clean and stable semantic field of the street scene, we can more accurately identify the Gaussians that need be removed. This facilitates obtaining high-quality 2D) inpainting result, which serves as effective guidance for re-optimization.
B.3 COMPARISON WITH LEFTREFILL
We additionally discuss the qualitative comparison with LeftRefill Cao al. (2024) as another baseline. Since LeftRefill requires an image as a reference, LeftRefill can't I be naturally run as unconditional inpainting methods like LAMA [uvvrrvv al. (202I) and SDXL et (2023) in Tab. [ We adapt LeftRefill with the 10th future frame as a condition and use the mask obtained after our reconstruction stage. LeftRefill also operated reverse order.
From Tab. we can observe that our time-reversal inpainting pipeline generates better results than LeftRefill. From Fig. [0] we observe that due this adapted naive reverse inpainting with LeftRefill taking the future frame a reference, some regions are not visible in the future frame. This limitation can lead to low-quality inpainting, as highlighted in the red frame LeftRefill's result. contrast, pipeline generates more natural inpainting result through 2DGS re-optimization, ultimately achieving clear 3D inpainting result.
2D inpaint
3D inpaint
Removed Object
LeftRefill
Ours pseudo supervision
Figure 10: Illustration comparison with LeftRefill Cao al, (2024). Since naive reverse inpainting with LeftRefill takes the future frame as reference, some regions are not visible in the future frame. We observe that this will lead to a low quality of inpainting, as highlighted in the red frame of LeftRefill's result. Our pipeline generates more natural inpainting result for 2DGS re-optimization and finally obtains a clear 3D inpainting result.
20

---

Published as a conference paper at ICLR 2025
(a) 3DGS w/soft label
(b) 3DGS w/hard label
(c) 2DGS w/soft label
(d) 2DGS w/hard label(Ours)
Figure 11: Illustration of hard semantic label ablation. Black rectangles in the figures include the
noise in the semantic fields. The comparison between (a) and (b) demonstrates that hard semantic labels effectively reduce noise within semantic fields. Similarly, the comparison between (a) and (c) indicates that the 2DGS representation contributes more stable semantic fields. Finally, (d) illustrates the clean and stable semantic field achieved by employing hard-label semantic 2DGS in our method.
21

---

Published as a conference paper at ICLR 2025
Dataset(Frames)
Reconstruction
Our Inpaint
Re-optimize
Waymo
10016 sec
1035 sec
524 sec
Pandaset(80)
9640 sec
311 sec
257 sec
Table 5: Computational cost of each stage with our pipeline seconds): We evaluate on both Waymo and Pandaset. We use scenes from each dataset and average their time consumption experiments. "Reconstruction" is the main efficiency bottleneck in our pipeline.
Waymo (198)
Pandaset (80)
Method
LaMa Suvorov et al. (2021)
0.20 sec
0.28 sec
SDXL Podell et (203))
5.18 sec
5.21 sec
Propainter| Zhou et al. (2023a)
0.59 sec
0.53 sec
Ours
5.22 sec
5.09 sec
Table 6: Total per-frame time cost comparison from 2D inpainting aspect: We use 8 scenes from each dataset and average their time consumption in experiments. Our method comparable SDXL for per-frame time cost.
B.4 QUALITATIVE COMPARISON OF RENDERED GEOMETRY
Since want reconstruct the empty street, also want compare the geometry property of our method other than just appearance. From Fig. [4] Fig. 15 Fig. 16 Fig. 17 we can observe that
our method produces both better appearance quality and geometry quality from rendered RGB and rendered normal images.
B.5 COMPUTATIONAL ANALYSIS
We additionally conduct a computational analysis of each stage our pipeline and per-frame inpainting time cost. From Tab. B we observe that the "Reconstruction" stage is the main efficiency bottleneck our pipeline. By utilizing recent techniques like Instantsplat et al. (2024), our whole pipeline has the potential to be accelerated and may reach the result of reconstructing empty street in 30 minutes. From Tab. [6] our pipeline isn't inferior to the Diffusion-based PPdell et al. (2023) method from the efficiency aspect.
C ADDITIONAL RESULTS
C.1 EMPTY STREET SCENE MESH EXTRACTION
We can further extract the mesh for our reconstructed empty street scene using TSDF fusion following 2DGS Huang al. (2024a) with Open3D Zhou al. (2018). In Fig. 19aadd Fig. 20 we visualize extracted colored mesh before and after our unveiling. Our inpainting framework can successfully remove unwanted cars from the street and finally reconstruct an empty street mesh representation. Mesh extraction results further verify the correct geometry produced through method.
We clarify that our problem formulation not exactly the same as StreetSurf Guo et al. (2023). The target of our method is to reconstruct the empty street. However, the extracted mesh and reconstructed street StreetSurf is the original static scene "before unveiled".
Another key difference is that our setting lacks ground-truth "after unveiled" training data for both Lidar and images. StreetSurf relies on ground-truth "before unveiled" data for both training and evaluation.
While we are still able to evaluate the reconstructed "before unveiled" scenes to compare with StreetSurf, which will provide meaningful insights for future work.
Following StreetSurf, we utilize the Lidar data under the real-world scale for geometry evaluation. In StreetSurf, the extracted mesh may include some parts outside of the scene. To ensure fair
22

---

Published as a conference paper at ICLR 2025
Monocular Prior
Method
CD2
F-Score
Geometry-related input
Lidar
StreetSurf Guo et al. (002))
0.52
56.70
StreetSurfGins al. (2023)
Ours(before unverled)
0.55
61.54
Ours(before unvelled)
Table 7: Comparison of the reconstruction performance on "before unveiled" geometry with Street- Surf Guo et al. (023)) on 24 scenes from Waymo dataset [unn et al. (2020). Left: We observe that our Chamfer Distance is higher than StreetSurf, while F-Score is higher than StreetSurf. The reconstruction performance of observed geometry appears to be comparable. Right: The checkbox of the geometry-relatee input data. We rum StreetSurf with both monocular prior and Lidar as input. While in approach, operate without relying the monocular geometry prior.
comparison, we crop these out-of-range meshes from the extracted mesh. Specifically, any mesh that more than 5 meters away from the closest Lidar point will be cropped.
For the experimental setup, we select both Chamfer Distance(CD) and F-score with 0.25-mette meter threshold as our geometry evaluation metrics. We evaluate StreetSurf using both monocular geometry priors and Lidar as scometry-relatee inputs, while our method tested exclusively with Lidar as the geometry- input.
From Tab. 7 we observe that our Chamfer Distance is greater than that of StreetSurf, while our Score surpasses StreetSurf'ss s. From Fig. [2] empirically observe that the StreetSurf achieves higher accuracy on the observable ground the street. Even after fair mesh cropping, the reconstructed mesh of StreetSurf is still affected by out-of-range meshes some extent. This results from the inherent methodological differences between SDF extraction and TSDF fusion, which may lead to a slightly lower F-Score for the SDF extraction approach. Overall, the reconstruction performance of the observed geometry appears to be comparable.
StreetSurf
Ours
Figure 12: Illustration of the "before unveiled" geometry comparison with StreetSurf Guo et al. (2023). "Green" indicates "more accurate" regions, while "Red" represents "less accurate' regions. We empirically observe that the StreetSurf achieves higher accuracy on the observable ground of the street. Even after fair mesh cropping. the reconstructed mesh of StreetSurf is still influenced by out-of-range meshes to some extent. This effect stems from inherent methodological differences between SDF extraction and TSDF fusion, which may contribute slightly lower F-Score for the SDF extraction approach. Overall, the reconstruction performances are comparable.
C.2 EXAMPLE OF REMOVING THE STANDING PEDESTRIAN
As in Fig. 21 highlight example of removing the standing pedestrian from the scene.
C.3 MORE VISUAL COMPARISON
We provide additional qualitative comparisons of inpainting results for the Pandaset dataset Xiao et al. (021) in Fig.
23

---

Published as a conference paper at ICLR 2025
C.4 VIDEO VISUALIZATIONS
In order to conveniently view our video results, we prepare a web viewer at /iinde.hhml" html" from the root path of the supplementary materials.
C.4.1
NOVEL VIEW SYNTHESIS VIDEOS VISUALIZATIONS
As in Fig.
23
showcase two novel view synthesis videos. The file paths are illustrated.
C.4.2
MORE VIDEO VISUALIZATIONS
As in Fig. 25 we visualize three scenes involved in Tab for better comparison. The file paths are illustrated. From video comparison, can be observed that our method outperforms other baselines.
D DISCUSSION ON SEMANTIC LABEL SUPERVISION
We use SegFormer Xie al. (2021) as the pre-trained model for segmentation, and our results appear to be robust as whole. Both our quantitative and qualitative results show that our method is stable with SegFormer.
While it is inevitable that certain failure cases occur with small objects object corners, these challenges are common across most segmentation methods. The example cases are illustrated in Fig. 13 As segmentation techniques continue evolve, method is poised to benefit and improve
alongside them.
E DISCUSSION ON DYNAMIC OBJECT REMOVAL
We illustrate a case to handle simple dynamic object removal in Fig. 22 which can be observed the scene 3(a moving car) in the supplementary videos. For more challenging dynamic cases, utilizing optical flow or dynamic object detection over the video sequence to identify dynamic objects from 21) may be considerable solution.
Alternatively, we can model dynamic Gaussians in street scenes like Chen et al. (023b)) and Yan al. (2024) for more challenging dynamic object removal.
The separation of dynamic and static elements may also facilitate the removal of dynamic objects from scenes. Current methods, such StreetGaussians Yan et al. (2024) (which utilizes 3D box-based scene decomposition) and S3Gaussian Huang et al. (2024b) (a self-supervised approach to scene decomposition), aim distinguish between dynamic and static components. However, these methods may not consistently differentiate static removable objects (like stopping cars) from essential scene elements (like traffic signs).
DISCUSSION ON 360 OR WIDE RANGE SURROUNDING VIDEOS
Our time-reversal inpainting pipeline works for "forward-facing cameras. Thus, we used frontal cameras for the experiment, which is the same as the experimental setup in Yan et al. (2024): Chen al. (2023b); Zhou al. (2024). found the inpainting results of the frontal view sufficient for recovering 3D unveiled street with satisfactory geometry properties. For left-back and right- back(side and back cameras), our method doesn't naturally promise to maintain the consistency of the inpainting. The homography technique may help maintain consistency between different views by leveraging the overlaps in images across those views.
G SOCIETAL IMPACTS
This technology can distort public space representations urban planning, potentially leading to flawed decisions. Additionally, it may be misused to alter important archaeological sites in digital reconstructions, resulting in misinformation about historical facts.
24

---

Published as a conference paper at ICLR 2025
STOP
0162
158
Figure 13: For the semantic segmentation mask predicted by SegFormer Xie et al. 2021 in original full resolution, some noise exists at the boundaries between regions with different semantic tags. These failure cases likely to occur the corners of small objects.
25

---

Published as a conference paper at ICLR 2025
Original Scene
Reconstructed Normal
SPIn-NeRF (in 2DGS)
SPIn-NeRF (in 2DGS)
Normal
Infusion
Infusion Normal
Ours 3D Removal
Normal
Figure 14: Illustration of geometry performance comparison.
26

---

Published as a conference paper at ICLR 2025
Original Scene
Reconstructed Normal
SPIn-NeRF (in 2D65)
SPIn-NeRF (in 2DGS)
Normal
Infusion
Infusion Normal
Ours 3D Removal
Ours Normal
Figure 15: Illustration of geometry performance comparison.
27

---

Published as a conference paper at ICLR 2025
Original Scene
Reconstructed Normal
SPIn-NeRF (in 2DGS)
SPIn-NeRF (in 2DGS)
Normal
Infusion
Infusion Normal
Ours 3D Removal
Ours Normal
Figure 16: Illustration of geometry performance comparison.
28

---

Published as a conference paper at ICLR 2025
Original Scene
ST
Reconstructed Normal
SPIn-NeRF (in 2DGS)
ST
SPIn-NeRF (in 2DGS) Normal
Infusion
ST
Infusion Normal
Ours 3D Removal
215
Ours Normal
Figure 17: Illustration of geometry performance comparison.
29

---

Published as a conference paper at ICLR 2025
Original Scene
Reconstructed Normal
SPIn-NeRF (in 2DGS)
SPIn-NeRF (in 2DGS)
Normal
Infusion
Infusion Normal
Ours 3D Removal
Ours Normal
Figure 18: Illustration of geometry performance comparison.
30

---

Published as a conference paper at ICLR 2025
Reconstruction
Unveiled
Reconstruction
Unveiled
Figure 19: Illustration of colored mesh comparison between before and after the unveiling.
31

---

Published as a conference paper at ICLR 2025
Reconstruction
Unveiled
Reconstruction
C01S
Unveiled
Figure 20: Illustration of colored mesh comparison between before and after the unveiling.
32

---

Published as a conference paper at ICLR 2025
Training data
Our unveiled
Figure 21: Illustration of removing the standing pedestriam.
Training data
Our unveiled
Figure 22: Illustration of removing simple dynamic case.
Novel View Synthesis Videos
VIDEO
/ssaticfwidoos///////vss....
/statickviddoo//////ssss..m
Figure 23: Illustration of novel view synthesis videos and their file paths. It's recommended to open our web viewer located at "_findexhtml"".
33

---

Published as a conference paper at ICLR 2025
Video Inpaiint
3D Inpaint
Training Data
Removed Object
ProPainter
Spin-NeRF (in 2DGS)
Infusion
Ours
Figure 24: More qualitative comparison on Pandaset dataset Xiao al. (2021). Our method produced
a clearer result of the ground and trees behind the removed object compared to the baselines.
data_vs_ours.mp4
scene1
infusion_vs_ours.mpp
lama_vs_ours.mp4
//static/videos
/Experiment/
propainter_vs_ours.mne
sdxl_vs
vs _______ours.pp4
scene2
spin_vs_ours.nne
vs vs___uu_urrmppp
scene3
Figure 25: Illustration of video comparison with baseline and their file paths. It's recommended to open our web viewer located at " "//nndxxhtmll". "data_vs_oo vs_oors..pp shows our results compared with training data for visualization. "infusion _a___ss_oouree ss_ouussppp" shows both RGB and normal results between Infusion and our method. "Jama_vs _s___v__o__ vs_ourr.mp4"" shows our results compared to LaMa. "propainter_ vs_ vs_ours.mp4" shows results compared ProPainter, which is state-of-the-art video inpainting method. "sdxl_vs se_vvvssss____eee "ss_ours.mpp"" shows our results compared to SDXL. "spin_w satpi_ss_oussuu.aee "_ours.mp4" shows both RGB and normal results between SPIn-NeRF in 2DGS representation and our method.
34


Reviews:
[{"review_id": "SmP3hRl1Io", "reviewer": "Reviewer_SpDA", "summary": "This paper introduces StreetUnveiler, a method to reconstruct empty street scenes from crowded video captured by moving vehicles. The key challenge is removing temporarily static objects (like parked cars and pedestrians) while maintaining temporal consistency over long trajectories. The main contributions are: 1) adapting 2DGS with hard-label semantic for scalable scene representation with explicit object removal, 2) using rendered alpha map to classify areas as fully observed, partially observed, or unobserved for creating inpainting masks, and 3) a time-reversal inpainting framework that maintains temporal consistency by using future frames to condition earlier frames. The paper shows better inpainting results compared to other off-the-shelf methods.", "questions": "I think the paper is well-written with a good application as it proposed an inpainting method that seems to work well to maintain temporal consistency.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- I particularly like the idea of time-traversal inpainting. While single-image inpainting has been extensively explored, video-based inpainting (where temporal consistency is required) remains relatively understudied. Although the overall process isn't completely novel, I appreciate that the method is specifically tailored for driving videos with predominantly forward motion. Reference-based inpainting shows great promise for many applications and seems like a valuable direction to pursue.\n\n- The paper is well-written, with clear motivation. The process, insights, and technical details are thorough and convincing. I especially appreciate the comprehensive related work section, which is very helpful for readers who don't necessarily work in inpainting or neural scene representation.\n\n- The results are impressive, with both quantitative and qualitative improvements over baseline inpainting methods. The method appears to handle challenging cases well, particularly in maintaining consistency across long sequences.", "weaknesses": "- My only concern is that while the ultimate goal is temporally-consistent inpainting, the authors chose a more complex approach going through neural scene representation (2D/3DGS) and using rendered alpha masks to identify different region types (observable/unobservable) for generating inpainting masks. While this is interesting in principle, I wonder if a simpler pipeline could achieve similar goals - for example, establishing object correspondences across the video frames and then applying LeftRefill. I also note that LeftRefill is missing from the quantitative comparisons.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ixCZRd9AEZ", "reviewer": "Reviewer_roan", "summary": "This work aims to remove static movable objects (e.g., parked vehicles) in street view data for 3D reconstruction (e.g., 3DGS). The authors propose two main improvement to achieve better inpainting results: better inpainting mask, better reference for image inpainting.\n\nExperiments conducted on Waymo and PandaSet datasets demonstrate better performance compared to image/video inpainting methods and other 3D inpainting approaches.", "questions": "please address the concern in the weakness session", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "1. Clear writing and presentation, with a well-designed supplementary website\n2. Two novel and effective design elements:\n   * An improved method for generating inpainting masks that incorporates observations from other frames\n   * Time-reversal inpainting that utilizes future frames as references", "weaknesses": "1. The contribution seems somewhat limited. While the two major contributions are effective, they appear to be incremental rather than groundbreaking. The work could serve as a baseline, but not much more that.\n\n2. The performance metrics are not entirely convincing. Table 1 doesn't demonstrate significant advantages over other baselines. The video in the supplementary materials appears noticeably inferior to other NVS methods on Waymo (e.g., EmerNeRF, StreetSurf, StreetGaussian).\n\n3. The benchmark methodology raises some questions (please correct me if I'm mistaken, as I haven't encountered this benchmark setting before). What are the ground truth images used for LPIPS? If all rendered images lack vehicles while the ground truth images typically include them, is this an appropriate measurement for LPIPS/FID?\n\n4. I have reservations about the task itself: Is it truly necessary to remove static components from the scene? Static elements (e.g., trees, traffic signs, rubbish bin) are integral parts of the environment. Even if removal is desired, wouldn't it be more efficient to accomplish this through multi-pass data collection?\n\n\n\n\n## updated response to the authors after rebuttal\nI want to clarify that my use of 'this issue will kill the paper' is a common academic shorthand expressing concern about a critical technical issue that needs to be addressed.  My colleagues and I often use this phrase in academic discussions to highlight fundamental issues that could seriously affect a paper's contribution. If you're unfamiliar with this expression and find it harsh, I sincerely apologize – that wasn't my intention.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xA75elSun1", "reviewer": "Reviewer_orfH", "summary": "The paper presents StreetUnveiler for reconstructing an empty street from in-car camera videos. It uses hard-label semantic 2DGS for representation, divides the scene into regions located by an alpha map, and introduces a time-reversal inpainting framework. Experiments show its effectiveness in reconstructing the empty street for autonomous driving applications.", "questions": "In fact, there are many methods that directly predict the masks of dynamic objects, and these are usually more reliable than the masks obtained by optical flow. Additionally, in many complex situations (such as vehicles driving sideways at an intersection), can the prior advantage of time-reversal inpainting still be maintained?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The use of 2D Gaussian combined with semantic information is reasonable. It allows for better identification and removal of objects by injecting semantic labels into 2D Gaussians.\n- Experiments on street scene datasets demonstrate the effectiveness of the method.\n- The proposed time-reversal framework inpaints frames in reverse order and uses later frames as references for earlier frames. This enhances the temporal consistency of the inpainting results, making the reconstructed street more realistic and consistent across different views.", "weaknesses": "- The 2DGS reconstruction and optimization processes might be computationally complex and costly, especially for large-scale street scenes, leading to long training and inference times. The author may consider other faster reconstruction baselines.\n- The method has limited ability to handle dynamic objects. Although some approaches for simple dynamic cases are proposed, more improvements are needed for complex dynamic scenes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gaqstAx7fL", "reviewer": "Reviewer_SLKs", "summary": "The paper presents StreetUnveiler, a method based on semantic 2DGS to achieve the empty street scene reconstruction. \nStreetUnveiler first utilizes the rendered alpha map to locate unobservable regions with the aid of distortion loss and shrinking loss.\nThen a time-reversal inpainting framework is introduced to optimize temporal consistency by leveraging later frames as references for earlier ones. \nExperiments are conducted on Waymo and Pandaset to demonstrate the effectiveness of StreetUnveiler for the empty street reconstruction.", "questions": "1. Applications of Unveiled Street Scene Reconstruction: The task of reconstructing an unveiled street scene is indeed challenging and engaging. However, real-world street environments include more than just vehicles, and vehicles can often obscure parts of other objects, such as the lower half of a road sign. Would it be more practical to first apply 2DGS to separate scenes based on specific requirements, then proceed with targeted inpainting? This approach could provide more utility by selectively reconstructing essential scene elements.\n2. Potential to Build on Existing 3DGS Street Scene Reconstruction Methods: Current methods, such as Street Gaussians[1] (using 3D box-based scene decomposition) and S3Gaussian[2] (self-supervised scene decomposition), aim to separate scenes into dynamic and static components. Could integrating these approaches enhance StreetUnveiler’s ability to more effectively distinguish between removable objects and essential scene elements? This could potentially improve the precision in identifying and preserving critical components.\n\n[1]Yan Y, Lin H, Zhou C, et al. Street gaussians for modeling dynamic urban scenes[J]. arXiv preprint arXiv:2401.01339, 2024.\n[2]Huang N, Wei X, Zheng W, et al. $\\textit {S}^ 3$ Gaussian: Self-Supervised Street Gaussians for Autonomous Driving[J]. arXiv preprint arXiv:2405.20323, 2024.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The writing of the paper is clear and easy to follow.\n2. The task of empty street scene reconstruction is interesting. The limitations discussed in the introduction are well studied in the following section. \n3. The proposed method is evaluated on two datasets. Many quantitative and qualitative experiments are conducted to demonstrate the effectiveness of this method.", "weaknesses": "1. Lack of Quantitative Metrics for Mesh: While the authors discuss the differences in problem formulation between StreetSurf and StreetUnveiler, a quantitative comparison of mesh reconstruction is still lacking. StreetSurf uses point clouds as ground truth to calculate Chamfer Distance as a metric. Could a point cloud derived from Structure from Motion (SfM) be used as a ground truth for evaluating StreetUnveiler’s performance? Even relative values across different baselines could provide meaningful insight.\n2. Limitations in Semantic Label Supervision: Current semantic supervision seems limited to common objects like vehicles, yet street environments also include pedestrians and less common obstacles. Could expanding the range of semantic (instance) labels improve the model’s robustness across diverse urban scenes?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)