InstaInpaint: Instant 3D-Scene Inpainting with Masked Large Reconstruction Model
Authors: Junqi You
Chieh Hubert Lin
Weijie Lyu
Zhengbo Zhang
Ming-Hsuan Yang
Venue: NeurIPS 2025 poster ()
Keywords: 3D Inpainting
Primary area: applications

Abstract: Recent advances in 3D scene reconstruction enable real-time viewing in virtual and augmented reality. To support interactive operations for better immersiveness, such as moving or editing objects, 3D scene inpainting methods are proposed to repair or complete the altered geometry. To support users in interacting (such as moving or editing objects) with the scene for the next level of immersiveness, 3D scene inpainting methods are developed to repair the altered geometry. However, current approaches rely on lengthy and computationally intensive optimization, making them impractical for real-time or online applications. We propose InstaInpaint, a reference-based feed-forward framework that produces 3D-scene inpainting from a 2D inpainting proposal within 0.4 seconds. We develop a self-supervised masked-finetuning strategy to enable training of our custom large reconstruction model (LRM) on the large-scale dataset. Through extensive experiments, we analyze and identify several key designs that improve generalization, textural consistency, and geometric correctness. InstaInpaint achieves a 1000$\times$ speed-up from prior methods while maintaining a state-of-the-art performance across two standard benchmarks. Moreover, we show that InstaInpaint generalizes well to flexible downstream applications such as object insertion and multi-region inpainting.

Instalnpaint: Instant 3D-Scene Inpainting with Masked Large Reconstruction Model
Junqi You 1.2
Chieh Hubert Lin'
Weijie Lyu'
Zhengbo Zhang
Ming-Hsuan Yang'
UC Merced
Shanghai Jiao Tong University
Singapore University of Technology and Design
Instance Masks
Posed Images
0.4s
Insta Inpaint
Reference Image
Novel View Synthesis
Object Insertion
Multi-Region Inpainting
Figure 1: Overview. Instalnpaint generates inpainted 3D scene from set of posed images, set multi-view foreground instance masks, and a 2D reference image. The 3D reconstruction and inpainting process takes only 0.4 seconds. InstaInpaint is generalizable framework supporting background inpainting, object insertion and multi-region inpainting simultaneously.
Abstract
Recent advances in 3D scene reconstruction enable real-time viewing in virtual and augmented reality. To support interactive operations for better immersiveness, such as moving or editing objects, 3D scene inpainting methods are proposed to repair complete the altered geometry. However, current approaches rely lengthy and computationally intensive optimization, making them impractical for real-time online applications. We propose Instalnpaint, reference-basee feed-forward framework that produces 3D-scene inpainting from 2D inpainting proposal within 0.4 seconds. We develop self-supervised masked-finetunin strategy enable training our custom large reconstruction model (LRM) on the large- scale dataset. Through extensive experiments, analyze and identify several key designs that improve generalization, textural consistency, and geometric correctness. InstaInpaint achieves 1000 speed-up up from prior methods while maintaining a state-of- oh-th--rrt performance across two standard benchmarks. Moreover, we show that InstaInpaint generalizes well to flexible downstream applications such object insertion and multi-region inpainting. More video results are available at our project page: http::/dhhmbb2.iihhub wtw.lhhw..onsssaaaaaan...peeeee
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(Bubblle size represent number ingut views)
MALD-NoRF
350
SPIN-NoRF
FID
120
GSeeeam
(Seream
InFiska
50
MALD-NORF
InFusion
2D Ti Mission Inpainting
Ours
Ours
0.4st
Reconstructio 3D Inpainting
Scene Reconstruction
3D Inpainling Time 1
Time
Figure 2: 3D Inpainting quality and speed. Left: Our proposed method reconstructs the inpainted scene at a much faster speed with more competitive quality compared with existing approaches. Right: Our proposed method takes only 0.4s for reconstruction and 3D inpainting.
Introduction
Recent advances neural reconstruction [1, 2, 3, 4, 5] achieve photorealistic and real-time rendering in virtual and augmented reality, enabling users navigate within the digital twin of real-world envi- ronments freely. However, pure viewing without meaningful interactions with the digital content lacks practical applications, thereby motivating the rising interest in manipulating and editing the recon- structed 3D scenes. As existing frameworks all rely on scenes reconstructed with optimization- methods before editing, it becomes intuitive design editing algorithms that are also optimization- based. Unfortunately, such a design choice leads to lengthy operation time, causing inherently unbearable wait time and infeasible resource requirements. Some recent approaches mitigate the execution time problem with reference- image and then achieves 3D edits by propagating the appearance to other views. Nevertheless, these methods still require impractical runtime, and regularizing the 3D geometry with 2D appearance forms an ill-posed problem in which hand-crafted heuristics often fall short and lead to artifacts.
algorithm design, which first generates one 2D reference
To address the execution speed problem, a natural solution to leverage the Large Reconstruction Models (LRM) [6, that can freeforwardly generate 3D geometry from sparse view images in less than a second. Learning from large-scale 3D scene data [8, 9]. LRMs can produce high-quality and high-fidelity reconstructions even on unseen data. However, LRMs require input images to present consistent 3D information and solve the geometry using crows-view correspondence. In Fig. 3, we show that the cross-vits view 3D consistency generated from state-of- offtthe-rrt diffusion model (i.e. MVInpainter [10]) is insufficient for LRMs to solve a plausible geometry, leading to noticeable blurriness. In addition, multi-view diffusion models introduce substantial overhead in computational time, which conflicts with our real-time 3D inpainting goal. These observations motivate us to develop a single-stage and end-to-end method that learns to constitute 3D geometry within the LRMss
We propose Instalnpaint, a new variant of LRMs that is tailored to achieve feed-forward reconstruction and edits simultaneously. Given a set of images paired with 3D-consistent 2D masks and one of the views being inpainted and served as the reference view, Instalnpaint predicts per-pixel Gaussian Splatting (GS) [7] parameters reconstruct the scene. For pixels visible across views, the model still solves the geometry similarly to other LRMs. Mcanwhile, the model learns to identify geometry from the surrounding context for pixels marked as reference pixels that have no geometric clues from other views. For instance, the extended geometry on the same plane should have a smooth depth transition, while inserted objects should have clear separation and stay in front of the background.
challenging to train such a model due to the lack of serge-scale datasets that simultaneously provide (a) multi-view images with camera poses, (b) before-and-after image pairs where objects are physically removed, and (c) accurate masks of the objects being removed. We thereby design a self-supervised masked-finetunin
scheme that utilizes large-scale dataset that complies with (a) while circumventing the need for (b) and (c). In this work, we show that obtaining meaningful training masks is the most critical design. By masking the edited regions with gray pixels, we can compel the model to ignore the pre-edit appearance and directly produce post-edit results. We artificially create three types of masks: cross-view consistent object masks using an off-the-shelf video segmentation model [1 I], cross-view consistent geometric masks with LRM self- reff-predicted depth, and randomly sampled image masks without cross-view consistency. Section 4.2, we show that these masking mechanisms are essential to avoid object bias that leads to generalization failures. For each training sample, we subsample few frames from scene as InstaTnpaint input views, while leaving the other frames as candidate supervision views. An input view is chosen as the reference view, while the editing regions in the remaining views are masked with gray pixels. InstaInpaint takes both masked input views, the reference view, and masks as input. Then, the network is end-to-end trained with
2

---

Reference Franc
Ingeinted Image Inpur Views
NYS
Figure Limitation of MVInpainter with LRM. MVInpainter creates notice- able multi-view inconsis- tency in high-frequency re- gions, leading to blurriness in the final LRM reconstruc- tion results.
MVinpainter
LRM
Blurisen
photometric losses using the unmasked output views, serving as the post-edit ground-truth As such, our design unlocks scalable training on large-scale real-world datasets.
We conduct extensive experiments on two standard 3D inpainting benchmarks with diverse and challenging real-world scenes. In Fig. 2, we highlight that Instalnpaint achieves state-of-the-art performance on both speed and quality axes. We also provide ablations of the key design choices on the masking strategies and the encoding design. The contributions of this work are:
We propose Instalnpaint, a reference-base based feed-forwart framework that infers 1000 faster than previous methods while achieving state-of-the-a quality across two 3D inpainting benchmarks. Our self-supervised musked-finetunnig strategy unlocks efficient training on large-scale data.
We demonstrate effective mask-creation strategies that avoid bias and improve generalization. Instalnpaint is unified and flexible framework that simultaneously supports object removal, object insertion and multi-region inpainting without additional training.
2 Related Work
Large Reconstruction Model. Transformer-baser 3D large reconstruction models (LRMs) [6] starts to emerge in recent years. With the scalable and generalization architecture and large 3d training dataset, LRMs are able to regress the 3D shapes from multi-view images a feed-forward manner, represented as triplane NeRFs [6]. 3D Gaussians [7, 12, 13, 14), latent tokens [15], or deformable Gassians [16]. Although LRM demonstrates high-quality sparse-view reconstruction ability, struggles with inconsistent or even incomplete input views, limiting its application in 3D editing tasks. In this work, we aim to impart LRM the capability of deducing the texture and geometry of a multi-view inconsistent region from a single reference view. MaskedLRM [17] is a concurrent work that employs a similar masked training strategy for the mesh inpainting task in an object-centric setup using large-scale synthetic data. In contrast, Instalnpaint targets more challenging real-world scenes with complicated textures and geometries. Furthermore, our proposed superuuee framework avoids the need for synthetic training data, avoiding the additional simulation- ro-rell domain gap.
3D Scene Inpainting. Existing 3D scene inpainting methods [18, 19, 20, 21, 22, 23, 24, 25, 26] mainly leverage the 2D image inpainting models to provide visual prior. Although some works [27, 28, 29, 10] try to enhance the 3D consistency of multi-view 2D inpainted results, there is still a notable textural shift in high- tighffrequency details. One line of work paints each individual 2D image and addresses the 3D inconsistency problem with tailored designs. For instance, SPIn-NeRF [30] proposes replacing pixel loss with perceptual loss for better high-frequency detail reconstruction. InpaintNeRF330 [31] also develops similar strategies with perceptual and depth losses. MALD- NeRF [32] presents masked adversarial training and per-scene diffusion model customization for better consistency. While the carefully designed mechanics effectively enhance inpainting quality, inconsistency brought about by multi-view inpainting is hard completely erase. Other works bypass multi-view 2D inpainting and adopt a reference-based inpainting strategy that imposes consisteney by adhering to the provided reference image. This paradigm is adopted by some NeRF inpainting work [33]. but is most common in 3DGS inpainting [34, 35, 36, 37, 38, 26]. For example, GScream [35] proposes a feature propagation mechanism to endow Gaussian blobs in inpainted areas with cross-view consistency. Infusion [39] starts with partial Gaussian scene and directly optimizes the depth-initialized Gaussian blobs on the reference view. However, inpainted regions often exhibit a clear boundary, and foreground floaters tend to appear. In this work, we propose reference-basee feedforward model that guarantees the consistency and smooth integration of inpainted regions with self relf-supervised masked training.
3

---

Masking Preprocessing
Scene Reconstructin
Ray
Unpatichify & Gaussian Upsampling
Loss
Transformer Blocks
Plocker Rey
Tokenizer
Reparr
Video Cip
N
Fromes
Inpaint Views X
Maska
Apply
Novel View Synthesis
Figure 4: Overall pipeline of Masked Finetuning. Given a video clip, a reference view and three inpaint views are selected. The reference view remains intact its RGB values and corresponding Plücker ray coordinates are directly tokenized. For the inpainting views. we first apply multi-view masks the images, then concatenate these masked images with their Plücker coordinates and the binary masks before tokenization. Tokens are sent into transformer blocks to predict pixel-aligned Gaussians. Supervision views are randomly sampled from the remaining frames of the video clips to compute photometric loss against novel view renders.
Methodology
InstaInpaint is trained in two stages. We first train a naive LRM to equip the model with sufficient 3D reconstruction knowledge, then apply masked fine-tuning to perform the inpainting task. In Section 3.1, we give an overview of the model architecture and stage one training pipeline. We elaborate the designs of the masked fine-tuning pipeline in Section 3.2 and the generation process of three types of masks in Section 3.3.
3.1 Overview
Large reconstruction models learn regress 3D geometry directly from multi-view 2D images. Inspired by GS-LRM [7], Instalnpaint uses ViT-based [40] self-attention transformer as its backbone. For given of multi-view input images [I, H
1 .....N, N). we first concatenate the images channel-wise, the Plücker ray coordinates [41] each view P,. We patchify and linearize them into H/p x W/p tokens, where is the patch size. The patchifier is implemented as p x convolution kernel with stride These feature tokens T; are then flattened and concatenated as input to the self-atte aff-attention transformer. Each transformer block consists of pre-Layemorm, multi-head self-attention, MLP and residual connections. The output tokens T from the transformer are then decoded into Gaussian parameters with single linear layer:
T; = Conv, (ConcatCC,,,,, P.)); T², Transformer (T)); G; L-inearTT]].
(1) represents
where Tú, T E "W///">D x and D the token dimension, G; R(HW//²)x('') 3D Gaussians and q is the number of Gaussian parameters. We then unpatchify G, to a per-pixel Gaussian map G'; E IR Hxxxg where each 2D pixel has corresponding 3D Gaussian. The final 3D scene is obtained by merging the multi-view per-pixel Gaussians, resulting in a total of N x H x W Gaussians. We compute photometric losses on M novel views. Following previous works [6, 7], we apply a combination of MSE loss and perceptual loss [42, 43].
3.2 Masked Finetuning Pipeline
In the second training stage, we employ masked fine-tuning strategy to adapt LRM from 3D reconstruction to 3D inpainting using existing large-scale real-world datasets [8, 9].
Selection of Input and Supervision Views. As is shown in Fig. 2, for given N frames video clip and corresponding multi-view masks (we will detail the generation process in Section 3.3), we use its quartile frames input frames (e.g., the 1st, 5th, 10th, 15th frame for a 15 frames clip). We randomly pick a frame the reference frame and keep the reference image complete. For the three non non-reference input frames, deemed inpaint views, we directly apply the multi-view masks onto the images by substituting the masked regions with gray pixels. We then randomly sample M supervision views from the remaining N - 4 frames in the video clip and apply photometric losses. The masking process compels the transformer to predict cross-view consistent geometry for inpainted regions of the reference view that produce plausible content when projected onto novel views.

---

Object
b) Rumdom Image Mask
Video
Scene Video
Object Masks
masks different views
Geometric Generation
Sample
Reference Frame
Oval Mask
Metric Depth
Depth Warping
Geometric Masks
Figure 5: Overview of mask generation methods. a) Object Mask Generation. We propagate scene videos through video segmentation model to obtain object masks. b) Random Image Mask. We randomly mask the same area for all input views. e) Geometric Mask Generation. We use a GS-LRM to predict metric depthmaps and then warp randomly sampled oval masks inpaint views.
Mask Encoding Strategy. We directly encode multi-view masks to provide the model with additional information to recover geometry of missing regions. For inpaint views, the masked image is then channel-wise concatenated with binary multi-view masks before being tokenized.
We do not provide the inpaint region mask directly to the reference view. This means that when the reference image fed into our pipeline, the binary mask image indicating the inpaint region is not given as an additional input alongside the reference RGB image itself, nor is the reference image's 's content explicitly altered (e-g. grayed out) in the masked region. This design inspired by key observation: the inpainted region and the reconstruction regions should maintain similar data distribution during training. To clarify the model's perception of different areas, we conceptualize three kinds of tokens within our pipeline:
1. Unmasked regions with intact texture: These areas are fully visible and consistent across both reference and supervision views, providing clear multi-view stereo information vital for accurate 3D reconstruction.
2. "Masked regions" in the reference view with original texture: This refers to the conceptual area within the reference view that corresponds the region proposed for inpainting. The model "sees" the original, unaltered texture in this region. However, since other non-reference views do have these corresponding regions explicitly masked out, the model is implicitly tasked with inferring the geometry and fine details of this area within the reference view based solely on its surrounding unmasked context. This unique setup is vital for maintaining a similar data distribution between token types (1) and (2), enabling the model to learn to faithfully reconstruct both areas equally well, whether the mask explicitly presented or not.
Masked regions in non-reference views filled with gray pixels: These regions in non-reference views are explicitly altered (grayed out) with provided mask and are provided with explicit binary mask. This clear alteration forces the model to learn to ignore these areas or infer their content from other available views.
We aim for the model to faithfully reconstruct (1) using multi-view stereo vision, to accurately speculate the geometry of (2) based on neighboring regions in the reference view, and mitigate negative effects from (3). This approach ensures sufficient distinction between (2) and (3) to prevent feature contamination while promoting consistent learning across visible and conceptually "masked" areas.
ref (ConetCon Pref)); T Tnouref Convp(Concat(T, @ M. P.P M,))
(2)
3.3 Mask Generation Process
Most current large-scale 3D datasets contain only videos captured in different scenes and camera annotations obtained from COLMAP [44]. In this section, we describe in detail how to obtain meaningful training masks from posed videos. As in Figure 5, we present three strategies for training masks that maximize data efficiency and model generalizability. At each iteration during training, we randomly pick one of these three types of masks based on assigned probability.
5

---

Object Mask. We obtain multi-view consistent foreground instance masks by devising a data collection pipeline with an off-the-shelf video segmentation model [11]. For a given video clip. we first perform global segmentation on the first frame to acquire proposal prompts. We filter out background segmentation and shuttered artifacts by removing segmentation masks that are either excessively large (covering >50% of the image area) or excessively small (<0.5% of the image area). We also filter out masks too close to image edges, as they are likely to disappear in subsequent frames.
The segmentation masks collected { {M² i = 1,..., n are then transformed into axis-aligned bounding boxes to serve as visual prompts. The video segmentation model processes the video clip and produces series of object masks {M; bounding- -ooxxing-boxxoo prompts. To prevent flickering artifacts that commonly uppear in video segmentation. we perform a sanity check by filtering out masks whose Insection-overt Union value between two subsequent frames (M n M " U M
I..... nz = .,.... Th) based on the generated
is too low. This is based on the heuristic that multi-view masks of two immediate frames should share a large common area. We cache the object masks for all frames in the clip. During training, we only use masks for selected input frames.
Although object masks provide high-quality multi consistent masking signals, object regions exhibit a distinct discontinuity from their neighboring regions, both in texture and in depth. However, during validation,
want the inpainted background to scamlessly integrate into the adjoining regions. This object bias leads to significant train-val gaps, causing the model to produce protruding depth in the inpainted area regardless of the actual geometry implied by the reference image. Therefore, propose the following two types of masks to solve the problem.
Geometric Mask. Geometric masks cover random multi-view consistent area. We collect geometric masks by warping masked pixels from reference views to inpaint views. Specifically, for the four selected unmasked frames I, along with their camera poses and intrinsics P,, K,, we first send them into the GS-LRM trained in the first stage to predict per-view Gaussian maps G'; E IR xxxxx Xg, We can extract the depthmap D, E ## of each input view. For the reference frame, we first randomly sample mask M E Hx with one or more ovals in the pixel space. (Refer to Section for more details on mask generation.) We project the masked pixels into 3D global coordinates, garnering a 3D point cloud:
X
x ref (K,d) Dro((t.))):s.11))) Mer((t,)) =1
(3)
where Poef is the camera extrinsic of reference view, Kref is the intrinsic for reference view, Dier j) and j) means the depth value and mask value at pixel [i,j] and h (xz, y, 2) + (x, z, 1) is the homogeneous mapping. The masking point cloud X is then projected onto the screen space of inpainting views, forming created by assigning a value of true to all pixels within these projected regions.
of pixel coordinates for each view. The inpainting masks are then
X, PKP X M.(P,9) = [p. g) E X,
(4)
iver
where P. and K; are the extrinsic and intrinsic for the i-th inpaint view, Flvver means the Iverson bracket. We further apply morphological closing to the projected mask to reduce the shattered points. Note that we can directly warp the project point cloud from the reference frame because the depthmap predicted by GS-LRM is metric depth strictly aligned with input camera parameters.
Random Image Mask. Aside from the two types of cross-view consistent masks, we further improve the robustness of our model with cross-view inconsistent masks. Random image masks are generated by sampling four identical pixel-space masks, which contain one or more rectangular regions. The model learns to detect and reduce the inconsistency of the projected Gaussians during training.
4 Experiments
Implementations. InstaInpaint is trained on DL3DV [9]. which is one of the largest open-source real- world 3D multi-view datasets, featuring a diverse variety of scene types and camera motion patterns. Its training set, DL3DV-10K, contains 10k+ videos from both indoor and outdoor scenes. Each scene video contains 200-300 key frames with camera pose annotation obtained from COLMAP [44].
We rigorously follow the architectural designs of the transformer and the Gaussian upsampler from GS-LRM [7]. We set the image patch size to 8 and the token dimension to be 1024. In the first stage, we train the model on a resolution of 256x 256 for 80K iterations. In the second stage, we finetune InstaInpaint on resolution of 512x We add an additional channel to the image patchifier and initialize it with the mean of 3 original channels. We split each scene videos into 15-frame video

---

GT & Mask
SPIn-NeRF
MALD-NoRF
Gseream
InFusion
Ours
Figure 6: Qualitative comparisons with state-of-the-art methods.. InstaInpaint obtains more plausible inpainted texture, better consistency and smoother transition at the inpainting border. The top 4 rows are from SPIn-NeRF and the bottom 4 rows are from LLFF
clips and randomly select one at each iteration. InstaInpaint takes 4 input views and is supervised on 8 views in both training stages. For all experiments, we train InstaInpaint for 12800 iterations in the second stage with 8 A6000 GPUs, which costs about 40 hours. We use Adam optimizer with 5 and total batch size of 80. For training efficiency, we cache advance all
learning rate of object masks and metric depthmaps predicted by LRM.
Evaluation Datasets. We evaluate InstaInpaint two real-world datasets, SPIn-NeRF [30] and LLFF [45]. SPIn-NeRF is an object removal benchmark with 10 scenes. Each scene consists of 60 training views with foreground objects and 40 testing views with foreground objects removed. LLFF consists of several real-world 3D scenes with varying view numbers. Following prior work, we use six-scene subset with SPIn-NeHe annotated 3D multi-view object masks. For InstaInpaint, We select the 4 input views from the training views set to ensure 1) the reference view is posed close to the center and 2) the three other context views cover an area as large as possible.
1. For reference view, we calculate the mean of all training views (60 for SPIn-NeRF) and select the view whose position is closest to the mean as the reference view. The reference view inpainted with SOTA di-fffuuiin inpainting models [46].
2. For the three other views, we aim to choose three views that cover as wide range as possible. We first regress plane from all training views. Then project all training view positions onto the
7

---

Table 1: Quantitative comparisons with state-of-the-art methods. Instalnpaint not only achieves competitive results compared with previous optimization-oen based methods, but also demonstrates a great speed improvement. The best and second-best results are highlighted, respectively.
SPIn-NeRF
LLFF
Category
Method
Timel.
LPIPS
M-LPPISS,
FID
KID4
C-FIDJ
C-KID)
SPIn-NeRF
Sh
0.4973
0.3742
129.71
0.0256
228.86
0.0732
None Reference-Rased
SPIn-NeRF+SD
5h
0.5274
0.4081
143.09
0.0295
235.36
0.0734
InpaintNeRF360
5h
0.4562
0.3417
117.43
0.0216
229.85
0.0712
MALD-NeRF
15h
0.4240
0.3269
109.66
0.0192
223.71
0.0710
GSeream
3h
0.4704
0.3622
97.803
0.0162
Reference-Basse Instalnpaint(
InFusion
40m
0.4943
0.3591
89.621
0.0148
203.31
0.0601
0.4s
0.4147
0.3130
84.535
0.0135
198.54
0.0613
Table 2: Quantitative comparisons with ARM-based methods. The best and second-best results are highlighted, respectively.
SPIn-NeRF
LLFF
Category
Method
LPIPS
M-LPIPS
FIDJ
KIDI
C-FID
C-KIDD
None Referenced-Based
SD+LRM
0,4802
0.3645
108.43
0.0189
215.32
0,0705
MVInpainter+LLNM
0.4122
0.3147
90.247
0.0163
217.90
0.0758
Referenced-Band -Based
Instalnpain((Ours)
0.4147
0,3130
84.535
0.0135
198.54
0,0613
Original Soane
MVInagintrr+ LRM
SD+LRM
InstaInpaint
Carera Pose
InFusion
Instalnpaint
Figure 7: Qualitative comparisons with LRM-based methods. Instalnpaint gives cleaner and sharper results with better cross-view consistency.
Figure 8: Robustness to reference view selection. The yellow star and blue triangle represent reference views and novel views.
plane. We then compute the convex hull of these projected 2D camera positions. The vertices of this convex hull represent outermost camera viewpoints. From the vertices of the convex hull, we identify the three points that form a triangle with the largest area. Specifically, this is done by applying the rotating calipers algorithm on the convex hull's endpoints.
Evaluatied Methods. We compare against state-of- ofethe--rrrrrrrrr the-art?2 3D real-world scene inpainting methods:
NeRF based: SPIn-NeRF [30], SSDn-NeRF+SD and MALD-NeRU [32], in which SPIn-NeRF+SD means substituting the inpainting model from LaMA [47] to a diffusion model [48]; 2) 3D-Gaussian based: GScream [35] and InFusion [39]. addition, we introduce two intuitive baselines using LRM:
1) SD+LRM first inpaints each input view with diffusion inpainter [46] and then uses GS-LRM for 3D reconstruction: MVInpainter+LRM applies multi-view-consisee
2D inpainting model [10]. We use a state-of-the-art 2D diffusion model [46] to generate the same reference image for fair comparisons with reference-basee methods, including GScream, InFusion and MV WWInpainter+LRNN
Metrics. Following prior works [32, 30], we use LPIPS [49], M-LPIPS, FID [50] and KID [51] on SPIn-NeRF dataset and C-FID, C-KID on LLFF dataset. M-LPIPS masks out pixels outside of inpainted masks. FID and KID measure the distributional similarity between two sets of images. Compared with LPIPS, FID/KID more suitable 3D inpainting tasks because inpainted areas can have valid content even if it is completely different from the ground-truth testing image. C-FID and C-KID measure visual quality in the inpainting border, since there is no ground -ttth value where the foreground object is physically removed for LLFF.

---

Table 3: Comparison with reference-based methods using ground-truth image reference.
InstaInpaint demonstrates better adherence to the reference image on novel views.
Method
PNSR1
M-PNSR
SSIM 1
M-SSIM
LPIPS|
M-LPIPS
21.182
22.010
0,5782
0.6471
0.1741
0.2332
Infusion
19.776
20.956
0.3550
0.04880
0.3738
0.2596
Instalnpaint(Ours)
22.799
23.881
0.6684
0.7331
0.1684
0.1
Scene & Mask
InPusion
Instalnpaint
Scene & Mask
InFusion
Instalnpaint
"A cute minion toy.
"A stone pillor with religious curvature
Figure 9: Qualitative comparisons of object insertion ability. InstaInpaint predicts plausible geometry for inserted objects and seamlessly integrates inpaint regions with reconstruction regions.
4.1 Main Results
Comparison with State-of-the-art Methods. As shown Tab. and Fig. 6, InstaInpaint performs favorably against state-of-the-art methods both quantitatively and qualitatively. Compared with optimization- based methods, InstaInpaint reconstructs the scene feedforward manner, gaining a 1000x speedup. Instalnpaint also provides a smoother transition at the inpainting border. Please refer to our project page for video results.
Comparison with LRM-based Methods. As shown in Tab. 2, InstaInpaint outperforms two LRM- based baselines in FID/KID-relan scores. As illustrated in Fig. Instalnpaint produces sharper and geometrically more coherent results than the two proposed LRMM-ased basclines.
Using Ground-truth Image Reference. FID/KID scores highly dependent 2D diffusion inpaint proposal. To better evaluate adherence to the provided reference image for reference- based methods, provide ground-truth images reference and evaluate pixel-level metrics. As shown in Tab. 5, Instalnpaint demonstrates a competitive edge over other reference-basee methods.
Robustness to Reference View Selection. The selection of the reference image is crucial factor for reference-based inpainting methods. As shown Fig. Infusion performs well, given the center of the scene reference, but produces noticeable artifacts with reference image close to the edges of the scene. Instalnpaint performs consistently well in both cases.
Object Insertion Ability. Instainpaint can be easily extended to text-driven object insertion tasks by using a text-driven diffusion inpainter to provide 2D reference. In Fig. 9, we show that the baseline method fails to predict the correct geometry and paste the inpainted texture onto the background like sticker, while InstaInpaint can accurately deduce the geometry of the complete inpainted object and seamlessly merge it into the original scene.
Multi-region Inpainting. We show in Fig. 10 that instainpaint can produce consistent inpainted scene even with multiple disjoint inpainting regions.
4.2 Ablation Studies
Mask Type Selection. In Tab. we ablate three types of multi-view masks we propose in Section 3.3. The introduction of geometric masks and random image masks mitigates object bias and effectively narrows train-val gaps, yielding better LPIPS, FID and KID scores. Although training with without object masks produces similar quantitative metrics, we observe that object masks significantly enhance geometric consistency for inserted instances, as shown Fig. 11. This can attributed to the object masks' strong cross-view consistency, constraining the model to maintain rigid object structures. Training without object masks results in more deformable geometry of foreground instances (the tilted traffic cone and the twisted minion).

---

Table 4: Ablation study mask types and mask encoding methods. We underline default
settings in the method section. The best and second-best results are highlighted, respectively.
SPIn-NeRF
LLFF
Method
LPIPS)
M-LPIPS4
FID
KID
C-FIDL
C-KIDD
w/o object
0.4132
0.3113
85.645
0.0137
198.68
0.0603
Mask Type Selection
w/o random
0.4185
0.3149
87.177
0.0144
200.23
0.0634
w/o geometric
0.4214
0.3170
88.565
0.0154
198.79
0.0632
w/ three types of mask
0.4147
0.3130
84.535
0.0135
198.54
0.0613
only reference view
0.4247
0.3177
87.710
0.0147
199.43
0.0618
Mask Encoding Method
views
0.4156
0.3146
86.442
0.0140
199.56
0.0587
inpaint views
0.4147
0.3130
84.535
0.0135
198.54
0.0613
Scene & Mask
NVI
NV2
NVI
NV2
NVI
NV2
w/o Object
Mask
w/
Object
Mask
Figure 10: Multiple region inpainting. Instalnpaint supports multiple disjoint inpainting regions.
Figure 11:
Ablation on object masks. Training without object masks produces deformable geometry across differ- novel views for inserted objects.
Mask Encoding Method. In Tab. 4, we compare three variants of masking encoding options: 1) providing inpaint masks only for the reference view, 2) providing masks for all four views, and 3) providing masks only for three inpaint views. We find that providing masks only for three inpaint views yield the best performance. This can be attributed to the uniform token distribution of reconstruction regions and the targeted inpainting regions, discussed in Section 3.2.
Conclusion
Limitation: Although InstaInpaint produce high-quality results for static scene inpainting, its performance degrades when handling dynamic scenes with fast-moving objects, the reconstructions become blurry as the model averages conflicting information. Furthermore, if dynamic object is occluded in the reference view but visible from other context views, the inferred content for the inpainted area may largely contradict the observations from those surrounding views, leading to huge visual inconsistencies. Addressing the challenge of 4D scene (space-time) reconstruction and inpainting for dynamic elements remains an active and complex research area. Future work will investigate integrating Instalnpaint with models specifically designed for dynamic 4D scene reconstruction or incorporating explicit temporal modeling capabilities to mitigate these issucs.
We present Instalnpaint, a reference-basee feed-forward framework that produces 3D-scene inpainting from 2D inpainting proposal within 0.4 seconds. By leveraging self-supervised mask-ffnetuning strategy. Instalnpaint effectively adapts Large Reconstruction Models for 3D inpainting. Instalnpaint achieves 1000x speed-up up from prior methods while maintaining state-of-the-art performance across two standard benchmarks and demonstrates strong flexibility for multiple editing applications.
6 Acknowledgement
Supported by the Intelligence Advanced Research Projects Activity (IARPA) via Department of Interior/ Interior Business Center (DOI/IBC) contract number 140D0423C0074. The U.S. Govern- ment is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon. Disclaimer: The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of IARPA, DOI/IBC, or the U.S. Govermment.
10

---

References
[1] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. Communications of the ACM, 65(1))99-1066, 2021.
[2] Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su, Tensorf: Tensorial radiance fields. In ECCV, 2022. 2
[3] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler. and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph., 139-1, 2023. 2
[4] Tao Lu. Mulin Yu, Linning Xu, Yuanbo Xiangli, Limin Wang, Dahua Lin, and Bo Dai. Scaffold-gs: Structured 3d gaussians for view-adaptive rendering. In CVPR, pages 20654-20664, 2024. 2
[5] Binbin Huang, Zehao Yu, Anpei Chen, Andreas Geiger, and Shenghua Gao. 2d gaussian splatting for geometrically accurate radiance fields. In ACM SIGGRAPH 2024 conference papers, pages 1-11, 2024. 2
[6] Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. LRM: Large reconstruction model for single image to 3d. In ICLR, 2024. 3,
[7] Kai Zhang, Sai Bi, Hao Tan, Yuanbo Xiangli, Nanxuan Zhao, Kalyan Sunkavalli, and Zexiang Xu. Gs-Irm: Large reconstruction model for 3d gaussian splatting. In European Conference on Computer Vision, pages 1-19, 2024. 2, 3, 6, 6
[8] Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. ACM Trany. Gruph. (Proc. SIGGRAPH), 37, 2018. 2. 4
[9] Lu Ling. Yichen Sheng, Zhi Tu, Wentian Zhao, Cheng Xin, Kun Wan, Lantao Yu, Qianyu Guo, Zixun Yu, Yawen Lu, et al. D13dv-10k: A large-scale scene dataset for deep learningbbased 3d vision. In CVPR, 2024.
2, 4, 6, 20
[10] Chenjie Cao, Chaohui Yu, Fan Wang, Xiangyang Xue, and Yanwei Fu. Mvinpainter: Learning multi-view consistent inpainting to bridge 2d and 3d editing. In NeurlPS, 2024. 2, 3, 8
[11] Nikhila Ravi, Valentin Gabeur, Yuan- Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, H Khedr, R Rädle, C &Rolland, L Gustafson, et al. SAM 2: Segment anything in images and vidcos. In ICLR, 2025.
2,
[12] Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng. and Ziwei Liu. Lgm: Large multi-view gaussian model for high-resolution 3d content creation. In ECCV, 2024. 3
[13] Yuedong Chen, Haofei Xu, Chuanxia Zheng, Bohan Zhuang. Mare Pollefeys, Andreas Geiger, Tat-Jen Cham, and Jianfei Cai. Mvsplat: Efficient 3d gaussian splatting from sparse multi-view images. In ECCV, 2024. 3
[14] David Charatan, Sizhe Lester Li, Andrea Tagliasacchi, and Vincent Sitzmann. pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction. In CVPR, pages 19457-19467, 2024.
[15] Haian Jin, Hanwen Jiang, Hao Tan, Kai Zhang, Sai Bi, Tianyuan Zhang, Fujun Luan, Noah Snavely, and Zexiang Xu. Lvsm: A large view synthesis model with minimal 3d inductive bias. In ICLR, 2025. 3
[16] Chieh Hubert Lin, Zhaoyang Lv, Songyin Wu, Zhen Xu, Thu Nguyen-Phuos Hung- Yu Tseng, Julian Straub, Numair Khan. Lei Xiao, Ming- -suun Yang, et al. Dgs-Irm: Real-time deformable 3d gaussian reconstruction from monocular videos. 2025. 3
[17] William Gao, Dilin Wang, Yuchen Fan, Aljaž Božič Tuur Stuyck, Zhengqin Li, Zhao Dong, Rakesh Ranjan, and Nikolaos Sarafianos. 3d mesh editing using masked Irms. arXiv preprint arXi::2412.0864 2024. 3
[18] Hao- Kang Liu, 1 Shen, Bing- Yu Chen, et al. Nerf-in: Free-form nerf inpainting with rgb-d priors. arXiv preprint arXiv:2206. 04901, 2022.3 3
[19] Silvan Weder, Guillermo Garcia-Hernande Aron Monszpart, Mare Pollefeys, Gabriel Brostow, Michael Firman, and Sara Vicente. Removing objects from neural radiance fields. In CVPR, pages 16528-16538, 2023. 3
[20] Youtan Yin. Zhoujie Fu, Fan Yang, and Guosheng Lin. Or-nerf: Object removing from 3d scenes guided by multiview segmentation with neural radiance fields. arXiv preprint arXiv:2305.10500 10503, 2023.
11

---

[21] Honghua Chen, Chen Change Loy, and Xingang Pan. Mvip-nerf: Multi-view 3d inpainting nerf scenes via diffuuson prior. In CVPR, 2024.
[22] Yiwen Chen, Zilong Chen, Chi Zhang, Feng Wang, Xiaofeng Yang, Yikai Wang, Zhongang Cai, Lei Yang. Huaping Liu, and Guosheng Lin. Gaussianeditor: Swift and controllable 3d editing with gaussian splatting. In CVPR. 2023. 3
[23] Ethan Weher, Aleksander Holynski, Varun Jampani, Saurabh Saxena, Noah Snavely, Abhishek Kar, and Angjoo Kanazawa, Nerfiller: Completing scenes via generative 3d inpainting. In CVPR, pages 20731-20741, 2024.
[24] Yubin Hu, Sheng Ye, Wang Zhao, Matthicu Lin, Yuze Hc, Yu-Hui Wen, Ying Hc, and Yong-Jin Liu. O 2-recon: completing 3d reconstruction of occluded objects in the scene with pre- rre-trained 2d diffusion
model. In AAAI, 2024. 3
[25] Dongting Hu, Huan Fu, Jiaxian Guo, Liuhua Peng, Tingjin Chu, Feng Liu, Tongliang Liu, and Mingming Gong. In-n-out: Lifting 2d diffusion prior for 3d object removal via tuning-free latents alignment. 2024. 3
[26] Zhihao Shi, Dong Huo, Yuhongze Zhou, Yan Min, Juwei Lu, and Xinxin Zuo. Infine: 3d inpainting via geometry-guides multi-view refinement. In CVPR. 2025. 3
[27] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. 7ero-1-to-3: Zero-shot one image to 3d object. In CVPR, pages 9298-9309, 2023. 3
[28] Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang. Mvdream: Multi-viow diffusion for 3d generation. arXiv preprint arXiv:2308. 165/2, 2023. 3
[29] Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su. 1erol23++: a single image to consistent multi-view diffusion base model. arXiv preprint aiv: 151 2023. 3
[30] Ashkan Mirzaei. Tristan Aumentado-Armsstonng. Konstantinos G Derpanis. Jonathan Kelly, Marcus A Brubaker, Igor Gilitschenski, and Alex Levinshtein. Spin-nerf:: Multiview segmentation and perceptual inpainting with neural radiance fields. In CVPR, pages 20666, 2023. 3, 8, 20
[31] Dongqing Wang. Tong Zhang, Alaa Abboud, and Sabine Süsstrunk. Inpaintnerf360: Text-guided 3d
inpainting on unbounded neural radiance fields. CVPR, 2023.
[32] Chich Hubert Lin, Changil Kim, Jia-Bin Huang, Qinbo Li, Chih-Yao Yao Ma, Johannes Kopf, Ming-Hsuan Yang, and Hung-Yu Yu Tseng. Taming latent diffusion model for neural radiance field inpainting. In ECCV, pages 149-165, 2024. 3, 8, 20
[33] Ashkan Mirzaci, Tristan Humentado-Armsternn Marcus A Brubaker, Jonathan Kelly, Alex Levinshtein, Konstantinos G Derpanis, and Igor Gilitschenski. Reference-guidde controllable inpainting of neural radiance fields. In ICCV, pages 17815-17825 2023.
[34] Ashkan Mirzaci, Riccardo De Lutio, Seung Wook Kim, David Acuna, Jonathan Kelly, Sanja Fidler, Igor Gilitschenski, and Zan Gojeie. Reffusion: Reference adapted diffusion models for 3d scene inpainting. arXiv preprint arXiv:2404. 1004. 2024. 3
[35] Yuxin Wang. Qianyi Wu, Guofeng Zhang, and Dan Xu. Learning 3d geometry and feature consistent gaussian splatting for object removal. In ECCV, pages 1-17, 2024. 3, 8, 20
[36] Mingqiao Ye, Martin Danelljan, Fisher Yu, and Lei Ke. Gaussian grouping: Segment and edit anything in 3d scenes. In ECCV, pages 162-179. 179, 2024. 3
[37] Zhihao Shi, Dong Huo, Yuhongze Zhou, Kejia Yin, Yan Min, Juwei Lu, and Xinxin Zuo. Imfine: 3d inpainting via geometry-guided nieti-view refinement. In CVPR, 2025. 3
[38] Chung-Ho Ho Wu, Yang-Jung Chen, Ying- Chen, Jie- King Lec, Bo-Hsu Kc, Chun- Wei Tuan Mu, Yi- Chuan Huang, Chin- Yang Lin, Min-Hung Chen, Yen- Yu Lin, and Yu-Lum Liu. Aurafusion360: Augmented unseen region alignment for reference-bart based 360 unbounded scene inpainting, In CVPR, 2025. 3
[39] Zhiheng Liu, Hao Ouyang. Qiuyu Wang, Ka Leong Cheng, Jie Xiao, Kai Zhu, Nan Xue. Yu Liu, Yujun Shen, and Yang Cao. Infusion: Inpainting 3d gaussians via learning depth completion from diffusion prior.
arXiv preprint 2404. 1613, 2024. 3, 8, 20
[40] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv 11929, 2020. 4
12

---

[41] Julius Plucker. Xvii. on new geometry of space. Philosophical Transactions of the Royal Society of London, (155):725-791, 1863. 4
[42] Qifeng Chen and Vladlen Koltun. Photographic image synthesis with cascaded refinement networks, In ICCV, pages 1511-1520, 2017. 4
[43] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. arXiv preprint arXiv:1409. 1556, 2014. 4
[44] Johannes I, Schonberger and Jan-Michacl Frahm. Structurefrrommmotiin revisited. In CVPR, pages 4104-4113, 2016. 5,
[45] Ben Mildenhall, Pratul P. Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng. and Abhishek Kar. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines, ACM Transactions on Graphics (TOG), 2019. 7, 20
[46] Black Forest Labs. Flux. htttp:://ggtuub..mmm//aeeeffeote
2024. 8
[47] Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizava, Arsenii Ashukha, Aleksei Silvestrov, Naejin Kong, Harshith Goka, Kiwoong Park, and Victor Lempitsky. Resolution-rohust large mask inpainting with fourier convolutions. arXiv preprint arXiv:2109.07661, 2021.
[48] Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models, In CVPR, pages 11461-11471, 2022.
[49] Richard Zhang, Phillip Isola, Alexci Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metrie. In CVPR, 2018. 8
[50] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. NewrIPS, 30, 2017.
[51] Mikotaj Binkowski, Danica Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans. arXiv preprint arXiv: 000,,1,0,1 2018. 8
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Abstract and introduction accurately reflect the paper's contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in Section 5.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximation: only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms
and how they scale with dataset size.
If applicable, authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer:
[NA]
Justification:
This paper does not include theoretical results
Guidelines:
14

---

The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems. The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented
by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
The experiments are reproducible and we will release our code.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: The code release requires additional review. We will put our best effort into
releasing the codes and the pretrained model.
15

---

Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co
bablic/guides// Stteeeeeeeooeeeeeiiiiiiiieeeeeee onPolicy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / //nips.co wwt.cc//pppppppeeeeeeeeeee www.//ggg//////eee// /ttp:////g.ieee/eeeuuuubiiaiiiia /CodeSubmiss eeiiinnooliiiooo for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We detail our experimental settings in Section 4
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification: We provide relavent details Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
16

---

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
We provide relavent details in Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethies
Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: / /ttp//eeeripp/e//ee//eee/e cc cccppblic/eeeeecccccccc del.ei/eeeecccuueeeeeeeeeeeeeeee
Question:
Answer: [Yes]
Justification:
We conducted the research that conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the societal impact of our work in the supplementary materials.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal
impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
17

---

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks. mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
We discuss the safeguards of our work in the supplementary materials.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All assets are properly credited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: New assets well documented.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodll as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
18

---

The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to LLM policy (https:/ /ttpaaa////erps...//e http://urippccc/ooffeeeeeeeeeeee wwt//ooffeeenee//2000000 for what should or should not be described.
19

---

Supplementary Material
A More Details
Camera Normalization and Selection. For SPIn-NeRF [30] and LLFF [45], we normalize all cameras in a scene into a [-1, 1J world space, We first calculate the mean of all camera locations and make all poses relative to the mean pose. We then scale all the poses based on the maximum camera deviation. During training, we perform the same normalization procedure on each DL3DV [9] clip (i.e. 15 frames). During validation, we use an intuitive way to select the 4 input views. We choose the camera closest to the mean position for the reference viewpoint. We choose another 3 cameras from the remaining camera positions so that they span a triangle with maximum coverage.
Oval Mask Generation Startegy. The random oval masks are generated on the fly during training. For each oval, we first randomly determine its center within the image boundaries. The major and minor axes of the ellipse are sampled with their sizes constrained to be within a certain proportion of the image dimensions to ensure masks are neither too small nor large. random angle between and 360 degrees is selected to introduce rotational variability. We randomly sample I to 4 ovals and combine them to formulate the final geometric mask.
Masking Finetuning, During training, we randomly seleets one of three mask types at each iteration: instance masks, geometric masks, random masks, with sampling probabilities of 25%, 25%, and 50% respectively. We provide ablation for the sampling rate Section B. After choosing the mask type, we randomly apply 1 to 4 masks to the inpaint views. For random masks, we randomly sample rectangular masks with edge length in range [size/6, 66, size/ 4]. For geometric masks, we define elliptical regions with axis lengths determined by the mask count. We sample the axis from range [size/8, size/0 /6] if the mask number is or 2, and [size/ 12, size/8) if the mask number is 3 or 4.
B More Ablation Studies
Multi-region Inpainting Results. We extend the SPIn-NeRF benchmark to support multi-region inpainting evaluation. Specifically, for each scene in the SPIn-NeRF benchmark, we randomly sample three oval masks on the reference view. We then warp the masks to other views the same way as the Geometric Masks in the paper (with metric depths predicted by LRM trained in the first stage). During Inference, we the original image from the reference view as reference. We measure pixel-level metrics (PNSR, SSIM, LPIPS) on SPIn-NeRF's testing novel views. In the default training setup, Instalnpaint processes input masks with 1 to 4 masking regions. We find that the number of masking regions combined during training influences the model's 's performance on multi-region inpainting tasks.
Table 5: Ablation Study on trainging- time mask numbers. We underline our default settings in the method section. The best result are highlighted
Method
PNSR1
M-PNSR
SSIM
M-SSIMt
LPIPS,
M-LPIPS
masking region
21.463
23.019
0.6493
0.7012
0.1729
0.1325
1-4 masking regions
21.901
24.298
0.6574
0.7253
0.1628
1.1
Mask sampling probability. We provide ablation study on different mask sampling strategies in Table 6. We can find that the introduction of geometric and random masks effectively improves inpainting performance. The optimal mask sampling distribution is allocating 50% probability to random masks and 25% each to geometric and object masks. This distribution best balance random masks' ability to prevent inconsistent inpainting, geometric masks' preservation of spatial relationships and object masks' high multiview consistency.
C More Qualitative Comparisons and Visualizations
Please check our project page for a more straightforward comparison of inpainting consistency. We provide in the video: comparison on SPIn-NeRF and L.LFF with previous 3D inpainting methods [32, 35, 39], comparison object insertion ability and more text-guided 3D inpainting results.
20

---

Table 6: Ablation study mask sampling probability. We underline default settings in the method section. The best and second-best results are highlighted, respectively.
SPIn-NeRF
LLFF
Object Mask
Geometric Mask
Random Mask
LPIPS
M-LPIPS]
FIDI
KIDI
C-FIDI
C-KIDJ
33%
33%
33%
0.4289
0.3532
86.432
0.0139
199.25
0.0628
50%
25%
25%
0.4326
0.3678
86.973
0.0142
199.89
0.0623
25%
50%
25%
0.4191
0.3484
85.385
0.0132
198.94
0.061
25%
25%
50%
0.4147
0.3130
84.535
0.0135
198.54
0.0613
D Boarder Impact
Our work bridges 3D reconstruction and inpainting by adapting Large Reconstruction Models for real-time 3D completion. This advancement enables two key opportunities:
First, when combined with 2D generative models, our system allows instant creation of 3D-consistent assets from text prompts particularly valuable for VR/AR applications where users can interactively modify 3D environments.
Second, the technical approach demonstrates how reconstruction-focused models can be repurposed for generative tasks, suggesting brand new research direction.
Admittedly, like any technology, method could potentially be misused generate manipulated 3D reconstructions or factually inaccurate renderings. We recognize the need for safeguards and mechenisms attribute Al-generated assets.
21


Reviews:
[{"review_id": "IrXwsBBOsA", "reviewer": "Reviewer_hGee", "summary": "This paper presents a single-stage feed-forward method for 3D reconstruction and editing, which supports both 3D inpainting and object replacement.\n\nThe authors introduce a self-supervised masked fine-tuning scheme to train the model. In the fine-tuning stage, masked, incomplete multi-view images are used as inputs, while original images from other views serve as supervision for generating the 3D scenes.\nTo generate the masks for the fine-tuning process, InstaInpaint employs three different types: Object Mask, Geometric Mask, and Random Image Mask.\n\nExperimental results demonstrate that InstaInpaint can perform view-consistent 3D editing in under 1 second while maintaining high-quality inpainting performance. The ablation studies further confirm the effectiveness of the proposed mask types used during the fine-tuning stage.", "questions": "1. Could this method be applied in autonomous driving scenarios? Adding objects into street scenes would be valuable for simulations in driving environments. Demonstrating successful applications in such scenarios would further highlight the impact and versatility of this method.\n\n2. Was the inpainting mask generation process inspired by the training stage of image inpainting models? If so, it would be helpful to reference related works in this area.\n\n\nMy concerns primarily arise from some unclear descriptions. I would be happy to raise my score if the authors could clarify the points I have mentioned.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths:\n1. The paper is well-written and easy to follow, with key technical details provided that help in understanding the proposed method.\n2. The proposed method achieves better 3D inpainting results than the baseline methods, with significantly faster inference speeds.\n3. The use of video segmentation methods and GS-LRM to generate data for 3D inpainting training could inspire future research.\n\n\n### Weakness:\n1. Typo: \"InstaInapint\" in Figure 1 (should be \"InstaInpaint\").\n2. Including NeRFiller [A] and O2-Recon [B] in the related works section on 3D scene inpainting would strengthen this part. These methods use image grids and mask reprojection techniques to enhance the multi-view consistency of inpainted results.\n3. The explanation in lines L142-L149 is unclear. Since the authors state that there are no masked regions in the reference image in L142, it’s hard to understand what (2) in L145 means.\n4. It is unclear how the method selects one or more ovals, as described in L182. A more detailed explanation would be helpful.\n5. The motivation behind using the \"Random Image Mask\" in the fine-tuning stage needs further clarification. Specifically, why does it improve the model’s robustness?\n\n\n### References:\nA. Weber, Ethan, et al. \"Nerfiller: Completing scenes via generative 3d inpainting.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\nB. Hu, Yubin, et al. \"O^ 2-recon: completing 3D reconstruction of occluded objects in the scene with a pre-trained 2D diffusion model.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 3. 2024.", "was_revised": true, "final_justification": "Most of my concerns have been well addressed in the detailed replies, and I appreciate the updated, clear explanations of the different tokens—these have greatly enhanced my understanding of the processing procedure. Considering all the reviews and responses, I've raised my score to accept.", "rebuttal": "We thank reviewer hGee for the valuable suggestions.\n\n**W1:** Typo in Figure 1.\n\nWe will fix the typo in the revised paper. \n\n**W2:** Missing Related Works.\n\nWe will cite and discuss the suggested references in the revised paper. \n\n**W3:** Clarification on L142-L145.\n\nWe revise and expand L142-L145 for better clarity.\n\nWe do not provide the inpaint region mask directly to the reference view. This means that when the reference image is fed into our pipeline, the binary mask image indicating the inpaint region is not given as an additional input alongside the reference RGB image itself, nor is the reference image's content explicitly altered (e.g., grayed out) in the masked region.\n\nThis design is inspired by a key observation: the inpainted region and the reconstruction regions should maintain a similar data distribution during training. To clarify the model's perception of different areas, we conceptualize three kinds of tokens within our pipeline:\n\n**(1) Unmasked regions with intact texture:** These areas are fully visible and consistent across both reference and supervision views, providing clear multi-view stereo information vital for accurate 3D reconstruction.\n\n**(2) \"Masked regions\" in the reference view with original texture:** This refers to the conceptual area within the reference view that corresponds to the region proposed for inpainting. The model \"sees\" the original, unaltered texture in this region. However, since other non-reference views do have these corresponding regions explicitly masked out, the model is implicitly tasked with inferring the geometry and fine details of this area within the reference view based solely on its surrounding unmasked context. **This unique setup is vital for maintaining a similar data distribution between token types (1) and (2), enabling the model to learn to faithfully reconstruct both areas equally well, whether the mask is explicitly presented or not.**\n\n**(3) Masked regions in non-reference views filled with gray pixels**: These regions in non-reference views are explicitly altered (grayed out) with a provided mask and are provided with an explicit binary mask. This clear alteration forces the model to learn to ignore these areas or infer their content from other available views.\n\nWe aim for the model to faithfully reconstruct (1) using multi-view stereo vision, to accurately speculate the geometry of (2) based on neighboring regions in the reference view, and to mitigate negative effects from (3). This approach ensures a sufficient distinction between (2) and (3) to prevent feature contamination while promoting consistent learning across visible and conceptually \"masked\" areas.\n\n**W4:** Generation details on random oval masks for geometric masks.\n\nThe random oval masks are generated on the fly during training. For each oval, we first randomly determine its center within the image boundaries. The major and minor axes of the ellipse are sampled with their sizes constrained to be within a certain proportion of the image dimensions to ensure masks are neither too small nor too large. A random angle between 0 and 360 degrees is selected to introduce rotational variability. We randomly sample 1 to 4 ovals and combine them to formulate the final geometric mask.\n\n**W5:** Clarification on the motivation of random image masks.\n\nWe will include more design motivations in the updated paper. Here are two main motivations:\n- **Test-time 3D inconsistency.** The testing benchmarks have their own annotation strategies different from our training strategy. The human annotation errors and mask dilation strategy often create substantial leftover masks bleeding through the actual object boundary, leading to 3D inconsistent masks in the background. Our random masks provide distinctive learning signals aside from the other two 3D consistent masking strategies.\n- **Data diversity.** Similar to we address the object bias of object masks using randomized geometric masks. The geometric mask still follows certain biased data distribution and potentially introduces unattended biases. Our random masks provide challenging 3D inconsistent masks for the model to resolve the conflicts through finding 3D correspondences across images. We believe this strategy increases the data diversity and augments important learning tasks for the model to learn a more robust and generalizable representation.\n\n**P1:** Potential application in autonomous driving. \n\nInstaInpaint could be used for editing static elements in street scenes. For example, one could add or remove fixed objects like traffic signs, streetlights, or parked vehicles. Our method, trained on static scene data, can seamlessly integrate or remove these stationary items while maintaining 3D consistency.\n\nHowever, applying InstaInpaint to dynamic road users (e.g., moving cars or pedestrians) would be challenging. InstaInpaint is trained for static 3D scene reconstruction and inpainting. Adapting our method for dynamic scene editing is an exciting area for future work.\n\n**P2:** Related work from image editing area.\n\nMasking training indeed is a common technique in image inpainting area, such as RePaint [1] and LaMa [2]. We will add representative works in the related work. \n\n\n[1] RePaint: Inpainting using Denoising Diffusion Probabilistic Models. Lugmayr, et al. CVPR 2022\n\n[2] Resolution-robust Large Mask Inpainting with Fourier Convolutions. Suvorov, et al. WACV 2022"}, {"review_id": "hBTuZkIRFt", "reviewer": "Reviewer_nJGu", "summary": "This paper presents a 3D inpainting method based on large reconstruction models (LRMs). It introduces a specifically designed LRM pipeline tailored for the multi-view inpainting task. The approach involves fine-tuning an LRM conditioned on one inpainted reference view and three additional masked views to generate Gaussians representing the full scene, including occluded (inpainted) regions. The training of such a model is non-trivial, as it requires multi-view masked image data. To address this, the authors leverage video segmentation priors and geometric priors to generate both object-level and scene-level masks, along with additional random masks. With this fine-tuning strategy, the method achieves effective and efficient 3D inpainting.", "questions": "Did the authors try any per-scene optimization, and how does it compare to direct prediction from the LRM? From my understanding, per-scene optimization should lead to improved results in practical scenarios, especially since supervision is available for the unmasked regions.\n\nHow would author consider to scale this approach to 360-degree scenes?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\n1.\tThe method effectively addresses the challenge of inconsistency in inpainting using large reconstruction models (LRMs).\n2.\tThe authors propose multiple variants of mask generation strategies to compensate for the lack of multi-view masked data, and these approaches are self-supervised.\n3.\tTraining an LRM for this task is non-trivial; the pipeline appears to involve significant engineering effort and rigorous sanity checks.\n4.\tThe results are promising in terms of both quality and efficiency, and the evaluation is thorough and complete.\n\n\nWeaknesses:\n\n1.\tAs the authors acknowledge, the method relies on a fixed number of input images, which limits its scalability to 360 scenes.\n2.\tThe paper does not clearly explain how the reference view is selected and inpainted, nor how the three target views are chosen during the inpainting pipeline described in the experiments.\n3.\tThe paper omits relevant citations and credit to prior works that use similar reference-based inpainting strategies, such as “In-N-Out: Lifting 2D Diffusion Prior for 3D Object Removal via Tuning-Free Latents Alignment” (NeurIPS 2024) and “Imfine: 3D Inpainting via Geometry-Guided Multi-View Refinement” (CVPR 2025).", "was_revised": true, "final_justification": "The rebuttal thoroughly addresses all of my earlier concerns. The authors provide a clear and detailed explanation of their input view selection strategy, commit to adding missing citations to relevant prior works, and outline feasible extensions for scaling beyond sparse-view settings. They also present new per-scene optimization results showing consistent quality gains with minimal overhead, reinforcing the method’s practicality. With these clarifications and additional evidence, I now find the contribution to be solid, well-motivated, and sufficiently validated, and I am raising my score from borderline accept to accept.", "rebuttal": "We thank reviewer nJGu for the valuable suggestions.\n\n**W1/P2:** Scalability to 360 scenes.\n\nWe acknowledge that InstaInpaint is primarily designed for sparse-view inpainting and reconstruction, and full 360-degree scene reconstruction is indeed beyond its scope. But we have several promising solutions for extending our work to the challenge.\n1. **Finetuning on dense-view feed-forward reconstruction model (e.g. Long-LRM[1]).** InstaInpaint's mask finetuning pipeline is very flexible and is suitable for feed-forward reconstruction models other than GS-LRM[2]. \n2. **A two-stage 360-degree scene inpainting method.** First, we can optimize a \"partial\" 3D-Gaussian scene with the inpainting area masked out. The inpainting area is left as a hole to be completed. Then we sample a sparse set of view images as inputs to InstaInpaint. Since Gaussian blobs predicted by InstaInpaint are strictly registered to the camera, we can store the predicted Gaussian blobs from the inpaint area at the original scene scale with a simple affine transformation and merge them into the optimized partial Gaussian Scene. To further enhance quality, especially at the merging boundaries, the combined Gaussian scene could undergo a few additional optimization steps.\n\n**W2:** Details on input views selection in the experiments.\n\nWe select the 4 input views to ensure **1) reference view is posed close to the center and 2) the three other context views cover an area as large as possible.**\n1. **For reference view,** we calculate the mean of all training views (60 for SPIn-NeRF) and select the view whose position is closest to the mean as the reference view. The reference view is inpainted with SOTA 2D-diffusion inpainting models[3].\n2. **For the three other views**, we aim to choose three views that cover as wide range as possible. We first regress a plane from all training views. Then we project all training view positions onto the plane. We then compute the convex hull of these projected 2D camera positions. The vertices of this convex hull represent the outermost camera viewpoints. From the vertices of the convex hull, we identify the three points that form a triangle with the largest area. Specifically, this is done by applying the rotating calipers algorithm on the convex hull's endpoints.\n\n**W3:** Missing citations.\n\nWe will cite and discuss the suggested references in the revised paper. \n\n**P1**: Per-scene Optimization Results\n\nYes, per-scene optimization does improve reconstruction quality compared with direct prediction from LRM. We experiment to test this.\n\n| Method | PNSR | MPNSR | SSIM | MSSIM | LPIPS | MLPIPS |\n|---|---|---|---|---|---|---|\n| InstaInpaint | 22.799 | 23.881 | 0.6684 | 0.7331  | 0.1684  | 0.1180 |\n| InstaInpaint+post-optimization | **23.252** | **24.452** | **0.6823** | **0.7429** | **0.1592** | **0.1073** |\n\nThis experiment is conducted following the setting mentioned in the paper from L241-L244. We optimize the LRM-predicted Gaussians on the 4 input views for 100 steps, which takes roughly 5 seconds. For reference view, we compute photometric loss on the entire image and all Gaussian parameters as learnable. For non-reference views, we compute photometric loss on the unmasked region and set Gaussians within the inpainted region as unlearnable. \n\n[1] Long-LRM: Long-sequence Large Reconstruction Model for Wide-coverage Gaussian Splats. Chen, et al. ICCV 2025.\n\n[2] GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting. Zhang, et al. ECCV 2024.\n\n[3] FLUX. Black Forest Lab."}, {"review_id": "LCAZojEkr3", "reviewer": "Reviewer_JFCW", "summary": "This paper proposes InstaInpaint, a novel 3D scene inpainting framework based on a feedforward large reconstruction model (LRM). The authors introduce a self-supervised masked fine-tuning strategy and demonstrate strong performance in both speed and quality on standard 3D inpainting benchmarks. The method achieves a remarkable 1000× speed-up over prior optimization-based approaches and supports multiple tasks including object removal, insertion, and multi-region inpainting.", "questions": "(1) The abstract emphasizes multi-region inpainting as a key feature, yet the main body only briefly touches on this in Figure 10. Please provide detailed technical explanations (e.g., mask handling, training design) and more extensive quantitative and qualitative evaluations to validate this claim. \n(2) Given that InstaInpaint is built on GS-LRM, the paper would benefit from clearer justification or discussion of any architectural innovations beyond standard LRM usage.\n(3) The discussion of limitations in Section 5 is very brief. Could the authors provide additional examples or figures that illustrate common failure modes of InstaInpaint (e.g., dynamic scenes, inconsistent masks), and how future work might address these?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n(1) The proposed framework is novel in applying a feedforward LRM to the inpainting task, achieving a substantial 1000× speed-up over traditional optimization-based methods.\n(2) The training methodology is effective, utilizing three distinct mask generation strategies (object masks, geometric masks, and random image masks) to construct diverse and scalable self-supervised training data.\n(3) The experimental section is comprehensive. The proposed approach achieves state-of-the-art performance on standard 3D inpainting benchmarks, both in terms of quality and speed.\nWeaknesses:\n(1) Although the abstract claims support for multi-region inpainting, the main text lacks detailed technical descriptions and corresponding experimental results that explicitly evaluate this capability.\n(2) The framework heavily relies on GS-LRM architecture; the paper introduces limited innovation in network design, and the application of LRM appears relatively straightforward without significant architectural modifications.\n(3) The claimed 1000× acceleration is largely attributed to the use of LRM, rather than innovations introduced by the authors in this paper.", "was_revised": true, "final_justification": "The rebuttal and discussions with the authors have addressed my concerns. Thus I would support on accepting it.", "rebuttal": "We thank reviewer JFCW for the valuable suggestions.\n\n**W1/P1:** More details and experiments on multi-region inpainting.\n\nWe mention InstaInpaint's multi-region inpainting to emphasize its flexibility and generalizability that various tasks can be supported without much specific design. However, we find that the number of training masks is relevant to model performance on multi-region inpainting. In our current training setup, the model processes input masks with 1 to 4 masking regions. We find that the number of masking regions combined during training influences the model's performance on multi-region inpainting tasks.\n\nWe add an ablation study on the number of masking regions during the training stage. We extend the SPIn-NeRF benchmark to support multi-region inpainting evaluation. Specifically, for each scene in the SPIn-NeRF benchmark, we randomly sample three oval masks on the reference view. We then warp the masks to other views the same way as the Geometric Masks in the paper (with metric depths predicted by LRM trained in the first stage). During Inference, we use the original image from the reference view as reference. We measure pixel-level metrics (PNSR, SSIM, LPIPS) on SPIn-NeRF's testing novel views. The results are presented in the table below:\n\n| Method | PNSR | MPNSR | SSIM | MSSIM | LPIPS | MLPIPS |\n|---|---|---|---|---|---|---|\n| 1 masking region | 21.463 | 23.019 | 0.6493 | 0.7012 | 0.1729 | 0.1325 |\n| 1-4 masking regions | **21.901** | **24.298** | **0.6574** | **0.7253**  | **0.1628**  | **0.1193** |\n\n**W2/P2:** Architectural innovations.\n\nWe acknowledge that InstaInpaint does not introduce a novel network architecture to GS-LRM [1] model. Our innovation centers on a flexible masking fine-tuning pipeline. The pipeline and its specific mask designs effectively adapt reconstruction models to the 3D inpainting task. **As a data-driven method, this pipeline's ability to leverage and guide existing powerful models is a significant and equally important contribution to the field.**\n\n**W3:** Acceleration stems from LRM but not our designs\n\nIt is true that the 1000x acceleration is indeed powered by the underlying Large Reconstruction Model (LRM). But LRMs are not natively equipped for 3D inpainting. The main innovation is the masking framework designed to overcome the challenges involved in adapting powerful data-driven reconstruction models to the specialized domain of 3D inpainting, including a lack of inpainting dataset, object bias, etc. **We are the first to adapt LRM to 3D inpainting field, unlocking their speed for this complex task, which is a problem not addressed by LRMs alone.**\n\n**P3:** More discussions on limitations.\n\nWhile we are unable to provide visual examples in this rebuttal, we can elaborate on how dynamic scenes affect InstaInpaint. In dynamic regions, the reconstructions become blurry as the model averages conflicting information. Furthermore, if a dynamic object is occluded in the reference view but visible from other context views, the inferred content for the inpainted area may largely contradict the observations from those surrounding views, leading to huge visual inconsistencies. Addressing the challenge of 4D scene (space-time) reconstruction and inpainting for dynamic elements remains an active and complex research area. Future work will investigate integrating InstaInpaint with models specifically designed for dynamic 4D scene reconstruction or incorporating explicit temporal modeling capabilities to mitigate these issues.\n\nWe would like to clarify that InstaInpaint is designed to handle mask inconsistencies. The inconsistent random image masks help build robustness to slight inconsistencies introduced by inconsistent maskings.\n\n[1] GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting. Zhang, et al. ECCV 2024."}, {"review_id": "6i98ywuKca", "reviewer": "Reviewer_6csA", "summary": "This paper propose InstaInpaint, a novel inpainting framework that restores missing regions of 3D scenes based on multi-view images. The ViT-based large-scale reconstruction model is trained in two stages, and the second stage enhances the inpainting ability through mask-based fine-tuning. Various mask generation strategies (object, geometric, and random masks) are introduced to achieve both data efficiency and generalization. Experiments show that InstaInpaint provides more consistent and natural 3D inpainting results than existing methods.", "questions": "The proposed method shows promising results in real-world sparse-view scenarios. However, how does InstaInpaint perform when the number of input views increases significantly? Can the method generalize or scale to dense-view reconstruction tasks?\n\nWhile the proposed training strategy is effective, the model architecture remains largely similar to MaskedLRM. Could the authors clarify whether any architectural innovations were introduced, and if so, how they specifically contribute to performance improvements?\n\nRecent work such as SimVS [1] addresses multi-view inconsistencies explicitly through simulation and robustness training. How does InstaInpaint differ conceptually or empirically from SimVS? Are there any comparative experiments to justify its advantages in handling inconsistent inputs?\n\nThe paper lacks direct comparisons with other multi-view 2D inpainting-based methods, such as InpaintNeRF360, SPIn-NeRF, or GScream. Can the authors provide additional quantitative results or ablation studies to better position their work among existing techniques?\n\nTraditional metrics like PSNR and SSIM are not reported or discussed. Could the authors clarify why these metrics were omitted, and whether the proposed method was evaluated under those metrics to ensure fair comparison with prior inpainting works?", "limitations": "Yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n1. To overcome the limitations of existing LRM-based models, this approach proposes a self-supervised masked fine-tuning strategy centered on reference images to enable effective 3D inpainting in real-world scenes.\n2. By combining object masks, geometric masks, and random masks and using them as training masks, it simultaneously improves the generalization performance and robustness of the model.\n3. By utilizing multi-view masks and photometric loss during the learning process, it induces the restored area to be naturally reconstructed from various viewpoints, thereby solving the unnatural boundary and floating object problems of existing methods.\n\nWeaknesses\n1. InstaInpaint is tailored for sparse-view settings and does not scale well to scenarios with a large number of input images, limiting its applicability in dense multi-view reconstruction tasks.\n\n2. InstaInpaint adopts a vision transformer-based architecture similar to MaskedLRM, but lacks fundamental architectural differences and no clear structural improvements.\n\n3. SimVS (CVPR 2025) simulates multi-view inconsistency to enhance the robustness of learning, while InstaInpaint does not directly address this kind of challenging setting and lacks comparative experiments or differentiation strategies with SimVS.\n\n4. Quantitative comparison with existing multi-view 2D inpainting-based NeRF/3DGS studies is limited, and the persuasiveness of the argument needs to be strengthened through head-to-head performance comparison with competing techniques.\n\n5. The reasons for not using or ignoring traditional performance metrics such as PSNR and SSIM, which are widely used in existing inpainting literature, are not clearly described. This makes it difficult to compare the quality of the proposed technique and may raise questions about its objectivity.\n\n[1]SimVS: Simulating World Inconsistencies for Robust View Synthesis. CVPR 2025", "was_revised": true, "final_justification": "The rebuttal has addressed all the concerns I raised. The clarifications regarding the sparse-view setting, architectural contributions, comparison with SimVS, quantitative evaluations, and the use of PSNR/SSIM are convincing and well-reasoned.", "rebuttal": "We thank reviewer 6csA for the valuable suggestions.\n\n**W1/P1:** Limitation to sparse view settings\n\nAs acknowledged in our limitation section, InstaInpaint is indeed limited to a sparse-view setting and the performance degrades when the input view number increases. \n\nHowever, the sparse-view setting also represents a feature for many real-world downstream applications where capturing dense multi-view data is impractical (e.g., mobile capture) or redundant (e.g., benchmarks such as SPIn-NeRF where the scenes are overly densely captured). \n\nRegarding scaling to a larger number of input views and dense-view reconstruction tasks, one potential method to scale InstaInpaint to a larger number of input views is by applying the masking finetuning pipeline on dense-view feed-forward reconstruction model such as Long-LRM [1]. \n\n**W2/P2:** Lack of architectural improvements\n\nWe acknowledge that InstaInpaint does not introduce novel network architecture to GS-LRM model. However, our innovation centers on a flexible masking fine-tuning pipeline and its specific mask designs that effectively adapt reconstruction models to the 3D inpainting task. As a data-driven method, we believe that this pipeline's ability to leverage and guide existing powerful models is a significant and equally important contribution to the field.\n\nFurthermore, while both InstaInpaint and MaskedLRM share architectural similarities with GS-LRM, we differ significantly in 3D representation and target application. MaskedLRM targets object-centric mesh editing and represents the 3D object in triplane-NeRF. InstaInpaint targets real-world scene editing, utilizing 3D Gaussian Splatting as its scene representation. The complexity of real-world scenes forces us to exploit more sophisticated masking strategies to adapt real-world multi-view datasets to 3D inpainting tasks.\n\n**W3/P3:** Comparison with SimVS.\n\nThank you for bringing SimVS [2] to our attention and for asking about the differentiation.\n\nThe difference between SimVS and InstaInpaint highlights the benefit and the design philosophy of InstaInpaint. SimVS handles inherently 3D inconsistent multi-view images. On the other hand, InstaInpaint aims to lift 2D-generated content to 3D using 3D consistent context from the non-inpainting area. Our design naturally avoids resolving the challenging 3D inconsistency arising from multi-view inpainting models. We also show the benefit of our approach by comparing with \"MVInpainter+LRM\" (in Table 3), where the moderate 3D inconsistency of this approach leads to a more blurry appearance and less favorable performance.\n\nUnfortunately, we are unable to compare to SimVS, as the model is not open-sourced. However, the \"MVInpainter+LRM\" baseline is highly relevant to the role of SimVS in the 3D inpainting problem: using a learned cross-view consistent diffusion model as reconstruction target, then performing feed-forward 3D reconstruction of the inpainted object. MVInpainter is our best effort in finding an open-sourced cross-view consistent generative model. However, MVInpainter produces very inconsistent results in high-frequency areas, leading to sub-optimal results compared to InstaInpaint, as is shown in Figure 3, Figure 7 and Table 2. This suggests a fundamental challenge in achieving perfect multi-view consistency solely through 2D generation for 3D reconstruction.\n\n**W4/P4:** Direct comparision with multi-view 2D inpainting-based methods.\n\nWe would like to clarify that **we have provided direct quantitative results with SPIn-NeRF, MALD-NeRF, GScream and InFusion in Table 1.** We add comparison with InpaintNeRF360 in the table below.\n\n| Method | LPIPS(SPIn-NeRF) | MLPIPS(SPIn-NeRF) | FID(SPIn-NeRF) | KID(SPIn-NeRF) | C-FID(LLFF) | C-KID(LLFF) |\n|---|---|---|---|---|---|---|\n| InpaintNeRF360 | 0.4562 | 0.3417 | 117.43 | 0.0216 | 229.85 | 0.0712 |\n| InstaInpaint | **0.4147** | **0.3130** | **84.535** | **0.0135** | **198.54** | **0.0613** |\n\n**W5/P5:** Experiments with PSNR and SSIM metric\n\nFirst, we would like to clarify that **we do report PSNR, SSIM and LPIPS for both InstaInpaint and other reference-based 3D inpainting works (Including Infusion and MVInpainter+LRM) in Table 3.** We further provide a comparison with GScream in the table below.\n\n| Method | PNSR | MPNSR | SSIM | MSSIM | LPIPS | MLPIPS |\n|---|---|---|---|---|---|---|\n| GScream | 20.425 | 21.323 | 0.5696 | 0.6234 | 0.2341 | 0.2146 |\n| InstaInpaint | **22.799** | **23.881** | **0.6684** | **0.7331**  | **0.1684**  | **0.1180** |\n\nBelow, we will explain why PNSR and SSIM can only be used to compare reference-based inpainting methods. PNSR and SSIM are low-level metrics that directly measure the resemblance between pixels. \n\n**For non-reference inpainting methods, the content within the inpainted area is non-deterministic.** This means there can be multiple perceptually plausible and equally \"correct\" ways to fill a missing region, even if their pixel values differ significantly from the specific ground truth. In such scenarios, a low PSNR or SSIM score does not necessarily indicate poor perceptual quality, as the generated result might be reasonable despite differing from the exact ground truth pixels. For these methods, higher-level perceptual metrics (such as LPIPS and FID, which we include in Table 1 of the manuscript) are often more appropriate. We note that existing non-reference inpainting methods, such as SPIn-NeRF and MALD-NeRF, do not report PNSR and SSIM.\n\nIn the context of reference-based inpainting methods, the \"inpainting\" task is constrained by a reference. By providing the ground truth texture as a reference to these models, we effectively make the target content in the inpainted region more deterministic. For our evaluations in Table 3, we use the ground-truth image as this explicit reference. By providing this ground-truth content, the target for the inpainted region becomes deterministic, enabling PSNR and SSIM to accurately measure how closely the method reconstructs the specific ground-truth pixels at novel views.\n\n[1] Long-LRM: Long-sequence Large Reconstruction Model for Wide-coverage Gaussian Splats. Chen, et al. ICCV 2025.\n\n[2] SimVS: Simulating World Inconsistencies for Robust View Synthesis. Trevithick, et al. CVPR 2025."}]

Author rebuttal:

Meta-review: This paper presents a feedforward approach that allows for inpainting and editing of a 3D scene, by leveraging a Large Reconstruction Model (LRM). Initially, the paper received one Reject and three Borderline Accept ratings. The reviewers appreciated the thorough evaluation, the strong results and the challenge in training an LRM for this task. However, there were some remaining concerns, e.g., related to perfomance of the approach when using a large number of input images, the minimal changes to the existing model architecture, missing related work and some unclear design choices. The authors submitted a rebuttal which helped increase the ratings of all reviewers. Eventually, the paper received one Borderline Accept and three Accept ratings. Given the unanimous acceptance recommendation from four knowledgable reviewers, there is no basis to overturn reviews. The AC recommends acceptance. Authors should still consider any additional comments or feedback from the reviewers while preparing their final version and of course update the manuscript to include any additional promised changes, analysis, results and/or discussion they provided in their rebuttal.
Decision: Accept (poster)