ReplaceAnything3D: Text-Guided Object Replacement in 3D Scenes with Compositional Scene Representations
Authors: Edward Bartrum
Thu Nguyen-Phuoc
Chris Xie
Zhengqin Li
Numair Khan
Armen Avetisyan
Douglas Lanman
Lei Xiao
Venue: NeurIPS 2024 poster (2024)
Keywords: 3D inpainting
Text-to-3D
Diffusion
Score-based Distillation
3D scenes
Primary area: diffusion_based_models
TL;DR: Replace objects in 3D scenes

Abstract: We introduce ReplaceAnything3D model RAM3D, a novel method for 3D object replacement in 3D scenes based on users' text description. Given multi-view images of a scene, a text prompt describing the object to replace, and another describing the new object, our Erase-and-Replace approach can effectively swap objects in 3D scenes with newly generated content while maintaining 3D consistency across multiple viewpoints. We demonstrate the versatility of RAM3D by applying it to various realistic 3D scene types, showcasing results of modified objects that blend in seamlessly with the scene without impacting its overall integrity.

ReplaceAnything3DD: Text-Guided Object Replacement in 3D Scenes with Compositional Scene
Representations
Edward Bartrum"
Thu Nguyen-Phuoc Meta Reality Labs London, England thunp0meta, com
Chris Xie Meta Reality Labs Redmond, Washington chrisdxie@metaa com
University College London
London, England edward. bartrum. adwaru.bartrum. 18cucl8ccc... ac. uk
Zhengqin Li Meta Reality Labs Redmond, Washington zh1 chl@meta.com com
Numair Khan
Armen Avetisyan
Meta Reality Labs Redmond, Washington Ammairkhan@mett
Meta Reality Labs London, England aavetisy swwn@meta..........o com
Douglas Lanman Meta Reality Labs Redmond, Washington douglas. www.an@meta.ccoommmmmmmmm com com
Lei Xiao
Meta Reality Labs Redmond, Washington lei xi xiao@meta.c com
Abstract
We introduce ReplaceAnything3D model (RAM3D). novel method for 3D object replacement in 3D scenes based on users' text description. Given multi-view images of a scene, text prompt describing the object to replace. and another describing the new object, our
approach can effectively swap objects 3D scenes with newly generated content while maintaining 3D consistency across multiple viewpoints. We demonstrate the versatility of RAM3D by applying various realistic 3D scene types, showcasing results of modified objects that blend in seamlessly with the scene without impacting its overall integrity.
1 Introduction
The explosion of new media platforms and display devices has sparked a surge in demand for high- quality 3D content, and thus increasing need for efficient tools for generating and editing them. While there has been significant progress in 3D reconstruction and generation, 3D scene editing remain less-studied area. In this work, focus the task of replacing or adding new 3D objects to an existing 3D scene using only input language prompts from a user. Compared to other 3D scene editing methods such as relighting or stylization, this task involves intricate local edits seamlessly integrate new objects into the scene without disrupting its overall coherence. This goes beyond just generating realistic visuals and demands nuanced understanding of both the global scene context and the interaction between the newly added object and the rest of the scene.
Naively using text-to-3D methods to generate 3D objects and manually adding them to scene can be tedious process. More importantly. completely ignores the interaction between objects'
Work done during an internship at Meta Reality Labs Research Project page: ittpp://reelaccanyyhiggg wit.ub.ithii
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Original
"Wichelanpeio statue dog browsing celiphore
dragon
octopus
Buddtha vtatue
hunvy on top pancalon
pile cookees
pot"
"Halloween pumaken"
Figure I: Our method enables prompt-driven object replacement for a variety of realistic 3D scenes.
appearance and the rest of the scene, such as lighting and shadows. Here, we instead formulate the task of object replacement as a 3D scene inpainting problem. Specifically, our goal is to seamlessly fill in the region occupied by the old object with a new object, making indistinguishable from the rest of the 3D scene. To avoid manual object placement and blending, we adopt powerful text-guided image inpainting model, enabling 3D object replacement based solely on input text prompts.
model (RAM3D), text-guided method for object replacement in 3D scenes using an Erase-and Frase-and-Reee and-Replace strategy. RAM3D takes multiview images of a static scene as input, along with text prompts specifying which object to erase and what should replace Our approach comprises four key steps: 1) We use text-to-mask model with text prompt to detect and segment the object to be erased from the input images. 2) We use a text-guided 3D inpainting technique fill in the background region obscured by the removed object in a multi-view consistent manner. 3) Next, we use a similar text-guided 3D inpainting technique to generate new 3D object corresponding to the input text description, which is seamlessly composited onto the background in all views. We thus obtain multiview consistent images of the edited 3D scene. 4) Finally, these updated dataset images are used to reconstruct the modified 3D scene, enabling novel view synthesis. Instead of relying generie image editing methods such Instruct-Pix2Pix [[], we adopt HiFA [2), a state-of-th thrttherrt distillation approach, to distill a pretrained model into a 3D scene representation. This allows to freely add or entirely remove detailed 3D objects from scenes, a significant challenge for methods like Instruct-Pix2Pix [ and its derivatives such as Instruct-NeRF2NeRR B due to their reliance instructions. By integrating a text- text-guided image inpainting model with a compositional scene structure that distinguishes the object of interest from the rest of the scene, ReplaceAnythingge can seamlessly generate edited 3D scenes with new objects harmoniously integrated into their environment. In summary, our contributions are:
In this work, we present the ReplaceAnything33
limited dataset of images and editing
We introduce a method for text-guided object replacement in 3D scenes that removes the need for manual 3D modelling.
We propose multi-stage approach that supports object replacement or even object addition in 3D scenes in high-fideling
We present 3D consistent results on multiple scene types (human avatar, forward-facing and 360 scenes), and challenging edit prompts requiring detailed texture synthesis.

---

2
Related work
Diffusion model for text-guided image editing Diffusion models trained on extensive text-image datasets have demonstrated remarkable results, showcasing their ability to capture intricate semantics from text prompts 4 1 6]. As a result, these models provide strong priors for various text-guided image editing tasks 7 I In particular, methods for text text-guided image inpainting [2] enable local image editing by replacing masked regions with new content that seamlessly blends with the rest of the image, allowing for object removal, replacement, and addition. These methods direct 2D counterparts our approach for 3D scenes, where each view can be treated as an image inpainting task. However, 3D scenes present additional challenges, such as the requirement for multi-view consistency and memory constraints due to the underlying 3D representations. In this work, RAM3D addresses these challenges by combining a pre-trained image inpainting model with compositional 3D scene representations.
Text-to-3D synthesis With the remarkable success of text-to-image diffusion models, text-to-3D synthesis has garnered increasing attention. Most work in this area focuses distilling pre-trained text-to-image models into 3D models, starting with the seminal works Dreamfusion [13] and Score
Jacobian Chaining (SJC) [14] Subsequent research has explored various methods enhance the quality of synthesized objects [15] 2 T6 17] and disentangle geometry and appearance (18). Instead of relying solely on pre-trained toxt-to--mage models, recent work has utilized large-scale 3D datasets such as Objaverse [19 to improve the quality of 3D synthesis from text or single images [20] [2].
Here, we move beyond text-to-3D synthesis by incorporating both text prompts and the surrounding scene information as inputs. This approach introduces additional complexities, such as ensuring the appearance of the 3D object harmoniously blends with the rest of the scene and accurately modeling object-object interactions like occlusion and shadows. Combining HiFA [2], a text-to-3D distillation approach, with text-o--mage-inpaininng model, RAM3D aims create more realistic and coherent 3D scenes that seamlessly integrate the synthesized 3D objects (Figure
3D Editing Many existing 3D editing methods focus editing an individual object's appearance geometry [22 23 24 25 26). For scene-level editing, recent works primarily address object removal tasks for weward-facing NeRF scenes 2 28 29] Fnstuuct-NeRF2NeRF (3] and similar followup
works [3 31 32 offer a comprehensive approach to both appearance editing and object addition, leveraging InstructPix2Pixx (1 to update the scene dataset. However, they modify the entire scene, they struggle to synthesise objects with complex geometrical texture patterns, and completely fail to remove objects from scenes. Blended-Nerf (26) and DreamEditor (33 allow localized object editing but do not support object removal. One closely related work is 28 which can remove and replace objects using one single image reference from the user. However, since this method relies only on a single inpainted image, it cannot handle regions with large occlusions across different views. and thus is only applied on forward-facing scenes. Another closely related work is RePaint-NeRF 34], which similarly allows text-guided scene editing on a masked region, but uses SDS loss to update a pretrained NeRF towards the text content. contrast, RAM3D adopts an Erase-and- and-Reelace approach for localized scene editing, instead of modifying the existing geometry or appearance of the scene's 's content, leading to superior qualitative results when compared RePaint-NeRF.
Unlike NeRF editing methods, the recently proposed Gaussian Editor 35 uses 3D Gaussian Splats 36
its underlying representation. Exploiting the explicit nature of this representation, this method enables localised edits by selectively updating targeted Gaussians, using guidance from InstructPix2Pix (.. Like our work, it also supports adding new objects to scenes. Similarly to (28), a single scene image is inpainted in 2D, providing a reference image of the new object. A pre-existing image-to-3D object model (37] then generates a 3D object from the segmented reference image, in isolation from the original scene. The coarse object mesh is transformed back into a 3D Gaussian representation and integrated into the scene through laborious modelling process where the object is is manually placed into the scene. More importantly, the new object is not guaranteed to integrate seamlessly into the surrounding scene, which results in visible artifacts and quality issues when the method is applied to 360 scenes. In contrast, RAM3D performs distillation using the inpainting network as diffusion prior, which leads to harmonious blending between the new object and its surroundings, even when applied to 360 scenes with challenging edit prompts.
3

---

Input
1. Erare Stage
2.Replace Stage
Composite
"Statue"
'Chocolate Chip cookies"
HiFA
HiFA Loss
Loss
Inpaintinn LDM
Inpainting LDM
Figure 2: An overview of RAM3D Erase and Replace stages.
3 Method
3.1 Preliminary
Distilling diffusion models Dreamfusion Sampling (SDS) to compute gradients from a 2D pre-trained text-to-image diffusion model, to optimize the parameters 3D neural radiance fields (NeRF). Recently, HiFA (2) significantly improves the quality of text-to-3D object generation by introducing an alternative loss formulation to SDS that can be computed explicitly for a Latent Diffusion Model (LDM). Let Owene be the parameters of implicit 3D scene, y text prompt, E be the pre-trained LDM model with encoder E and decoder D, Oscene can be optimized using:
[3] proposes a technique called Score Distillation
LIFFA (0, x) (t, [[|z 2||² + ARCB |x x|]
(1)
where Exx is the latent vector by encoding rendered image x of Oseme from camera viewpoint from the training dataset, & is the estimate latent vector by the denoiser eg, and & = D(2) is a recovered image obtained through the decoder D of the LDM. Note that for brevity, we incorporate coefficients related to timesteps to wtt).
Here we deviate from the text-to-3D synthesis task where the generated object is solely conditioned on text prompt. Instead, we consider a collection scene views as additional inputs for the synthesized object. To achieve this, we utilize the HiFA distillation loss function and timestep- annealing strategy, in conjunction with an open-source text-to-image impainting LDM. This LDM (z,, t, y, m) requires not only text prompt y, but also binary mask indicating the area to be filled in.
Overview
The input to RAM3D consists of collection of n images I, corresponding camera viewpoints and a text prompt Merase describing the object the user wishes to replace. Using this text prompt we can obtain masks m: corresponding to every image and camera viewpoint using pretrained text-to-mask model LangSAM [38]. Note that these masks are not necessarily multi-view consistent. We additionally provide text prompt Vreplace describing a new object replace the old object. Our goal is to modify the masked object in every image in the dataset to match the text prompt Vreplace in 3D 3D-consistent manner. We can then reconstruct the edited scene using any choice of 3D representations such as NeRF or Gaussian Splats (36] to obtain renderings of the edited 3D scene from novel viewpoints.
Figure illustrates the overall pipeline of our Erase and Replace framework. Instead of modifying existing objects' geometry and appearance to match the target text descriptions like other methods
approach. Firstly, for Erase stage, we remove masked objects completely and inpaint the occluded region in the background, using a neural field
33 we adopt an Erase-and-RRplaa
Org- Secondly, for the Replace stage, we generate new objects using a neural field Org. compositing
them so that they blend in with the inpainted background scene. Finally, we create new training set

---

using the edited images and camera poses from the original scene, and reconstruct the modified 3D scene using any choice of 3D representations for novel view synthesis.
To enable text-guided object replacement in 3D scenes, we distill an open-source text-to-image inpainting LDM using the HiFA loss function (Equation[]) ( and timestep annealing strategy 2 Note that LDM distillation using 3D Gaussian Splats is still challenging to optimise, leading to blurry results and thus requiring further refinement process in recent 5xt-to-3D-obbec works 40 We therefore opt to use a NeRF-based representation instead, for RAM3D's 's Erase and Replace stages (Sections 33 3.4). To eircumvent the memory constraints and slow training speed inherent to NeRF's implicit representations, we propose a Bubble-Ner Nefle- representation (see Figure 3 Left side) which only models the localised part of the scene that is affected by the editing operation, instead of the whole scene.
3.3 Erase stage
In the Erase stage, we aim to remove the object described by Jemss from the scene and inpaint the occluded background region in multi-view consistent manner. To do so, we optimise RAM3D parameters Pgg which implicitly represent the inpainted background scene. Note that the Erase stage only needs to be performed once to remove the desired object, after which the Replace stage (Section 3.4 can be used to generate different objects or even add new objects to the scene, as demonstrated the Results section. As a pre-processing step, we use LangSAM (38] with text prompt Veruse to obtain a mask m, for each image in the dataset. We then dilate each m; to obtain halo regions h; around the original input mask (see Figure 3 Left side).
At each training step, we sample image I;, camera Vi. mask m,, and halo region h, for a random i {1..n). providing them as inputs to RAM3D compute training losses (Figure Left side) (we henceforth drop the subscript for clarity). RAM3D volume renders the implicit 3D representation Bog over rays emitted from camera viewpoint v which pass through the visible pixels in m and h (the Bubble-NeRF region). The RGB values of the remaining pixels on the exterior of the Bubble-NeRF are sampled from 1 (see Figure 3 Left side). These rendered and sampled pixel rgb-values are arranged into 2D array, and form RAM3D's inpainting result for the given view, xBR, Following the HiFA distillation objective (see Section 3.1). we use the frozen LDM's E encode xb obtain z z", add noise, denoise with Ee to obtain ~मर and decode with D to obtain &bg, We condition e with I, m and the empty prompt, since we not aim to inpaint new content at this stage.
We now use these inputs to compute LITFA (see Equation T). We next compute Lreson and Layg on h (see Figure 3 guiding the distilled Oog towards accurate reconstruction of the background.
Lio00 MSE(x bg @ h)
Lugs MSE (SSE(ug91666 bx h), Vggia O h))
(3)
This step is critical to ensuring that RAM3D inpaints the background correctly (see Figure 7 Following [42). we compute depth regularisation Ldepth, leveraging the geometric prior from pretrained depth estimator 43). In summary, the total loss for the Erase stage is:
Crase
LHIFA + Areson Crccon + AuggCveg + Adepth
(4)
3.4 Replace stage
In the Replace stage, we aim to add a new object described by Preplace into the inpainted scene. To do so, we optimise the foreground neural field fig to render xfx, which is then composited with hh to form x. Unlike Ong in the Erase stage, Og does not seek to reconstruct the background scene, but instead only the L.M-inppinnee object which is located on the interior of m. Therefore in the Replace stage, RAM3D does not consider the halo rays which intersect h, but only those intersecting m (Figure 3, Right side). These rendered pixels are arranged in the masked region into a 2D array to give the foreground image ffs whilst the unmasked pixels are assigned an RGB value of O. The accumulated densities are similarly arranged into a foreground alpha map A, whilst the unmasked pixels are assigned an alpha value of O. We now composite the foreground x with the background x bg using alpha blending:
x AO fg + (1
A) x bg
(5)
5

---

Inpainting Mask m,
Sample RGB
Render Pixels
Composite
Halo Region h
Figure 3: Left: Erase stage. The masked region (blue) serves as a conditioning signal for the LDM, indicating the area to be inpainted. The surrounding nearby pixels form the halo region (green), which is also rendered by RAM3D during the Erase stage. The union of these 2 regions is the Bubble-NeRF region, whilst the remaining pixels are sampled from the input image (red). Right: Replace stage. RAM3D volumetrically renders the masked pixels (shown in blue) to give xf8, The result composited with bg to form the combined image x.
Using the composited result x, we compute LIRTA as hefore, but now condition 6p with the prompt Preplace: which specifies the object for inpainting. As longer require the other losses, set Arecon Avga, Adepth to O.
Since the Erase stage already provides us with good background, in this stage, Ogg only needs to represent the foreground object. To encourage foregroundd/backroundd disentanglement, on every k-th training step, substitute bg with constant-value RGB tensor, with randomly sampled RGB intensity. This guides the distillation of fgg to only include density for the new object; critical augmentation to avoid spurious floaters over the background (see Figure 7 Left side).
3.5 Reconstructing the edited scene
Once the inpainted background and objects have been generated inside the Bubble-NeRF region (Figure during the Erase and Replace stages, we composite the Bubble-NeRF renderings onto all original scene images. We finally obtain full 3D representation of the edited scene by applying off-the-shelf scene reconstruction method such as NeRF or Gaussian splats [36].
rubber
pat"
strawberry"
mushrooms"
Blended NeRF
Figure 4: Left: Qualitative comparison with Reference-Guide Inpainting 28 the original paper) for object replacement. Right: Qualitative comparison with Blended-NeRF [12 for object replacement. Our method generates results with higher quality and capture more realistic lighting and details.
(images adapted from

---

We show results for 3 challenging edit prompts on the GARDEN scene (top rows) and FACE scene (bottom rows). the GARDEN scene. our method generates more realistic objects which are better integrated with the surrounding scene. In the FACE scene, our method generates more detailed texture patterns and geometry which better aligned with the edit prompts.
Figure 5: Qualitative comparison with Gaussian Editor 35
4 Results
We conduct experiments on real 3D scenes varying in complexity: forward-facing scenes, 360 scenes and human avatar. For forward-facing scenes, show results for the STATUE and RED-NET scene from SPIn-NeRF dataset (27), as well as the FERN scene from NeRF [3)] For 360 scene, we the FACE dataset from Instruct-NeRF2NeRR [3] On each dataset, we train RAM3D with a variety of Yreplace generating a diverse set of edited 3D scenes (Figure D.. Please refer the supplemental video for more qualitative results for object replacement using personalized content.
show results from the GARDEN scene from Mip-NeRF 360 [4]) For the avatar result,
Note that RAM3D performs localised scene editing for 3D object replacement, conditioned by edit region mask and replacement object description prompt, as shown in Figure 2 For fair apples- to-apples comparison,
mostly compare with other state-of-the-art localised editing methods (see Figure Figure 5 Table []. This scope contrasts with the separate but related track of global scene editing methods, which modify entire scene based on instruction-style prompts, but do not support object replacement or removal
S 30 31 32). We nevertheless provide qualitative and
quantitative comparison with global-editing methods the Appendix (Sections F G
4.1 Qualitative Comparisons
Figure shows qualitative comparison with two methods for NeRF-based 3D object replacement; BleRded-NeRRF [2 and the method by (28). In Figure 5 we compare method with state-of-the. art 3D Gaussian Splat-based scene editing framework 335, which supports object removal, addition and replacement. In the Appendix (Section F. additionally compare with DreamEditor [33], Repaint-NeRF 34, generic scene editing method Instruct-NeRE2NeRR
3 and various similar
InstructPix2Pix-baase based followup works 30
31 32.
32.
Compared method by 28 on the left side Figure RAM3D achieves comparable object replacement results while handling more complex lighting effects such as shadows between the foreground and background objects. Note that the method by 28 only works with forward facing scenes, and thus cannot handle 360 scenes such as the GARDEN scene like our method. In Figure

---

Original
strawberry"
"Small
White bunny"
"A strawberry" and "A white bunny"
Figure 6: Given user-defined masks, ReplaceAnything3D can add completely new objects that blend in with the rest of the scene. Furthermore, due to its compositional structure, RAM3D can add multiple objects to 3D scenes while maintaining realistic appearance, lighting, and multi-view consistency (bottom right).
righht side, we note that RAM3D generates more realistic and detailed objects that blend in better with the rest of the scene. Meanwhile, Blended-NeRR only focuses on synthesizing completely new objects without taking the surrounding scenes into consideration. The synthesized object therefore looks saturated and outlandish compared to the rest of the scene. Moreover, due to the memory constraint of CLIP [45] and NeRF, Blended-NeRF only works with image resolutions 2-times smaller than ours (1008 x 756 vs. x04x378).
Figure 5 shows qualitative comparison with GaussianEditor 35 state-of-the-art scene-editing framework which supports object deletion, addition, and general editing capabilities. Applying it to the GARDEN scene, we first use the delete functionality remove the vase from the table, followed by the addition functionality to insert the new object. Note that this method generates the new object in isolation, using prior image-to-3D method (37], resulting in visible artifacts on the surface the table where the new object interacts with the surrounding scene. Furthermore, since this method is not guaranteed to place the new object correctly, its position the scene requires post-hoe manual adjustment- see Appendix (Section H for further details.
Applying GaussianEditor the FACE scene, use the general editing functionality. The explicit Gaussian Splat formulation allows edits to be localised to the relevant Gaussians (corresponding to the man's 's torso), which is selected using user interface, The editing process is then guided towards the desired prompt using 2D guidance from InstructPix2Pix [ and we observe similar limitations to other InstructPix2Pix based scene-editing methods B 32
31 30 (see Appendix (F). In particular, we observe that GaussianEdito struggles to synthesise detailed texture patterns on the man's 's torso, unlike RAM3D. Furthermore, we notice that the geometry of the man's clothes appears unchanged in the GaussianEdditor results, whereas our method successfully generates geometrical details such as the shirt collar and jacket lapels, matching the edit prompt in each case.
Adding multiple objects In addition to replacing objects in the scene, method can add new objects based on users' imput masks. rigurreddemonstraats that completely new objects with realistic lighting and shadows can be generated and composited to the current 3D scene. Notably, as shown in Figure boottom right, our method can add more than one object to the same scene while maintaining realistic scene appearance and multi- consistency.
4.2 Quantitative Results
3D scene editing is highly subjective task. Thus, we mainly show qualitative results and comparisons, and refer readers to the supplemental video for additional results. However, we follow 3 and report CLIP Imat- Direction Similarity, which measures the alignment of the performed object replacement with the input text description. Additionally, also quantitatively measure temporal

---

Table 1: We compute CLIP-based alignment metric, and optical flow-based temporal consisteney metric for various datasets and prompts. RAM3D shows the best overall edit prompt alignment and temporal consistency. (Top) GARDEN, (Middle) FACE, (Bottom) FERN.
Prompts
CLIP Text-Image Direction Similarity t
Warping error (x10-3 1110-)3
Ours
GaussianEditor
Ours
GaussianEditor
Pineapple 0.2246
-0.0631
1.2600
1.4600
Chess
0.0874
0.1857
1.2400
1.6700
Mushroom
0.1289
0.1030
1.2000
1.3900
Popcorn
0.2024
0.0400
1.3000
1.6800
Checker
0.0446
0.0016
0.4300
0.7600
Hawaiian
0.2169
0.1689
0.4900
0.8100
Tartan
0.1015
0.0379
0.4700
0.6700
Ours
BlendedNerf
Ours
BlendedNerf
Mushroom
0.0928
0.0535
2.3900
2.9500
Strawberry
0.3165
0.2224
2.3800
3.2300
(n)
(c)
(d)
(e)
Figure 7: Left: Results for 2 RAM3D variants trained on the Statue scene for the Erase stage. a) Training without any supervision the halo region surrounding the inpainting mask. The training objective is ambiguous and the Bubble-NeRF model collapses to a hazy cloud. b) Adding halo losses (Creson and Lugg) for the halo region surrounding the Bubble-NeRF guides the distillation of Ogg towards the true background, as observed on rays which pass nearby to the occluding object. RAM3D can now inpaint the background scene accurately.
Right: Results for 3 RAM3D variants, on the statue scene for prompt "A corgi". RGB samples are shown with accumulated NeRF density (alpha map) the top-left corner. The bubble rendering region is shown as dotted blue line. c) A monolithic scene representation which contains both the foreground and background. d) A compositional scene model but without random background augmentation. e) Our full model.
Specifically, (1) we use RAFT [49] to calculate the optical flow test videos, with each frame being rendered novel view of the original scene, (2) warp the corresponding frames from the modified scene according to it, and (3) measure the warping error. We compare RAM3D quantitatively with state-of-the- NeRF-based and Saussian-Splaating-basee methods ((35), (50)) for the object-replacement task three datasets.
consistency by calculating the Warping Error, following 46 47 48
In Table I we show that RAM3D achieves better overall prompt alignment (highest CLIP Text- Image Direction Similarity) than existing works, and the best temporal consistency (lowest warping error) across all methods and datasets. Interestingly, although Blended-NeRR -eRF directly optimizes for CLIP-similarin between the generated objects and target text prompts, it still achieves lower score than our method. Furthermore, we observe that our model's 's superior appearance synthesis quality (when compared to GaussianEditor in Figure is also reflected in the quantitative results in Table

---

4.3
Ablation studies
We conduct a series of ablation studies to demonstrate the effectiveness of our method and training strategy. In Figure Right side, we show the benefits of our compositional foreground/back structure and background augmentation training strategy. Specifically,, we train a version of RAM3D using a monolithic NeRF to model both the background and the new object (combining Obg and 0gy). In other words, this model trained to edit the scene one single stage, instead of separate Erase and Replace stages. We observe lower quality background reconstruction in this case, evident from the blurry hedge behind the corgi's head in Figure 7.
We also demonstrate the advantage of using random background augmentation in separating the foreground object from the background (see Section (.4). Without this augmentation, the model is unable to accurately separate the foreground and background alpha maps, resulting in a blurry background and floaters that are particularly noticeable when viewed on video (Figure 77)). In contrast, our full composited model trained with background augmentation successfully separates the foreground and background, producing sharp results for the entire scene (Figure
In Figure 7 Left side, we show the importance of the Halo region supervision for the Erase stage. Without it, our model lacks important nearby spatial information, and thus cannot successfully generate the background scene.
5 Conclusion
We introduce RAM3D, a text- text-guided 3D object replacement method for 3D scenes, offering a potential editing tool for VR/MR, gaming, and film production. With an Trase-and-Replase approach, RAM3D can effectively replace objects with significantly different contents that blends seamlessly with the original 3D scene. Our method can also add new objects while maintaining realistic appearance and multi-view consistency. We demonstrate the effectiveness of RAM3D in various realistic 3D scenes (including human avatar, newaadd-faiing and 360 scenes), and superior synthesis quality compared to current state-of-the-art NeRF and Gaussian Splatting based methods.
For future work, our Bubble-NeRF method could be extended to other representations such 3D Gaussian splats [36). similar to DreamGaussian [40. Other interesting future directions include disentangling geometry and appearance to enable more fine- fine-grained control for scene editing, ad- dressing multi-face problems using prompt-debiasing methods [51] or models that are pre-trained on multiview datasets [52 (3), and developing amortized models for faster editing, similar to (54)
References
[1] Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18392-18402, 2023.
[2] Junzhe Zhu and Peiye Zhuang. Hifa: High-fidelity text-to-3d generation with advanced diffusion guidance. In ArXiv:2305.
2023.
[3] Ayaan Haque, Matthew Tancik, Alexei Efros, Aleksander Holynski, and Angjoo Kanazawa. Instruct-nerf2nerft: Editing 3d scenes with instructions. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), 2023.
[4] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. In ArXiv:2204.0611 2022.
[5] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems (NeurIPS), 35:36479-36494 2022.
[6] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2022.
10

---

[7] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen- Or. compt-to-prompp image editing with cross attention control. ArXiv:2208.011 2200:01666, 2022.
[8] Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Ou Or. Null-text inversion for editing real images using guided diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2023.
[9] Chong Mou, Xintao Wang, Jiechong Song, Ying Shan, and Jian Zhang. DragonDiffusion: Enabling Drag-style Manipulation on Diffusion Models. In ArXiv:2307.02-2211 July 2023.
[10] Tianle Li, Max Ku, Cong Wei, and Wenhu Chen. Dreamedit: Subject-driven image editing.
Transactions on Machine Learning Research, 2023.
[11] Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended Diffusion for Text-driven Editing of Natural Images. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2022.
[12] Omri Avrahami, Ohad Fried, and Dani Lischinski, Blended latent diffusion. ACM transactions on graphics (TOG), 2023.
[13] Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. The Eleventh International Conference on Learning Representations (ICLR), 2023.
[14] Haochen Wang, Xiaodan Du, Jiahao Li, Raymond Ych, and Greg Shakhnarovich. Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[15] Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Pro- lifiedreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. Advances in Neural Information Processing Systems (NeurIPS), 36, 2024.
[16] Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen-Or. Latent-nerf for shape-guided generation of 3d shapes and textures. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[17] Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng. Xun Huang, Karsten Kreis, Sanja Fidler, Ming- Yu Liu, and Tsung- Yi Lin. Magic3d: High-resolution text-to-3d content creation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[18] Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. In Proceedings of the IÉEE/CVF International Conference on Computer Vision (ICCV), October 2023.
[19] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2023.
[20] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Bero-l-to-3 Zero-shot one image to 3d object. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), 2023.
[21] Guocheng Qian, Jinjie Mai, Abdullah Hamdi, Jian Ren, Aliaksandr Siarohin, Bing Li, Hsin- Hsin-Ying Lee, Ivan Skorokhodov, Peter Wonka, Sergey Tulyakov, and Bernard Ghanem. Magicl One image to high-quality 3d object generation using both 2d and 3d diffusion priors. In The Twelfth International Conference on Learning Representations (ICIR), 2024.
[22] Steven Liu, Xiuming Zhang, Zhoutong Zhang, Richard Zhang, Jun-Yan Zhu, and Bryan Russell. Editing conditional radiance fields. Proceedings of the International Conference on Computer Vision (ICCV), 2021.
11

---

[23] Can Wang, Menglei Chai, Mingming He, Dongdong Chen, and Jing Liao. Clip-nerf: Text- and-image driven manipulation of neural radiance fields. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2022.
[24] Clément Jambon, Bernhard Kerbl, Georgios Kopanas, Stavros Diolatzis, Thomas Leimkühler, and George" Drettakis. Nerfshop: Interactive editing of neural radiance fields". Proceedings of the ACM on Computer Graphics and Interactive Techniques, May 2023.
[25] Yu-Jie Yuan, Yang- Tian Sun, Yu-Kun Lai, Yuewen Ma. Rongfei Jia, and Lin Gao. Nerf-editing: geometry editing of neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[26] Hyeonseop Song, Seokhun Choi, Hoseok Do, Chul Lee, and Tachyeong Kim. Blending- nerf: Text-driven localized editing in neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
[27] Ashkan Mirzaei, Tristan Aumentado-Armstroong Konstantinos G Derpanis. Jonathan Kelly, Mareus A Brubaker, Igor Gilitschenski, and Alex Levinshtein. Spin-nerf: Multiview segmen- tation and perceptual inpainting with neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[28] Ashkan Mirzaci, Tristan Gumentado-Armstronn Levinshtein, Konstantinos G. Derpanis, and Igor Gilitschenski. Reference-guided controllable inpainting of neural radiance fields. In ICCV, 2023.
Marcus A. Brubaker, Jonathan Kelly, Alex
[29] Silvan Weder, Guillermo Garcia-Hernanee Áron Monszpart, Marc Pollefeys, Gabriel Brostow, Michael Firman, and Sara Vicente. Removing objects from neural radiance fields. In CVPR, 2023.
[30] Liangchen Song, Liangliang Cao, Jiatao Gu, Yifan Jiang, Junsong Yuan, and Hao Tang. Efficient- nerf2nerf: Streamlining text-driven 3d editing with multiview correspondence-entt enhanncd diffu- sion models. In ArXiv:2312.085633 2023.
[31] Jiahua Dong and Yu-Xiong Wang. Vica-nerf: Niew-consittenyywwaar 3d editing of neural radiance fields. Advances in Neural Information Processing Systems (NeurIPS), 2024.
[32] Subin Kim, Kyungmin Lee, June Suk Choi, Jongheon Jeong. Kihyuk Sohn, and Jinwoo Shin. Collaborative score distillation for consistent visual editing. In Thirty-seventh Conference Neural Information Processing Systems (NeurIPS), 2023.
[33] Jingyu Zhuang, Chen Wang, Liang Lin, Lingjie Liu, and Guanbin Li. Dreameditor: Text-driven 3d scene editing with neural fields. In SIGGRAPH Asia 2023 Conference Papers, SA '23. Association for Computing Machinery, 2023.
[34] Xingchen Zhou, Ying He, Richard Yu, Jianqiang Li, and You Li. RePaint-NeRF: Nerf editting via semantic masks and diffusion models. In IJCAI, 2023.
[35] Yiwen Chen, Zilong Chen, Chi Zhang, Feng Wang, Xiaofeng Yang, Yikai Wang. Zhongang Cai, Lei Yang, Huaping Liu, and Guosheng Lin. Gaussianeditor: Swift and controllable 3d editing with gaussian splatting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattem Recognition (CVPR), 2024.
[36] Bernhard Kerbl, Georgios Kopanas, Thomas Leinküüler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions Graphics, 42(4), July 2023.
[37] Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma, Song-Hai Zhang, Mare Habermann, Christian Thcobalt, et al. Wonder3d: Single image to 3d using cross-domain diffusion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2024.
[38] Luca Medeiros. Language segment anything. GitHub repository, 2021.
12

---

[39] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
[40] Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. In The Twelfth International Conference on Learning Representations (ICLR), 2024.
[41] Taoran Yi, Jiemin Fang, Junjie Wang, Guanjun Wu, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Qi Tian, and Xinggang Wang. Gaussiandreamer: Fast generation from text to 3d gaussians by bridging 2d and 3d diffusion models. CVPR, 2024.
[42] Junshu Tang, Tengfei Wang, Bo Zhang, Ting Zhang, Ran Yi, Lizhuang Ma. and Dong Chen. Make-it-3d: High-fidelity 3d creation from single image with diffusion prior. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), 2023.
[43] René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
[44] Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. CVPR, 2022.
[45] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, Proceedings of Machine Learning Research, 2021.
[46] Michal Geyer, Omer Bar-Tal, Shai Bagon, and Tali Dekel. Tokenflow: Consistent diffusion features for consistent video editing. In The Twelfth International Conference on Learning Representations (ICLR), 2024.
[47] Pei-Ze Chiang, Meng-Shine Shiun Tsai, Hung-Yu Tseng, Wei-Sheng Lai, and Wei-Chen Chiu. Styliz- ing 3d scene via implicit representation and hypernetwork. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), January 2022.
[48] Thu Sguyen-Phuoc, Feng Liu, and Lei Xiao. Snerf: stylized neural implicit representations for 3d scenes. ACM Trans. Graph., July 2022.
[49] Zachary Teed and Jia Deng. Raft: Recurrent all all-pairs field transforms for optical flow. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, ECCV 16th European Conference, Proceedings, 2020.
[50] Ori Gordon, Omri Avrahami, and Dani Lischinski. Blended-nerf, Zero-shot object generation and blending in existing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
[51] Susung Hong, Donghoon Ahn, and Seungryong Kim. Debiasing scores and prompts of 2d diffusion for view-consistent text-to-3d generation. Advances in Neural Information Processing Systems (NeurIPS), 2023.
[52] Yichun Shi, Peng Wang, Jianglong Ye, Long Mai, Kejie Li, and Xiao Yang. MVDream: Multi-view diffusion for 3d generation. In The Twelfth International Conference on Learning Representations (ICI.R), 2024.
[53] Guocheng Qian. Jinjie Mai, Abdullah Hamdi, Jian Ren, Aliaksandr Siarohin, Bing Li, Hsin- Hsin-Ying Lee, Ivan Skorokhodov, Peter Wonka, Sergey Tulyakov, and Bernard Ghanem. Magic123: One image to high-quality 3d object generation using both 2d and 3d diffusion priors. In The Twelfih International Conference on Learning Representations (ICLR), 2024.
[54] Jonathan Lorraine, Kevin Xie, Xiaohui Zeng, Chen-Hsuan Lin, Towaki Takikawa, Nicholas Sharp, Tsung- Yi Lin, Ming- Yu Liu, Sanja Fidler, and James Lucas. Att3d: Amortized text-to-3d object synthesis. Proceedings of the IEFF/CVE International Conference on Computer Vision (ICCV), October 2023.
13

---

[55] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitchead, Alexander Berg, Wan Yen Lo, al. Segment anything. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), 2023.
[56] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. DreamBooth: Fine Tuning text-to-lmage Diffusion Models for Subject-Driee Driven Generation. In ArXiv:2208. 122442 March 2023.
[57] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. NeurIPS, 2021.
[58] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with multiresolution hash encoding. ACM Trans. Graph., July 2022.
[59] Lin Yen-Chen. Nerf-pytorch. GitHub repo, 2020.
[60] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations (ICLR), 2015.
[61] Diederick P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Interna- lional Conference on Learning Representations (ICLR), 2015.
[62] Matthew Tancik, Ethan Weber, Evonne Ng, Ruilong Li, Brent Yi, Justin Kerr, Terrance Wang, Alexander Kristoffersen, Jake Austin, Kamyar Salahi, Abhik Ahuja, David McAllister, and Angjoo Kanazawa. Nerfstudio: A modular framework for neural radiance field development. In ACM SIGGRAPH 2023 Conference Proceedings, SIGGRAPH '23, 2023.
14

---

A
Appendix
In this appendix, we first provide an Impact statement Section B We next provide the results of User Study in Section C We provide additional results by our ReplaceAnything331 model (RAM3D) in Section as well as additional comparisons with other methods in Sections and G Section
D we show additional ablation results to further validate our method. We give further details on running the GaussianEditor comparisons in Section H We provide implementation details in Section
D Finally, we discuss current limitations of our method in Section
In the video submission, we show additional qualitative results and comparisons, which better showcase the multi-view consistency of the edited scenes by RAM3D,
B Impact Statements
Our work contributes the advancement 3D generative machine learning, field with significant ethical considerations, such as the potential for the creation and dissemination of deceptive or manipulated visual content. We acknowledge these concerns and emphasize our commitment to responsible research and development practices. We believe that this technology has the potential to empower professionals creative industries and individuals seeking to create 3D content responsibly, and we are dedicated to addressing ethical challenges in its deployment.
C User Study
We performed a User Study" comparing our results with 3 closely related scene works; Gaussian
Editor [35], BlendedNeRF [50] and InstructNeRF2NeRF [3]. 15 participants were asked to compare the results from RAM3Dwith these models on a variety of scenes and prompts. Users were asked to choose the result which best matches the input prompt, and which shows the highest visual quality. We report the preference rates for each model in Tables and 3 note that our model's 's results are preferred overall across both categories.
Table 2: User Study: GARDEN scene and FACE scene
Criterion
Prompt Matching (3 d.p) f
Visual Quality (3 d.p) t GaussianEditor
Prompt
Ours
IN2N GaussianEditor
Ours
IN2N
Pineapple
0.867
0.067
0.067
0.867
0.067
0.067
Plushie Mushroom
0.933
0.000
0.067
0.933
0.000
0.067
Chess
1.000
0.000
0.000
1.000
0.000
0.000
Popcorn
1.000
0.000
0.000
1.000
0.000
0.000
Checkered Jacket
0.800
0.000
0.200
0.530
0.067
0.400
Hawaïan Shirt
0.667
0.000
0.333
0.267
0.067
0.667
Tartan Jacket
0.733
0.067
0.200
0.533
0.000
0.467
Table 3: User Study: FERN scene
Criterion Prompt
Prompt Matching (3 d.p) 1
Visual Quality (3 d.p) 1
Ours
BlendedNeRF
Ours
BlendedNeRF
Mushrooms 0.533
0.467 0.200
0.533
0.467
Strawberry
0.800
0.800
0.200
D Additional ablation studies
We performed additional ablation studies to validate our model design (considering both Erase and Replace training stages), including testing the importance of HiFA and Depth loss terms, and Halo-region supervision.
fAll activities relating to this User Study took place whilst Edward Bartrum was visiting student at KAIST; respondents were KAIST researchers.
15

---

In Figure 8 we show Replace-stage results using simple SDS loss (with random diffusion timestep sampling), adding a corgi in the statue scene. The results show slightly less detailed texture synthesis as the RGB-space HiFA loss component is removed. Nevertheless, note that our pipeline still synthesises the new object in the correct position, which orthogonal to synthesis quality. We additionally provide quantitative ablation results in Table which show that using simple SDS loss leads to only a slight drop in CLIP Text- Text-Image Direction Similarity, whilst removing background augmentation or training model in a single stage also lead to worse performance.
In Figure 9 we show the performance of our model on the Etase-stage task, with Halo supervision and Depth loss components removed, and compare these variants to our full model. We run state-of-the-ard image segmentation model [55] on each model variant's results, detect the main segmentation mask inside a bounding box around the statue region. As shown in Figure (purple region), the original statue segmentation mask is still detected for our No Halo and No Depth loss model variants. However, for our full model, the statue mask is not detected; SAM instead correctly detects the hedge region behind the statue. This implies that our full method has successfully removed the statue, and realistically filled in the background, including the disoccluded region of the hedge. For a quantitative comparison of these model variants, see Table
No Halo
No Depth loss
Full
CLIP Sim. t Table 4: Erase-stage ablation results. We report CLIP Text-Image Direction Similarity scores for all model variants, using the prompt "A white plinth in park, in front of path", on the STATUE scene. Note that our full model performs best.
0.020
0.087
0.145
SDS
1 stage
No BG aug
Full
CLIP Sim. t
0.217 0.218
0.214
0.232
Table Replace-stage ablation results. We report CLIP Image Direction Similarity scores for all model variants, using the prompt "A corgi on white plinth", on the STATUE scene. Note that our full model performs best.
Figure Corgi in STATUE scene, using SDS loss instead of HiFA
Figure 9: STATUE Erase stage ablation: (SAM segmentation shown in purple). From Left Right:
No Halo supervision, No Depth Loss, Full model which successfully removed the original statue.
16

---

E Personalized ReplaceAnythingg3
In addition to text prompts, RAM3D enables users to replace or add their own assets to 3D scenes. This is achieved by first fine-tuning pre-trained inpainting diffusion model with multiple images of a target object using Dreambooth [56]. The resulting fine-tuned model then integrated into RAM3D to enable object replacement in 3D scenes. As shown in Figure TO after the fine-tuning
stage, RAM3D can effectively replace or add objects to new 3D scenes.
Input images
Replace
Add
Figure 10: Users can personalize a 3D scene by replacing or adding their own assets using fine-tuned RAM3D, We achieve this by first fine-tuning an inpainting diffusion model with five images of the target object (left), and then combining with RAM3D to perform object replacement and addition with custom content.
F Additional qualitative comparisons
In Figure 14 we compare our approach with naive 2D baseline where each image is processed individually. For each image (statue) and replace it with new object (corgi) using pre-trained text-to-image inpainting model (Figure T4f second column). We then train NeRF scene with these modified images. As shown in Figure third column, this results in corrupted, inconsistent foreground object since each view is very different from each other, in contrast to our multi-view consistent result.
training set (first column), we mask out the foreground object
In Figure II we compare RAM3D with IesF 3 general scene-editing framework. We note that this method struggles to handle cases where the new object is significantly different from the original one (for example, replace a vase and flowers with pineapple or chess piece in Figure IT second and third column). RAM3D can also generate objects with challenging texture such as the tartan pattern on the jacket, while Instruct-NeRF2Ne5 on inconsistent dataset updates from Instruct- Pnstruct-Pix2Pixx D. More importantly, Dnstruct-NeRF2NNRFF significantly changes the global structure of the scene even when the edit is supposed to be local (for example, only change the clothing). This is undesirable, especially for human face editing where the rest of the identity should be maintained after the edits. Finally, RAM3D is capable of removing objects from the scene completely, whilst Instruct-NeRF2Neet cannot (Figure column 1).
struggles due to its naive reliance
In Figure 12 we additionally show FACE dataset results obtained using 3 followup works to Instruct- NeRF-to-NeRF [31 32 33). These methods also rely on InstructPPxxPPix
to provide 2D image editing prior, and could not successfully edit the scenes in our experiments. resulting in poor image quality as shown Figure 12 We note that all of these methods are limited by the paired training dataset of text editing instructions and images before/after the edit that was used to train InstructPix2Pix. GaussianEditor,, when used general editing mode, also faces the same limitation it relies InstructPix2Pix to update the selected gaussians, demonstrated in the FACE dataset
17

---

Origroal
'Chesiered Jackee
"ltwan SiP
Thrinn Jocket"
OuEs
Figure 11: Comparison with Instruct-NeRF2Nett a general scene-editing framework [3]. Note that unlike our method, Instruct-NRFF2NRRF modifies the entire scene, cannot synthesise complex texture patterns in the FACE scene and completely fails generate a pineapple or chess piece object in the 360 GARDEN scene.
Table CLIP-based metrics for GARDEN and FACE datasets, comparing our method with
Prompts
CLIP Text-Image Direction Similarity t
CLIP Direction Consistency
Ours
InstructNeRF2NeeR
Ours
InstruciNeRF2NeRRF
Pineapple 0.2041
0.0661
0.9590
0.9660
Chess
0.1200
0.0061
0.9457
0.9705
Checker 0.0736
-0.0400
0.9909
0.9931
Hawaïaan Tartan 0.0953
0.1805
0. 0.1436
0.9908
0.9884
0.0714
0.9914
0.9897
results (Figure In contrast these methods, RAM3D distills state-of-the-ar image-inpainting diffusion model to provide 2D generative prior, which has been finetuned for inpainting real images, and consequently provides better 2D guidance for the object replacement scene editing task.
In Figure [3 Left side, we present qualitative comparison with RepaintNeRF [44, object replace- ment using the NeRF Fortress scene. Note that the Manse-and-Replann approach and Bubble- -NRF formulation adopted by RAM3D can generate higher fidelity novel content, with a more complete shape for the apple. In contrast, note that RePaint-NeRR generates incomplete shape which appears to be partially cut off by the boundary of the fortress in the input scene. We hypothesise that this artifact is consequence of the training procedure used by RepaintNeRF, in which the entire input scene modified directly with diffusion guidance.
In Figure 13 Right side, we demonstrate our model's competitive performance with DreamEditor 10 important to note that DreamEditor has limitations terms of handling unbounded scenes due to its reliance on object-centric NeuS [57 Additionally, since DreamEditor relies on mesh representations, it is not clear how this method will perform on editing operations such as object removal, or operations that require significant changes in mesh topologies.
18

---

CSD
VICA NeRF
Tartan Jacket
Checkered
Jacket
EN2N
Ours
Tartan
Jacket
Checkered Jacket
Figure 12: Qualitative comparison with Collaborative Score Distillation (CSD) (3). ViCA-NeRF 31] and EfficienNNeRF2NRRF [30]. All approaches apart from our RAM3D were unsuccessful in producing results with intricate texture details for both the checkered and tartan jackets. Results were obtained using official publicly available implementations. Note that in the case of CSD, there is no officially released 3D editing implementation at the time of writing. We therefore followed the official instructions to incorporate CSD image edits into the Feeruet-NeRF2NeRFF framework.
G Additional quantitative comparisons
In 6 provide further quantitative results, comparing our method to Instruct-NeRF2NeRF 3 follow R. reporting same metrics for both CLIP Text- Direction Similarity and CLIP Direction consistency (instead of warping error using optical flow in Section 4.2). However, we note that Instrect-NeRF2NeRF sometime scores higher for CLIP Direction consistency than our method on edit prompts where it completely fails (see Figure I1I For example, for the edit "checkered jacket" in the FACE dataset, Instruct-NeRF2NNet high-frequency details in the background, resulting in solid color background. We hypothesize that this boosts the CLIP-based temporal consistency score even when the edit is unsuccessful. Therefore, refer readers to the comparisons in the supplemental video for more details.
not only fails to create the checker pattern but also removes
Note that there are minor variations the CLIP Text-Image Direction Similarity scores that we report for RAM3D between Tables [ and 6 This discrepancy is due to requirement to evaluate GaussianFdiioo 35 and InstructNeRF2NeRF (3) results on 2 different rendering paths, as these models trained with different camera conventions (OPENCV vs PINHOLE cameras).
19

---

Original
Original
RePaint-NeRf
DreamEditor
Ours
Ours
Figure 13: Left: Qualitative comparison with RepaintNeRF
34 for object replacement. Figure
adapted from the original RepaintNeRF paper. Right: Qualitative comparison with DreamEditor [10 for object addition ("Add red top hat"). Figure adapted from the original DreamEditor paper.
H Gaussian Editor Framework Limitations
We use the official, publicly available GaussianEditor code implementation to obtain results on the GARDEN and FACE scene shown Figure [ When obtaining results for the GARDEN scene (top row) we first use the Delete functionality to remove the vase from the table. However, we found that we were unable replicate the quality of the object removal results reported the GaussianEditor paper 35 In particular, we found that the mask-dilation and hole-fixing refinement phase of the Delete method resulted in a cloudy cemi-transparent artifact in the area occupied by the removed gaussians (we show the before/after results for this refinement stage in Figure 15). Therefore, in order to compare with the strongest possible baseline, we opted to use the Deletion result from before this refinement step (shown on the left side of Figure 15).
We subsequently use the GaussianEditor Add functionality to incorporate the new object into the scene. We found that the new object was initially placed far from the table by GaussianEditor, which provides user interface for repositioning the object along the z-values of a camera ray passing through the centre of the reference image. We show the before/after results of this manual depth adjustment in Figure 16 when adding the pineapple object onto the table.
I Implementation details
I.1 NeRF architecture
We use an Instant-NGP 58 based implicit function for the RAM3D NeRF architecture, which includes memory- and speed-efficient Multiresolution Hash Encoding layer, together with a 3-layer
20

---

ingaineed trannod
Dun
Figure 14: Qualitative comparisons between our method RAM3D (last column) with a naive 2D baseline method, which produces view-inconsistent results (third column). This is because each input image is processed independently and thus vary widely from each other (second column).
ANKK
AHH
Delete result before refinement
Delete result after refinement
Figure 15: Results obtained using GaussianEditor remove the vase object from the GARDEN scene. Note that GaussianEditor's 's proposed mask dilation and hole-fixing refinement stage causes cloudy artifact appear above the table, where the vase was originally placed. We therefore the result on the left, prior to refinement, when adding new objects to replace the vase.
MLP, hidden dimension 64, which maps ray-sample position to RGB and density. We do not use view-direction as a feature. NeRF rendering code is adapted from the nerf carf-pytorch repo [59
I.2 Monolithic Erase+Replace RAM3D
We use a 2-stage Erase-and-Red and-Replace training schedule for the STATUE, RED-NET and GARDEN scenes. For the FERN scene, we use user-drawn object masks which cover region of empty space in the scene, therefore object removal is redundant. In this case, we perform object addition by providing the input scene-images background compositing images to RAM3D. For the FACE scene, the torso region which we aim to generate new content overlaps entirely with the mask region we therefore train monolithic RAM3D to inpaint new content in the mask in single training stage.
21

---

Object Insertion before depth adjustment
Object insertion after depth adjustment
Figure 16:
We observe that the Object Adding functionality of GaussianEditor does not place new objects into the scene the correct position. The images on the left show the initial pineapple object placement (far from the table), output by GaussianEditor. The images on the right show the results after manually refining the position of the pineapple, by adjusting its depth along ray passing through the centre of the reference image.
1.3 Input Masks
We obtain inpainting masks for object removal by passing dataset images off-the-shelf text-to- mask model (38), which we prompt with 1-word descriptions of the foreground objects to remove. The prompts used are: STATUE scene: "statue". GARDEN scene: "Centrepiece", FACE scene: "Clothes", RED-NET scene: "Bag". We dilate the predicted masks to make sure they fully cover the object.
For the Erase stage, we compute nearby pixels to the exterior of the inpainting mask, and use them as the Halo region (see Figure 3). We apply reconstruction supervision on the Halo region as detailed in I.5] For the object-addition experiments in the FERN scene, create annotated masks in consistent position across the dataset images, covering an unoccupied area of the scene.
In Figure 17, we show an example of the input masks, which can be obtained through either using LangSAM and text prompt to describe the object to replaced or by manual user placement to add more objects to the 3D scene.
Mask for replacing "tree trunk" using LangS/V (e.g: hod stawberry))
User-defred for object adding (e.g: "Add bunny')
Figure 17: Examples input masks: the red arrow indicates examples input masks for object replacement, while the blue arrow shows examples of input masks for object addition.
I.4 Cropping the denoiser inputs
The LDM denoising U-net takes input images of size 512x512. In contrast, RAM3D outputs are of equal resolution the input scene images, which can non-square. To ensure size compatibility, we need to crop and resize the RAM3D outputs to 512x512 before passing them to the denoiser (sec
22

---

Figure 2) For the STATUE, FACE and GARDEN scenes, we resize all images to height 512 and take a centre-crop of 512x512, which always contains the entire object mask region. For the RED-NET scene, the object mask is positioned on the left side of the images; we therefore select the left-most 512 pixels for cropping.
For the FERN scene, input images are annotated with small user-provided masks, We find that the previous approach provides too small of mask region the LDM's denoiser. In this case, we train RAM3D using the original dataset downsampled by a factor of 2 to a resolution of 2016x 1512, and select rectangular crop around the object mask. We compute the tightest rectangular crop which covers the mask region, and then double the crop-region height and width whilst keeping its centre intact. Finally, we increase the crop region height and width to the max of the height and width, obtaining square crop containing the inpainting mask region. We apply this crop to the output of RAM3D and then interpolate to 5 512x512 before proceeding as before.
1.5 Loss functions
During the Erase training stage, we find it necessary to backpropagate reconstruction loss gradients through pixels close to the inpainting mask (See Ablation Figure 7]. to successfully reconstruct the background scene. We therefore additionally render pixels inside the Halo region (Section [.3] Figure 3), and compute reconstruction loss Lrecon and perceptual loss Lugg on these pixels, together with the corresponding region on the input images. Note that the masked image content does not fall inside the Halo region the input images- therefore Lrecon and Luge only provide supervision on the scene backgrounds. For the reconstruction loss, we use mean-squared error computed between the input image and RAM3D's RGB output. For perceptual loss, we use mean-squared error between the features computed at layer 8 of a pre-trained and frozen VGG-16 network [60]. In both cases, the loss is calculated on the exterior of the inpainting mask and backpropagated through the Halo region. During the Replace training phase. following (2. we apply LBOT. loss between rendered output x, and the LDM denoised output &, obtaining gradients to update our NeRF-scene weights towards the LDM image prior (see HiFA Loss in Figure [ eqn 11 (2). No other loss functions are applied during this phase, thus loss gradients are only backpropogated to the pixels on the interior of the inpainting masks. For memory and speed efficiency, RAM3D only renders pixels which lie inside the inpainting mask at this stage (Figure 3), and otherwise samples RGB values directly from the corresponding input image.
Finally, following [42], we apply depth regularisation using the negative Pearson correlation coef- ficient between our NeRF-rendered depth map, and a monocular depth estimate computed on the RGB output. The depth estimate is obtained using an off-the-shell model [43]. This loss is backpropogated through all rendered pixels; i.e the union of the inpainting mask and Halo region shown in Figure 3
We do not apply this regularisation during the Replace stage. In summary, the total loss function for the Replace stage is:
Licaal LAGT + Adephh >rrecon + Auggforp
(6)
with loss weights as follows: Arsoon = 3, Avgg 0.03, Adepth = 3.
We use the Adam optimiser [61] with a learning rate of le-3, which is scaled up by 10 for the Instant-NGP hash encoding parameters.
I66 Training details
Each dataset is downsampled to have shortest image side-length (height) equal to 512 so that square crops provided to the LDM inpainter include the full height of the input image. The FERN scene is an exception, in which we sample smaller 512 image crop within dataset images with downsample factor of 2.
Following
1 we find that classifier-fres guidance (CFG) is critical to obtaining effective gradients for distillation sampling from the LDM denoiser. We use a CFG scale of 30 during the Replace stage, and 7.5 during the Erase stage. We also adopt the HiFA noise-level schedule, with t_min = 0.2, 1_max = 0.98, and use stochasticity hyperparameter v) = In the definition of LAGT loss (see eqn 11 in () we follow HiFA and choose a Aug value of 0.1. We render the RAM3D radiance function using coarse-to-fine sampling strategy, with 128 coarse and 128 fine raysamples. During the Replace training stage, we swap the composited background image with a randomly chosen plain
23

---

(b)
Figure 18: We show 2 failure cases for RAM3D. a) a challenging multi multi-object prompt results in an incoherent combination of both objects. b) we observe multi-face problem which results in unrealistic geometry for the watermelon slice.
Figure 19: We show 2 additional failure cases for RAM3D. a): Editing object properties changes object identity. b): replacing the statue with much larger objects leads to degraded synthesis quality.
RGB image every 3rd training step. As shown in the Ablation Figure 7 this step is critical to achieving clean separation of foreground and background.
We train RAM3D for 20,000 training steps, during both Erase and Replace training stages. Training hours on a single 32GB V100 GPU. The output of our Replace stage RAM3D training is a set of multiview images which match the input scene images on the visible region, and contain inpainted content on the interior of the masked region which is consistent across views. To obtain novel views, we train NeRFs with standard novel view synthesis methods using RAM3D edited images and the original scene cameras poses as training datasets. We use nerf- [59] for LLFF scenes (STATUE, FERN, RED-NET SCENES), the nerf-studio [62 Nerfacto model for the FACE scene and Gaussian Splatting 36 for the GARDEN scene.
takes approximately
Limitations
RAM3D is capable of adding multiple objects scenes by using multiple prompts. However, we note in Figure [8]a) that adding multiple objects using a single edit prompt can sometimes result in an implausible composition which lacks a coherent spatial relationship. is based on text-to-image model distillation techniques, our
As noted in Appendix 4.1
Furthermore, as method suffers from similar artifacts to these methods, such as the Janus multi-face problem, as shown in Figure b).
Our Erase-ande and- and-eelaaceee approach might remove important structural information from original objects, so it is not suitable for editing objects' properties such as appearance or geometry (for example, turning the statue gold, see Figure 19 a).). Our method generates Bubble NeRF from scratch inside the edit region, meaning that the structural information regarding the original statue is lost. Consequently, our model generates an entirely new statue, with gold appearance.
We note that our method works best when Erasing and Replacing objects of similar size. A significant size mismatch between Erased and Replaced objects may lead to degraded synthesis quality. This can be observed in Figure b)), (replacing the statue with a bus).
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We provide our goal for the paper, short description of the method and overview of the results included in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We dedicate section to discuss current limitations with examples for failure cases, and discuss future direction to rectify this (see Appendix section M).
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
25

---

Answer: [NA]
Justification:
Our work does not focus on theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We provide details on data pre- pre-processing and training steps in the Appendix section [.6
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
26

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We base our method on open-source pre-trained models without proposing new datasets. Comprehensive implementation and necessary infrastructure details for our method are provided in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//npps.cc/pulli//uiiiee htapb//ii/gguudee//doeeeeeeeeeee
swtsssssssiiiiyyyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permirted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide comprchensive details on data preprocessing, implementation, and necessary infrastructure for our method in the Appendix section [.6
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: scenes.
Our method is applied for each scene instead of large datasets of multiple
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
27

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide details on training time, process, and our infrastructure description (Appendix section[.66)
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We have carefully reviewed NeurIPS Code of Ethics
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We include an Impact Statements section in Appendix (section B
Guidelines:
The answer NA means that there is no societal impact of the work performed.
28

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do release models datasets.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We utilize MIT-licenser datasets and open-source source models, all of which have been widely used in prior publications.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
29

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: paper does not release assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mooel part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: We provided the questions that were asked in our User Study in Appendix Sec. C whilst the relevant videos are shown in the video on our project page. Respondents participated voluntarily and were not compensated.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [No]
Justification: Our User Study (Appendix Sec. C did not incur risks to study participants and does not require specific approval.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
30

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
31


Reviews:
[{"review_id": "rGEVJxSgQG", "reviewer": "Reviewer_NhGj", "summary": "This paper studies instruction-guided scene editing. They introduce ReplaceAnything3D (RAM3D, not sure why using this abbreviation), a two-stage method to first erase the object to be replaced and in-paint the background in the scene, and then generate the replace-to object and compose it to the scene after stage one. Some relatively good visualization results are shown in the paper.", "questions": "Please see \"Weaknesses\".", "limitations": "The authors have included broader impacts and limitations in the paper.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "- The proposed pipeline itself is novel and well-motivated.\n- Some of the visualization results look good.", "weaknesses": "- The method highly utilized HiFA's loss and design in their pipeline, but no ablation study was provided about HiFA to indicate whether it is HiFA or the proposed method contributes most to the good results. More specifically, I would like to understand:\n    - If adding HiFA's loss and design with Instruct-NeRF2NeRF or GaussianEditor, can they achieve comparable results as the proposed method?\n    - If the proposed method removes HiFA's loss and uses simple loss - so it is natural that the results are worse than the full model, will it still get better results in GaussianEditor?\n- Some of the editing results are not good enough. \n    - For clothes editing, there is always an obvious cutting edge (sometimes looks like the collar of a sweater) on the neck part.\n- The replacement task is easier than standard instruction-guided editing, which explicitly indicates the replace-from and replace-to objects instead of requiring the diffusion model to infer. \n    - This makes some comparisons unfair, especially the ones in the model that struggle to infer the replace-from region, e.g., Fig.9. \n    - On the other hand, this setting also disables some editing, like more implicit instructions like \"turn him into a bald\" (i.e., remove hair).\n    - Besides, removing an object and regenerating one prevents the model from keeping sufficient identities from the replace-from objects, which is not acceptable in some cases, e.g., color editing of an object may result in highly different objects.\n    - The authors need to argue why this setting is reasonable and more/comparably reasonable compared with traditional instruction-guided editing tasks.\n- The paper has multiple presentation deflects, especially the math formulas. \n    - For example, the subscripts should not be italicized. $\\lambda_{HiFA}$ is used in the paper, which means $\\lambda_{H\\times i\\times F\\times A}$ instead of $\\lambda_{\\mathrm{HiFA}}$.\n    - The same variable should be referenced consistently, but the paper contains $y_{\\mathrm{replace}}$ at L127/129/203 and $y_{replace}$ (which means $y_{r\\times e\\times p\\times l\\times a\\times c\\times e}$) at L174/185.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer NhGj for their review! We are glad that they found that our ‘visualization results look good’, and that the pipeline is ‘novel and well-motivated’. We now address the reviewer’s specific concerns and questions:\n\n“No ablation study was provided about HiFA” - we have now added this ablation study to the attached rebuttal PDF. Note that HiFA loss is highly related to SDS loss, and differs only by an RGB loss component (see eqn 4 in HiFA paper).  In Figure 19a), (rebuttal PDF) we show Replace-stage results using simple SDS loss (with random diffusion timestep sampling) adding a corgi in the statue scene. The results show slightly less detailed texture synthesis as the RGB-space loss is removed. Nevertheless, note that our pipeline still synthesises the new object in the correct position, which is orthogonal to synthesis quality. We additionally provide quantitative ablation results in Table 6, which show that using simple SDS loss leads to only a slight drop in CLIP Text-Image Direction Similarity.\n\n“If adding HiFA’s loss and design with Instruct-NeRF2NeRF or GaussianEditor, can they achieve comparable results?” - this is based on a misunderstanding; there is no SDS loss or other distillation loss in those methods. Instruct-NeRF2NeRF achieves scene editing using iterative dataset updates, which are obtained with a prior Instruct-Pix2Pix model. So it is not clear how HiFA loss could be incorporated into InstructNeRF2NeRF. Similarly, GaussianEditor employs the same training strategy for its Edit functionality. It differs from InstructNeRF2NeRF mainly in its choice of 3D representation (3DGS instead of NeRF). Both methods use an iterative dataset update strategy to update the existing 3D scene. In contrast, our method learns a localised 3D representation (Bubble-NeRF) for the edit region, from scratch. For this reason, our method is not biased towards the original scene geometry, which enables larger modifications.\n\nFinally, regarding the GaussianEditor object Addition functionality, this makes use of an off-the-shelf image-to-3d method (Wonder3D, which also does not use distillation) to generate the new object separately (which does not guarantee harmonisation with the original scene), and leaves it to the user to manually adjust the new object’s depth (see section F appendix main paper, and Figure 14). As we note in the main paper (line 226) this results in visible artefacts at the boundary between the object and its surroundings, and significant quality issues (see Figure 5, and User Study Table 3 in the Rebuttal PDF). It is unclear how these issues could be addressed using HiFA’s loss and design.\n\n“Some of the editing results are not good enough. For clothes editing, there is always an obvious cutting edge on the neck…” We only claim that our model outperforms the existing state-of-the-art on this task. We would be grateful if reviewer NhGj could indicate which published works can perform better, in terms of synthesising intricate localised texture/geometry patterns to match the edit prompt. We compared with the state of the art GaussianEditor in figure 5, and note that it fails to update the geometry of the clothes to match the edit prompt, whilst the texture synthesis is also inferior to our model. We would like to improve the generated geometry still further in future work, but still note that our model performs better than existing models on this challenging scene.\n\n“The replacement task is easier than standard instruction-guided editing … this makes some comparisons unfair, eg Fig.9” - Please note that in the main paper, we compare with state-of-the-art localised editing methods; Reference Guided Inpainting, Blended-NeRF, GaussianEditor, as well as DreamEditor and RePaintNeRF in the appendix. Each of these uses a spatial prompt to indicate the edit region; for example, GaussianEditor uses a SegmentAnything-based mask to select the edit region, exactly like ours. In Figure 9 (appendix) we compare with Instruct-NeRF2NeRF (standard instruction-guided editing) only for completeness - nevertheless, all main paper comparisons are like-for-like. Note that GaussianEditor (Addition functionality), Reference-Guided Inpainting, Blended NeRF, Repaint-NeRF and DreamEditor all use edit-descriptive prompts as guidance exactly like ours rather than being “instruction-guided”. We note that our Erase-and-Replace strategy for scene editing obtains better results than the existing SOTA 3D localised methods. Reducing the task to Erase-and-Replace should therefore be considered part of the appeal of our method.\n\n“The authors need to argue why this setting is reasonable compared with traditional instruction-guided editing…” Localised 3D scene editing is a well-established research track with multiple related publications (please see our Related Work section, in particular lines 82 to 103). These works may be seen as a 3D extension of 2D conditional inpainting research works such as RePaint and Blended Diffusion, and have myriad potential applications in Mixed Reality and Film Production. These applications require 3D object editing functionalities whilst keeping object surroundings consistent. Note that this is not possible for general scene-editing frameworks as they a) tend to modify the entire scene, (Fig. 9 and 10, main paper, and Fig. 3 in the DreamEditor paper) and b) struggle with object removal (Fig. 9 main paper). Therefore, the localised 3D scene editing track has emerged to address these shortcomings, and should not be considered ‘easier than standard instruction-guided editing’ as it is simply addressing a different task.\n\n“The paper has multiple presentation defects, especially the math formulas…” We are grateful to the reviewer for bringing this to our attention! We will fix these defects for the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8x6043H3S1", "reviewer": "Reviewer_MzLe", "summary": "The authors introduce a method for replacing 3D objects within 3D scenes based on text descriptions. They do this by using in-painting approaches for the set of  images that are used for down-stream novel-view synthesis.", "questions": "Will the trade-offs of using BubbleNerf representation harm the final stage of training a nerf according to multi-view images synthesized by BubbleNerfs? I'm assuming that's the reason why you want to use Nerfs for your final representation, and not the BubbleNerf representations you use for the erase and replace stages. Do you trade speed/expressive power for quality when you use BubbleNerfs?\n\nHow were the language prompts chosen for objects to add?", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper introduces an interesting and important application of distilling 2D diffusion models into 3D.\n\nThe use of a 3D implicit representation for both the erase and replace stages is well motivated for preserving 3D consistency. \n\nAdding a slight halo seems like a nice solution for getting slight interactions (shadows, reflections) between the generated objects and the background.\n\nThey demonstrate their method working on single and multiple objects.\n\nThe baselines they compare against seem sensible.", "weaknesses": "One big assumption from the way the method works is that the object you're attempting to replace is LARGER than the object you're replacing it with. SAM cannot necessarily anticipate the size difference between the two objects, and so during the erase stage the best it will be able to do is mask out the small area, in which case the area that you can inpaint over is very small. For instance, replacing the statue in Figure 2 with something larger, like a bus, will not work.\n\nThe opposite problem may also be true -- when the object being replaced is TOO much larger than the size of the object that is being added, the replace stage can end up over-sizing the object being added (perhaps encouraged by your foreground/background disentanglement, mentioned in Line 188 ). This can be seen in figure 3, where the cookies look (good but) all too gigantic in the scene.\n\nAdditionally, the Halo technique assumes that longer range interactions (further apart spatially within the image) between objects reflections don't exist, though this would easily not be true for reflective surfaces.\n\nA table/quantitative results for Ablation studies would be more convincing. I'm especially curious about the impact of the depth reconstruction loss.\n\nAs you say in line 245, 3D scene editing is highly subjective -- in which case, I think a user study might make your claims stronger.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to reviewer MzLe for their review! We are glad that they found our application “interesting and important”, that our Halo region method is a “nice solution”, and that the baselines we compare to are “sensible”. We now address the reviewer’s specific concerns and questions:\n\n“Replacing the statue with something larger like a bus will not work…” We are grateful for this idea, and have generated new results for the statue scene, replacing the statue scene with a bus. As can be seen in Figure 18b) (attached Rebuttal PDF), the model adds a small bus onto the plinth. However, as the reviewer hypothesised, the significant size mismatch between the Erased and newly-added object seems to lead to degraded synthesis quality. We are happy to add this to the limitations section in the camera ready.\n\n“In figure 3, the cookies look good but too gigantic in the scene…” This is true, and is a consequence of the generative prior from the inpainting model, which tends to generate objects that are scaled to fill up the masked region as much as possible, rather than a realistic scale relative to the scene. We would be happy to add a note on this to the limitations section.\n\n“The Halo technique assumes that longer range interactions don’t exist … this would not be true for reflective surfaces.” This is true, and modelling reflective surfaces would be an interesting challenge for future work. We note that none of the existing localised scene editing papers can handle reflections either. In principle the Instruct-NeRF2NeRF-based global 3D editing papers may be able to handle reflective surfaces, but as far as we know this has not yet been demonstrated in practice. \n\n“A table/quantitative results for Ablation studies would be more convincing. I'm especially curious about the impact of the depth reconstruction loss… a user study might make your claims stronger.” We thank the reviewer for these 2 great suggestions! We have now conducted a user study, and computed additional ablation results including quantitative results, shown in the rebuttal pdf Tables 5 and 6. The ablation results include a study on the impact of the depth loss, which is important for obtaining good quality results in the “Erase” stage. We also show SAM predictions (purple region) for segmentation masks in Figure 17, Rebuttal PDF, for the ablation model variants. Note that when the depth loss is removed, the Erased statue can still be detected by SAM, when applied to our model’s results. The same issue is observed when Halo loss is removed. However, SAM cannot detect the statue region when applied to our Full model’s Erase results, and instead correctly segments the hedge behind the statue.\n\nFor the user study, 15 participants were asked to compare our results with baseline models on Garden, Face and Fern scenes, using a variety of edit prompts. For each scene and prompt, participants were asked to indicate the best performing model in 2 categories. First, they were asked: “Which result most closely matches the given text prompt?”. And second, they were asked: “Which result shows the highest visual quality?” As shown in Tables 3,4 (Rebuttal PDF), our results were preferred overall in both categories.\n\n“Do you trade speed/expressive power for quality when you use BubbleNeRFs?” Thank you for this interesting question! No, there is no quality tradeoff when using BubbleNeRFs. The tradeoff is that they only represent the localised mask region in the training dataset. As a consequence, our method enables us to dedicate almost the entire GPU memory capacity to querying the relevant rays corresponding to the edit region. This allows our method to work with higher final scene resolutions than InstructNeRF2NeRF (max resolution 512) or BlendedNeRF (max resolution 168); for example, we show results on the Fern scene at a final resolution of 2016x1512.\n\nWe cannot perform Novel-View-Synthesis of the entire scene using the BubbleNeRF, as it doesn’t model the scene background outside of the masked region. That is why we render our trained BubbleNeRF from every training set camera pose, and composite these renderings with the training images to obtain updated training images. These are then used to optimise a new NeRF or 3DGS scene (see Section 3.5, main paper), to obtain the final output.\n\n“How were the language prompts chosen for objects to add?” We did not use a language model to generate prompts (as was done for example in InstructPix2Pix, as it requires a large training corpus of prompts), but rather simply wrote them out manually. Similarly to diffusion-based text-to-3D works, we experimented with adding real and mythical creatures (bunnies, corgis, dragons etc) as well as inanimate objects (including foods like pancakes and cookies) as can be seen in Figure 1 main paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SFCfYjv37T", "reviewer": "Reviewer_MnXs", "summary": "This paper introduces a novel method that utilizes the Erase-and-Replace strategy for text-guided object replacement in 3D scenes. Given a collection of multi-view images, camera viewpoints, text prompts to describe the objects to be replaced and to be added, this method first optimizes the background scene with the original object erased, and then optimizes the foreground object to be added. Specifically, in the erase stage, it uses the HiFA distillation technique and optimizes a localized NeRF region, which covers the mask region of the original object and the surrounding nearby pixels. In the replace stage, it optimizes the foreground NeRF within the mask of the original object while alternatively substituting the background with constant RGB tensor during the distillation. The proposed approach enables localized 3D editing and obtains more realistic and detailed edited results.", "questions": "Questions:\n(1) Figure 5 shows that the GaussianEditor makes less geometric modification compared to the proposed method. Is the main reason the Gaussian or NeRF representation?\n(2) Is the proposed method able to replace the original object with a larger one? What if specifying a mask much larger than the original object?\n(3) I’m curious about the property modification mentioned in the limitation. What if erasing the object with the original property and replacing it with the object with new property? \n(4) It’s better to clarify the meanings of \\theta_{bg} and \\theta_{fg} in Section 3.2. Line 150 says “optimise RAM3D parameters \\theta_{bg}”. It makes me assume that \\theta_{bg} is all the parameters of RAM3D at this moment.", "limitations": "The paper includes a discussion about the limitations.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The strengths of this paper include:\n(1) It proposes a novel Erase-and-Replace strategy that optimizes the background and foreground NeRF separately as compositional scenes, which obtains more realistic and detailed results.\n(2) It presents extensive experiments to compare with other SOTA methods and outperforms the others in terms of the qualitative and quantitative results.\n(3) The ablation study validates the effective role of the key designs of the proposed approach.", "weaknesses": "The limitation of the proposed approach is also obvious, as described in the limitation section. Is is only suitable for the remove, add, replace editing. The added objects should be within the pre-specified mask region. And the training is time-consuming.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to reviewer MnXs for their review! We are glad that they find our method “novel”, that it “enables localized 3D editing and obtains more realistic and detailed editing results”, that our experiments are “extensive”, and that our “ablation study validates the effective role of the key designs”.\n\nWe now address the reviewer’s specific concerns and questions: \n“The proposed approach is only suitable for the remove, add, replace editing… added objects should be within the pre-specified mask region”. This is an accurate limitation; the scope of our work is localised scene editing, on the same track as the works cited in lines 82 to 103 in the main paper.\n\n“The training is time-consuming” Our training time is in line with other NeRF-based localised scene editing approaches, but we recognise that GaussianEditor provides much faster results due to its usage of the highly efficient 3D Gaussian Splatting representation. Nevertheless, there is a clear quality tradeoff in this case as we show in figure 5 of our main paper, table 1 main paper, and the user study results in tables 3,4 of the rebuttal PDF. As we also note in figure 14 of our main paper appendix, GaussianEditor requires manual adjustment of the new inserted object’s position, unlike our method which places objects automatically.\n\n“Figure 5 shows that GaussianEditor makes less geometric modification compared to the proposed method. Is the main reason the Gaussian or NeRF representation?” We thank the reviewer for this interesting question! The main reason is the same as why Instruct-NeRF2NeRF also makes less geometric modification. Both of those methods use an iterative dataset update strategy to update the existing 3D scene (Gaussian in the case of GaussianEditor, and NeRF in the case of Instruct-NeRF2NeRF). In contrast, our method learns a localised 3D representation (Bubble-NeRF) for the edit region, from scratch. For this reason, our method is not biased towards the original scene geometry, and is able to make larger modifications.\n\n“Is the proposed method able to replace the original object with a larger one?” We tried replacing the statue with a bus in the rebuttal PDF in figure 18b). We note that the model succeeds in adding a small bus onto the plinth. However, the significant size mismatch between the Erased and newly-added object seems to lead to degraded synthesis quality. \n“What if specifying a mask much larger than the original object?”  We tried specifying a much larger mask than the original statue and generating “a super gigantic mechanical warrior robot” -  results are in the rebuttal PDF, Figure 19b). We show the outline of the original statue object in red.\n\n“What if erasing the object with the original property and replacing it with the object with the new property?”  We tried replacing the statue with a gold statue, see results in rebuttal PDF, Figure 18a). Because our method generates a Bubble NeRF from scratch inside the edit region, the structural information regarding the original statue is lost, as stated in the limitations section of the main paper on line 654. Consequently, our model generates an entirely new statue, with gold appearance - we are happy to add this figure into the limitations section in the camera-ready.\n“It’s better to clarify the meaning of \\theta_{bg} and \\theta_{fg} in Section 3.2…” We thank the reviewer for this recommendation, and will be happy to fix this in the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4Mgw9HSuKJ", "reviewer": "Reviewer_ZFSz", "summary": "This paper introduces a method that can add a novel generated or customized object into a 3D reconstructed scene. To enable this, it proposes a Erase-and-Replace strategy. The first step is to remove an object queried by a text prompt: it segment out the target object using existing language based model and fine-tune existing NeRF model based on the inpainted images. The second step is to replace the object where it fine-tune existing NeRF model using the generated or customized images. The model compared with existing instruction based gaussian splatting and NeRF methods.", "questions": "Based on the current Erase-and-Replace pipeline, it is not fully clear how the authors perform only style changes, e.g., the results in Figure 5-(bottom row). During the inpainting time, if the model fully removes the objects that belong to the dedicated categories in the text prompt, there should be some artifacts such as shape distortion. However, it seems the shape of the man’s torso is perfectly preserved. In such case, how could the shape be perfectly preserved while only changing the style?", "limitations": "This paper discussed about the limitations.", "rating": 3, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 1, "strengths": "- The paper is very straightforward and therefore it is easy to follow.\n- The quality is convincing and promising (without considering previous works, though).", "weaknesses": "]The main weakness of this paper is coming from the lack of novelty.]\n- The scope of the previous works highly within the previous works, and it actually does not push and advance well the boundary in the field of the research area of the 3D editing field. More specifically, it does not address any challenge which has not been addressed yet. Since the challenge is blurry, the contributions are highly suppressed.\n- The proposed method is quite obvious and too straightforward. The impression from this paper, this paper is just to combine existing 3D inpainting and replacement methods. Most components inherit from previous works (or more advanced LLM components) where this paper optimize the engineering parts of the framework (e.g., effectively combine the previous modules, and better and marginally handling the text prompt, data, and mask inputs). Therefore, it also leads to the impression of the product engineering documents not research paper. \n- Finally, the problem itself is not that novel and the proposed method does not even open extra-functionality in the field of 3D editing, which further suppresses the novelty of this paper.\n\n[Potential unfairness]\n- The way how existing methods use the prompt is different from the way how the authors used. Previous works have been using the prompts in a way of instruction style, e.g., turn the face to the hulk.\n- To be more fair way, it would be nice if the authors can show the results from the same data with the same prompt as shown in the previous works. Since quantifying the performance of the 3D generative works is indeed challenging, this could be one of the ideal and most fair way for apple-to-apple comparison.\n\n[Marginal quality improvement]\n- Overall, the quality improvement from the previous works is not that promising. Based on the visual results, it is quite difficult to say which one is showing the better results. Based on the prediction results from the “original papers” (not this paper), it is hard to tell which ones are better. It also leads to the impression that the results shown in the paper is highly cherry-picked.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to reviewer ZFSz for their review! However, we are concerned that the reviewer has misunderstood our proposed method, as the summary contains multiple factual inaccuracies. In particular, in their summary, the reviewer states that we:\n“Fine-tune the existing NeRF model based on inpainted images” during the Erase Stage, in order to remove the target object, and then “Fine-tune the existing NeRF model using generated or customized images” during the Replace stage. Both of these are misunderstandings of our method. We do not fine tune an existing NeRF scene in our method; we introduce a novel Bubble-NeRF representation (Please see Fig 3, and lines 159 to 163 in main paper) which is trained from scratch, and which models a localised NeRF region corresponding to the masked pixels. \n\nIn Section 3.3 main paper, we explain how we first learn a background Bubble-NeRF during the Erase Stage, and in Section 3.4, we explain how we learn a foreground Bubble-NeRF which is composited with the background. In neither stage do we optimise these representations “using generated or customised images” as the reviewer claimed: directly training using inpainted images produces view-inconsistent results as we show in Fig 12, appendix, main paper. That is why we instead use a distillation-based training objective. However, given that the reviewer made no mention of our compositional structure, or localised Bubble-NeRF rendering in their summary (recognised as novelties by the other reviewers), we would be very grateful if they could carefully review our method (Section 3 main paper) before reconsidering their contribution rating.\n\nWe now address the reviewer’s specific concerns and questions: \n“The proposed method is quite obvious and too straightforward… most components inherit from previous works…” As mentioned above, we are concerned that these statements are based on an erroneous understanding of our method. The novel components of our method; localised Bubble-NeRF rendering, Halo region supervision, compositional rendering approach were not mentioned by the reviewer in their summary. We show that an Erase-and-Replace methodology can simplify the task of 3D localised scene editing; an interesting insight that we believe the community can benefit from.\n\n“The problem itself is not that novel and the method does no open extra-functionality in the field of 3D editing” We showed how our model’s compositional structure allows adding multiple objects into the scene (Figure 6 main paper) as well as adding personalised objects using DreamBooth diffusion-finetuning in section C of the appendix, which is a practical novelty, as we are not aware of existing works with these capabilities. As far as we are aware, we are also the first to show high-quality object replacement results on 360 scenes such as the Garden scene, which constitutes a quality novelty.\n\n“The way how existing methods use the prompt is different from the way how the authors used.” Please note that we compare our method with 5 localised editing papers in our main paper (Gaussian Editor, Blended NeRF, DreamEditor, Reference-Guided Controllable Inpainting of NeRF, Repaint-NeRF). All of these methods are conditioned by a spatial prompt, indicating the region which should be edited, in addition to the target text prompt. For example, GaussianEditor specifies the editing region using segmentation masks obtained from SegmentAnythingModel, exactly like ours. The instruction-style of prompts used by InstructNeRF2NeRF (and the similar works shown in the appendix figure 10) is due to their reliance on InstructPix2Pix, an image translation method which is not required in our method. In contrast, our model is conditioned by prompts which directly describe the desired new object, exactly like GaussianEditor (Object Adding functionality), Blended NeRF, DreamEditor, Reference-Guided Controllable Inpainting of NeRF, Repaint-NeRF. We would be grateful if the reviewer could elaborate on why this is a disadvantage. Please note that InstructNeRF2NeRF tackles the related but separate problem of scene-level editing. That is why we compare our method with state-of-the-art localised editing papers in the main text of our paper, and compare with Instruct-NeRF2NeRF only for completeness in the appendix.\n\n“The quality improvement from the previous works is not that promising. Based on the visual results, it is quite difficult to say which one is showing the better results.” We have conducted a User Study which can be found in tables 3 and 4, rebuttal PDF, showing that our model’s results are preferred overall, compared with the existing works. We also show quantitative CLIP-based prompt-matching and temporal consistency results in Tables 1 and 2, which show that our model performs best overall.\n\n“It is not fully clear how the authors perform only style changes, e.g the in Figure 5 (bottom row)...” This is explained in section G2 of our appendix, main paper. As mentioned on line 571, we use a Replace-only variant of our model for the Face scene, because “the torso region in which we aim to generate new content overlaps entirely with the mask region”. Therefore, fully removing the torso in a separate Erase stage is unnecessary. Note that our method is able to generate new geometric details on the clothes (collars and jacket lapels), whilst the baseline methods keep the original geometry fixed, only updating the texture (see Figures 5 and 9 main paper).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their feedback! We are glad that reviewers NhGj, MzLe, MnXs found that our proposed method is novel and well-motivated, and that MzLe and MnXs in particular praised the quality of our results.

A concern that was raised by both reviewers NhGj and ZFSz was that InstructNeRF2NeRF does not use a spatial prompt to indicate the editing region, and instead uses instruction-style prompts. However, it is important to note that InstructNeRF2NeRF (together with the closely related methods shown in figure 10, appendix main paper) is a **global scene editing method**, which we only compare to in the appendix for completeness. In the main paper, we compared to state-of-art methods for **localised scene editing** (Table 1 main paper, Figure 4,5 main paper, and Figure 11 appendix), which like ours, are all conditioned with a spatial prompt indicating the 3D editing region. Furthermore, GaussianEditor (Object Adding functionality), Blended NeRF, DreamEditor, Reference-Guided Controllable Inpainting of NeRF, and Repaint-NeRF all use edit prompts which describe the new object directly (exactly as we do), rather than instruction-style prompts. The localised scene editing task has recently become a well established subtask of global scene editing, and active area of research due to its potential use cases in mixed reality applications (see lines 82-103 main paper for further details).

Reviewers MnXs and NhGj pointed out some minor presentation defects, which we are happy to address for the camera-ready version.

We provide multiple new qualitative and quantitative results in the attached rebuttal PDF document. Some reviewers proposed new experiments with variations on the input prompts and masks, for which we show qualitative results in Figures 18 and 19b).

 We ran multiple new ablation studies to validate our individual components, (on both the Erase and Replace training stages) and include quantitative results for these in Tables 5 and 6. These include new experiments testing the importance of the HiFA and depth loss terms. As in Table 1 main paper, we report CLIP Text-Image Direction Similarity for all model variants. For the “Replace” ablation studies, we used the prompt “a corgi on a white plinth”. For the “Erase” results, we evaluated using the prompt “A white plinth in a park, in front of a path”. 

As an additional evaluation of the “Erase” model variants, we ran SAM on each model’s results, obtaining the main segmentation mask detected inside a bounding box around the statue region. As shown in Figure 17, rebuttal PDF (purple region), the original statue segmentation mask is still detected for our No Halo and No Depth loss model variants. However, for our full model, the statue mask is not detected; SAM instead correctly detects the hedge region behind the statue. This implies that our method has successfully removed the statue, and realistically filled in the background, including the disoccluded region of the hedge.

We also performed a User Study (as suggested by MzLe), comparing our results with 3 closely related works; Gaussian Editor, BlendedNeRF and InstructNeRF2NeRF. 15 participants were asked to compare the results from these models on a variety of scenes and prompts. In each case, they were asked to choose which result best matches the input prompt, and which one shows the highest visual quality. We report the preference rates for each model in Tables 3 and 4 (rebuttal PDF), which show that our model’s results were preferred overall across both categories.

We now address in detail the specific concerns and questions raised by the reviewers, and would welcome further comments from the reviewers seeking further clarification!

Meta-review: The paper introduce ReplaceAnything3D (RAM3D), a method that allows users to replace an object in 3D scene (formed from multi-view images) by using text to describe the object to replace, and the object to add.  To accomplish this, the authors proposes a three-stage method that 1) detect and segment the object to be removed (using LangSAM), and then use 3D inpainting (LDM with HiFA) to 2) fill in the background and 3) add the requested object.  The paper includes some comparisons on selected scenes and prompts that shows the proposed method is better at replacing objects than alternative baselines. 

The submission received divergent ratings with two reviewers (MnXs, MzLe) leaning toward accept, and two reviewers (NhGj, ZFSz) voting to reject.

The main concerns expressed by reviewers are:
1. Fairness of comparison against prior work
   - Missing ablation of HiFA vs key contributions of this work [NhGj]
   - Lack of systematic comparisons with same scene and same prompt [ZFSz]
   - Potential cherry picking of results (based on comparison to results show in other papers originally) [ZFSz]    
2. Limited novelty with a straight-forward approach for a very specific targeted text-based editing [ZFSz]
3. Some details need to be clarified and discussion added, including:
  - clarification of specific parts of HiFA that was used [NhGj]
  - more motivation justifying the specific focus on replacement and discussion when comparing against baselines about how the baselines are more general instruction-based editing while the proposed method is designed for object replacement. 
  - more discussion of the limitations regarding object size [MzLe,MnXs] and replacement when only style changed desired [NhGj,MnXs,ZFSz]
  - replace Fig. 10 with results without `--downscale-factor 2` so that generation from prior work is artifact-free [NhGj]

From the AC's perspective, the main concerns are in the fairness of comparison to prior work (1).  During the rebuttal period, the authors provided additional ablations and an user study comparing the proposed methods to baselines.  The AC believes the user study provides a reasonably systematic comparison of prior work. Reviewer NhGj also did a deep dive to investigate the artifacts shown in Fig. 10 for prior work, and determined that while the figure should be replaced to more accurately reflect the performance of prior work, the proposed method does indeed produce more accurate replacements than prior work.  Thus the AC believes concern (1) to have been mostly addressed by the author response.

For the other concerns, (3) can be addressed by improvements to the exposition, and for novelty (2) most reviewers did not have an issue with the novelty of the approach. In addition, the fact that the approach is straight-forward and intuitive does not lessen the contribution.  All reviewers also recognized the approach is very targeted, but it can be practically useful.  

Due to the above, the AC believes that by incorporating the additional results and discussions from the author response period, the work would be appropriate to be presented at NeurIPS.

The authors should make sure to include the following:
- User study (from rebuttal).  It would be ideal if the the user study can be made to be more systematic (for instance, BlendedNERF and Reference-Guided Inpainting also compared against for GARDEN and FACE scenes).  
- Inclusion of ablation of HiFA loss vs SDS (from rebuttal)
- Include quantitive results for ablations (from rebuttal)
- Replace Fig. 10 with results without `--downscale-factor 2` so that generation from prior work is artifact-free
- Add clarification of specific parts of HiFA that was used
- Add discussion (in experiments) of differences between the more general text-based editing that is supported by methods that are compared against vs the very specific editing supported by the proposed method
- Improved discussion of limitations, including limitations on object size, replacement when there is a style change while preserving the object shape, etc.
- Other minor improvements to writing and presentation
  - Improve notation and typesetting of math equations  (see comments from NhGj)
  - Try to clean up the references (make sure paper venues are included, consistent names for conferences, remove extra information [51], paper titles are properly capitalized).   Alphabetic ordering is recommended.
Decision: Accept (poster)