ReplaceAnything3D: Text-Guided Object Replacement in 3D Scenes with Compositional Scene Representations

Authors: edward  bartrum
thu  nguyen-phuoc
chris  xie
zhengqin  li
numair  khan
armen  avetisyan
douglas  lanman
lei  xiao
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 11:22:47

Abstract: We introduce ReplaceAnything3D model RAM3D, a novel method for 3D object replacement in 3D scenes based on users' text description. Given multi-view images of a scene, a text prompt describing the object to replace, and another describing the new object, our Erase-and-Replace approach can effectively swap objects in 3D scenes with newly generated content while maintaining 3D consistency across multiple viewpoints. We demonstrate the versatility of RAM3D by applying it to various realistic 3D scene types, showcasing results of modified objects that blend in seamlessly with the scene without impacting its overall integrity.

Main paper: 
 introduction: 
 The explosion of new media platforms and display devices has sparked a surge in demand for highquality 3D content, and thus an increasing need for efficient tools for generating and editing them. While there has been significant progress in 3D reconstruction and generation, 3D scene editing remain a less-studied area. In this work, we focus on the task of replacing or adding new 3D objects to an existing 3D scene using only input language prompts from a user. Compared to other 3D scene editing methods such as relighting or stylization, this task involves intricate local edits to seamlessly integrate new objects into the scene without disrupting its overall coherence. This goes beyond just generating realistic visuals and demands a nuanced understanding of both the global scene context and the interaction between the newly added object and the rest of the scene.Naively using text-to-3D methods to generate 3D objects and manually adding them to a scene can be a tedious process. More importantly, it completely ignores the interaction between objects' Original "Michelangelo statue of dog browsing cellphone" "Buddha statue" "A dragon" "A bunny on top of a stack of pancakes" "An octopus" "A pile of cookies" "Cthulhu" "Orchid pot" "Halloween pumpkin"Figure 1 : Our method enables prompt-driven object replacement for a variety of realistic 3D scenes.appearance and the rest of the scene, such as lighting and shadows. Here, we instead formulate the task of object replacement as a 3D scene inpainting problem. Specifically, our goal is to seamlessly fill in the region occupied by the old object with a new object, making it indistinguishable from the rest of the 3D scene. To avoid manual object placement and blending, we adopt a powerful text-guided image inpainting model, enabling 3D object replacement based solely on input text prompts.In this work, we present the ReplaceAnything3D model (RAM3D), a text-guided method for object replacement in 3D scenes using an Erase-and-Replace strategy. RAM3D takes multiview images of a static scene as input, along with text prompts specifying which object to erase and what should replace it. Our approach comprises four key steps: 1) We use a text-to-mask model with a text prompt to detect and segment the object to be erased from the input images. 2) We use a text-guided 3D inpainting technique to fill in the background region obscured by the removed object in a multi-view consistent manner. 3) Next, we use a similar text-guided 3D inpainting technique to generate a new 3D object corresponding to the input text description, which is seamlessly composited onto the background in all views. We thus obtain multiview consistent images of the edited 3D scene. 4) Finally, these updated dataset images are used to reconstruct the modified 3D scene, enabling novel view synthesis. Instead of relying on generic image editing methods such as Instruct-Pix2Pix [1] , we adopt HiFA [2] , a state-of-the-art distillation approach, to distill a pretrained text-to-image-inpainting model into a 3D scene representation. This allows us to freely add or entirely remove detailed 3D objects from scenes, a significant challenge for methods like Instruct-Pix2Pix [1] and its derivatives such as Instruct-NeRF2NeRF [3] due to their reliance on a limited dataset of images and editing instructions. By integrating a text-guided image inpainting model with a compositional scene structure that distinguishes the object of interest from the rest of the scene, ReplaceAnything3D can seamlessly generate edited 3D scenes with new objects harmoniously integrated into their environment. In summary, our contributions are:• We introduce a method for text-guided object replacement in 3D scenes that removes the need for manual 3D modelling.• We propose a multi-stage approach that supports object replacement or even object addition in 3D scenes in high-fidelity.• We present 3D consistent results on multiple scene types (human avatar, forward-facing and 360 • scenes), and challenging edit prompts requiring detailed texture synthesis.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: We provide our goal for the paper, a short description of the method and overview of the results included in the paper.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 Diffusion model for text-guided image editing Diffusion models trained on extensive text-image datasets have demonstrated remarkable results, showcasing their ability to capture intricate semantics from text prompts [4, 5, 6] . As a result, these models provide strong priors for various text-guided image editing tasks [7, 8, 9, 1, 10] . In particular, methods for text-guided image inpainting [11, 12] enable local image editing by replacing masked regions with new content that seamlessly blends with the rest of the image, allowing for object removal, replacement, and addition. These methods are direct 2D counterparts to our approach for 3D scenes, where each view can be treated as an image inpainting task. However, 3D scenes present additional challenges, such as the requirement for multi-view consistency and memory constraints due to the underlying 3D representations. In this work, RAM3D addresses these challenges by combining a pre-trained image inpainting model with compositional 3D scene representations.Text-to-3D synthesis With the remarkable success of text-to-image diffusion models, text-to-3D synthesis has garnered increasing attention. Most work in this area focuses on distilling pre-trained text-to-image models into 3D models, starting with the seminal works Dreamfusion [13] and Score Jacobian Chaining (SJC) [14] . Subsequent research has explored various methods to enhance the quality of synthesized objects [15, 2, 16, 17] and disentangle geometry and appearance [18] . Instead of relying solely on pre-trained text-to-image models, recent work has utilized large-scale 3D datasets such as Objaverse [19] to improve the quality of 3D synthesis from text or single images [20, 21] .Here, we move beyond text-to-3D synthesis by incorporating both text prompts and the surrounding scene information as inputs. This approach introduces additional complexities, such as ensuring the appearance of the 3D object harmoniously blends with the rest of the scene and accurately modeling object-object interactions like occlusion and shadows. Combining HiFA [2] , a text-to-3D distillation approach, with a text-to-image-inpainting model, RAM3D aims to create more realistic and coherent 3D scenes that seamlessly integrate the synthesized 3D objects (Figure 1 ).Many existing 3D editing methods focus on editing an individual object's appearance or geometry [22, 23, 24, 25, 26] . For scene-level editing, recent works primarily address object removal tasks for forward-facing NeRF scenes [27, 28, 29] . Instruct-NeRF2NeRF [3] and similar followup works [30, 31, 32] offer a comprehensive approach to both appearance editing and object addition, leveraging InstructPix2Pix [1] to update the scene dataset. However, as they modify the entire scene, they struggle to synthesise objects with complex geometrical texture patterns, and completely fail to remove objects from scenes. Blended-Nerf [26] and DreamEditor [33] allow localized object editing but do not support object removal. One closely related work is [28] , which can remove and replace objects using one single image reference from the user. However, since this method relies only on a single inpainted image, it cannot handle regions with large occlusions across different views, and thus is only applied on forward-facing scenes. Another closely related work is RePaint-NeRF [34] , which similarly allows text-guided scene editing on a masked region, but uses SDS loss to update a pretrained NeRF towards the text content. In contrast, RAM3D adopts an Erase-and-Replace approach for localized scene editing, instead of modifying the existing geometry or appearance of the scene's content, leading to superior qualitative results when compared to RePaint-NeRF.Unlike NeRF editing methods, the recently proposed Gaussian Editor [35] , uses 3D Gaussian Splats [36] as its underlying representation. Exploiting the explicit nature of this representation, this method enables localised edits by selectively updating targeted Gaussians, using guidance from InstructPix2Pix [1] . Like our work, it also supports adding new objects to scenes. Similarly to [28] , a single scene image is inpainted in 2D, providing a reference image of the new object. A pre-existing image-to-3D object model [37] then generates a 3D object from the segmented reference image, in isolation from the original scene. The coarse object mesh is transformed back into a 3D Gaussian representation and integrated into the scene through a laborious modelling process where the object is is manually placed into the scene. More importantly, the new object is not guaranteed to integrate seamlessly into the surrounding scene, which results in visible artifacts and quality issues when the method is applied to 360 • scenes. In contrast, RAM3D performs distillation using the inpainting network as a diffusion prior, which leads to harmonious blending between the new object and its surroundings, even when applied to 360 • scenes with challenging edit prompts. Inpainting LDMInpainting LDM 3 MethodQuestion: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We dedicate a section to discuss current limitations with examples for failure cases, and discuss future direction to rectify this (see Appendix section J).Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
preliminary: 
 Distilling diffusion models Dreamfusion [13] proposes a technique called Score Distillation Sampling (SDS) to compute gradients from a 2D pre-trained text-to-image diffusion model, to optimize the parameters of 3D neural radiance fields (NeRF). Recently, HiFA [2] significantly improves the quality of text-to-3D object generation by introducing an alternative loss formulation to SDS that can be computed explicitly for a Latent Diffusion Model (LDM). Let θ scene be the parameters of a implicit 3D scene, y is a text prompt, ϵ ϕ (z t , t, y) be the pre-trained LDM model with encoder E and decoder D, θ scene can be optimized using:EQUATIONwhere z = E(x) is the latent vector by encoding a rendered image x of θ scene from a camera viewpoint from the training dataset, ẑ is the estimate of latent vector z by the denoiser ϵ ϕ , and x = D(ẑ) is a recovered image obtained through the decoder D of the LDM. Note that for brevity, we incorporate coefficients related to timesteps t to w(t).Here we deviate from the text-to-3D synthesis task where the generated object is solely conditioned on a text prompt. Instead, we consider a collection of scene views as additional inputs for the synthesized object. To achieve this, we utilize the HiFA distillation loss function and timestepannealing strategy, in conjunction with an open-source text-to-image inpainting LDM. This LDM ϵ ψ (z t , t, y, m) requires not only a text prompt y, but also a binary mask m indicating the area to be filled in.The input to RAM3D consists of a collection of n images I i , corresponding camera viewpoints v i and a text prompt y erase describing the object the user wishes to replace. Using this text prompt we can obtain masks m i corresponding to every image and camera viewpoint using a pretrained text-to-mask model LangSAM [38] . Note that these masks are not necessarily multi-view consistent. We additionally provide a text prompt y replace describing a new object to replace the old object.Our goal is to modify the masked object in every image in the dataset to match the text prompt y replace , in a 3D-consistent manner. We can then reconstruct the edited scene using any choice of 3D representations such as NeRF [39] or Gaussian Splats [36] to obtain renderings of the edited 3D scene from novel viewpoints.Figure 2 illustrates the overall pipeline of our Erase and Replace framework. Instead of modifying existing objects' geometry and appearance to match the target text descriptions like other methods [3, 33] , we adopt an Erase-and-Replace approach. Firstly, for the Erase stage, we remove the masked objects completely and inpaint the occluded region in the background, using a neural field θ bg . Secondly, for the Replace stage, we generate new objects using a neural field θ fg , compositing them so that they blend in with the inpainted background scene. Finally, we create a new training set using the edited images and camera poses from the original scene, and reconstruct the modified 3D scene using any choice of 3D representations for novel view synthesis.To enable text-guided object replacement in 3D scenes, we distill an open-source text-to-image inpainting LDM using the HiFA loss function (Equation 1) and timestep annealing strategy [2] . Note that LDM distillation using 3D Gaussian Splats is still challenging to optimise, leading to blurry results and thus requiring further refinement process in recent text-to-3D-object works [40, 41] . We therefore opt to use a NeRF-based representation instead, for RAM3D's Erase and Replace stages (Sections 3.3, 3.4). To circumvent the memory constraints and slow training speed inherent to NeRF's implicit representations, we propose a Bubble-NeRF representation (see Figure 3 , Left side) which only models the localised part of the scene that is affected by the editing operation, instead of the whole scene.In the Erase stage, we aim to remove the object described by y erase from the scene and inpaint the occluded background region in a multi-view consistent manner. To do so, we optimise RAM3D parameters θ bg which implicitly represent the inpainted background scene. Note that the Erase stage only needs to be performed once to remove the desired object, after which the Replace stage (Section 3.4) can be used to generate different objects or even add new objects to the scene, as demonstrated in the Results section. As a pre-processing step, we use LangSAM [38] with text prompt y erase to obtain a mask m i for each image in the dataset. We then dilate each m i to obtain halo regions h i around the original input mask (see Figure 3 , Left side).At each training step, we sample image I i , camera v i , mask m i , and halo region h i for a random i ∈ {1..n}, providing them as inputs to RAM3D to compute training losses (Figure 2 , Left side) (we henceforth drop the subscript i for clarity). RAM3D volume renders the implicit 3D representation θ bg over rays emitted from camera viewpoint v which pass through the visible pixels in m and h (the Bubble-NeRF region). The RGB values of the remaining pixels on the exterior of the Bubble-NeRF are sampled from I (see Figure 3 , Left side). These rendered and sampled pixel rgb-values are arranged into a 2D array, and form RAM3D's inpainting result for the given view, x bg . Following the HiFA distillation objective (see Section 3.1), we use the frozen LDM's E to encode x bg to obtain z bg , add noise, denoise with ϵ ψ to obtain ẑbg , and decode with D to obtain xbg . We condition ϵ ψ with I, m and the empty prompt, since we do not aim to inpaint new content at this stage.We now use these inputs to compute L HiFA (see Equation 1). We next compute L recon and L vgg on h (see Figure 3 ), guiding the distilled θ bg towards an accurate reconstruction of the background.EQUATIONEQUATIONThis step is critical to ensuring that RAM3D inpaints the background correctly (see Figure 7 ). Following [42] , we compute depth regularisation L depth , leveraging the geometric prior from a pretrained depth estimator [43] . In summary, the total loss for the Erase stage is:L Erase = L HiFA + λ recon L recon + λ vgg L vgg + λ depth L depth (4)In the Replace stage, we aim to add a new object described by y replace into the inpainted scene. To do so, we optimise the foreground neural field θ fg to render x fg , which is then composited with x bg to form x. Unlike θ bg in the Erase stage, θ fg does not seek to reconstruct the background scene, but instead only the LDM-inpainted object which is located on the interior of m. Therefore in the Replace stage, RAM3D does not consider the halo rays which intersect h, but only those intersecting m (Figure 3 , Right side). These rendered pixels are arranged in the masked region into a 2D array to give the foreground image x fg , whilst the unmasked pixels are assigned an RGB value of 0. The accumulated densities are similarly arranged into a foreground alpha map A, whilst the unmasked pixels are assigned an alpha value of 0. We now composite the foreground x fg with the background x bg using alpha blending: Using the composited result x, we compute L HiFA as before, but now condition ϵ ψ with the prompt y replace , which specifies the new object for inpainting. As we no longer require the other losses, we set λ recon , λ vgg , λ depth to 0.EQUATIONSince the Erase stage already provides us with a good background, in this stage, θ fg only needs to represent the foreground object. To encourage foreground/background disentanglement, on every k-th training step, we substitute x bg with a constant-value RGB tensor, with randomly sampled RGB intensity. This guides the distillation of θ fg to only include density for the new object; a critical augmentation to avoid spurious floaters over the background (see Figure 7 , Left side).Once the inpainted background and objects have been generated inside the Bubble-NeRF region (Figure 3 ) during the Erase and Replace stages, we composite the Bubble-NeRF renderings onto all original scene images. We finally obtain a full 3D representation of the edited scene by applying an off-the-shelf scene reconstruction method such as NeRF or Gaussian splats [36] . Right: Qualitative comparison with Blended-NeRF [12] for object replacement. Our method generates results with higher quality and capture more realistic lighting and details.Ours "Tartan jacket" "Checkered jacket" "Plushie mushroom" "Pineapple" "Chess piece" "Haiiwan shirt"Figure 5 : Qualitative comparison with Gaussian Editor [35] . We show results for 3 challenging edit prompts on the GARDEN scene (top 2 rows) and FACE scene (bottom 2 rows). In the GARDEN scene, our method generates more realistic objects which are better integrated with the surrounding scene.In the FACE scene, our method generates more detailed texture patterns and geometry which are better aligned with the edit prompts.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [No] Justification: We base our method on open-source pre-trained models without proposing new datasets. Comprehensive implementation and necessary infrastructure details for our method are provided in the supplementary material.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
results: 
 We conduct experiments on real 3D scenes varying in complexity: forward-facing scenes, 360 • scenes and human avatar. For forward-facing scenes, we show results for the STATUE and RED-NET scene from SPIn-NeRF dataset [27] , as well as the FERN scene from NeRF [39] . For 360 • scene, we show results from the GARDEN scene from Mip-NeRF 360 • [44] . For the avatar result, we use the FACE dataset from Instruct-NeRF2NeRF [3] . On each dataset, we train RAM3D with a variety of y replace , generating a diverse set of edited 3D scenes (Figure 1 ). Please refer to the supplemental video for more qualitative results for object replacement using personalized content.Note that RAM3D performs localised scene editing for 3D object replacement, conditioned by an edit region mask and replacement object description prompt, as shown in Figure 2 . For fair applesto-apples comparison, we mostly compare with other state-of-the-art localised editing methods (see Figure 4 , Figure 5 , Table 1 ). This scope contrasts with the separate but related track of global scene editing methods, which can modify an entire scene based on instruction-style prompts, but do not support object replacement or removal [3, 30, 31, 32] . We nevertheless provide qualitative and quantitative comparison with global-editing methods in the Appendix (Sections F, G).Figure 4 shows qualitative comparison with two methods for NeRF-based 3D object replacement; Blended-NeRF [12] and the method by [28] . In Figure 5 we compare our method with a state-of-theart 3D Gaussian Splat-based scene editing framework [35] , which supports object removal, addition and replacement. In the Appendix (Section F), we additionally compare with DreamEditor [33] , Repaint-NeRF [34] , generic scene editing method Instruct-NeRF2NeRF [3] and various similar InstructPix2Pix-based [1] followup works [30, 31, 32] .Compared to method by [28] on the left side of Figure 4 , RAM3D achieves comparable object replacement results while handling more complex lighting effects such as shadows between the foreground and background objects. Note that the method by [28] only works with forward facing scenes, and thus cannot handle 360 • scenes such as the GARDEN scene like our method. In Figure"White bunny" "A strawberry" "A strawberry" and "A white bunny" "Small saguaro cactus in a clay pot"Figure 6 : Given user-defined masks, ReplaceAnything3D can add completely new objects that blend in with the rest of the scene. Furthermore, due to its compositional structure, RAM3D can add multiple objects to 3D scenes while maintaining realistic appearance, lighting, and multi-view consistency (bottom right).4 right side, we note that RAM3D generates more realistic and detailed objects that blend in better with the rest of the scene. Meanwhile, Blended-NeRF only focuses on synthesizing completely new objects without taking the surrounding scenes into consideration. The synthesized object therefore looks saturated and outlandish compared to the rest of the scene. Moreover, due to the memory constraint of CLIP [45] and NeRF, Blended-NeRF only works with image resolutions 2-times smaller than ours (1008×756 vs. 504×378).Figure 5 shows qualitative comparison with GaussianEditor [35] , a state-of-the-art scene-editing framework which supports object deletion, addition, and general editing capabilities. Applying it to the GARDEN scene, we first use the delete functionality to remove the vase from the table, followed by the addition functionality to insert the new object. Note that this method generates the new object in isolation, using a prior image-to-3D method [37] , resulting in visible artifacts on the surface of the table where the new object interacts with the surrounding scene. Furthermore, since this method is not guaranteed to place the new object correctly, its position in the scene requires post-hoc manual adjustment -see Appendix (Section H) for further details.Applying GaussianEditor to the FACE scene, we use the general editing functionality. The explicit Gaussian Splat formulation allows edits to be localised to the relevant Gaussians (corresponding to the man's torso), which is selected using a user interface. The editing process is then guided towards the desired prompt using 2D guidance from InstructPix2Pix [1] , and we observe similar limitations to other InstructPix2Pix-based scene-editing methods [3, 32, 31, 30] (see Appendix F). In particular, we observe that GaussianEditor struggles to synthesise detailed texture patterns on the man's torso, unlike RAM3D. Furthermore, we notice that the geometry of the man's clothes appears unchanged in the GaussianEditor results, whereas our method successfully generates geometrical details such as the shirt collar and jacket lapels, matching the edit prompt in each case.Adding multiple objects In addition to replacing objects in the scene, our method can add new objects based on users' input masks. Figure 6 demonstrates that completely new objects with realistic lighting and shadows can be generated and composited to the current 3D scene. Notably, as shown in Figure 6 -bottom right, our method can add more than one object to the same scene while maintaining realistic scene appearance and multi-view consistency.3D scene editing is a highly subjective task. Thus, we mainly show qualitative results and comparisons, and refer readers to the supplemental video for additional results. However, we follow [3] and report CLIP Text-Image Direction Similarity, which measures the alignment of the performed object replacement with the input text description. Additionally, we also quantitatively measure temporal consistency by calculating the Warping Error, following [46, 47, 48] . Specifically, (1) we use RAFT [49] to calculate the optical flow of test videos, with each frame being a rendered novel view of the original scene, (2) warp the corresponding frames from the modified scene according to it, and (3) measure the warping error. We compare RAM3D quantitatively with 2 state-of-the-art NeRF-based and Gaussian-Splatting-based methods ( [35] , [50] ) for the object-replacement task on three datasets.In Table 1 , we show that RAM3D achieves better overall prompt alignment (highest CLIP Text-Image Direction Similarity) than existing works, and the best temporal consistency (lowest warping error) across all methods and datasets. Interestingly, although Blended-NeRF directly optimizes for CLIP-similarity between the generated objects and target text prompts, it still achieves a lower score than our method. Furthermore, we observe that our model's superior appearance synthesis quality (when compared to GaussianEditor in Figure 5 ) is also reflected in the quantitative results in Table 1 .We conduct a series of ablation studies to demonstrate the effectiveness of our method and training strategy. In Figure 7 Right side, we show the benefits of our compositional foreground/background structure and background augmentation training strategy. Specifically, we train a version of RAM3D using a monolithic NeRF to model both the background and the new object (combining θ bg and θ fg ).In other words, this model is trained to edit the scene in one single stage, instead of separate Erase and Replace stages. We observe lower quality background reconstruction in this case, as evident from the blurry hedge behind the corgi's head in Figure 7c .We also demonstrate the advantage of using random background augmentation in separating the foreground object from the background (see Section 3.4). Without this augmentation, the model is unable to accurately separate the foreground and background alpha maps, resulting in a blurry background and floaters that are particularly noticeable when viewed on video (Figure 7d ). In contrast, our full composited model trained with background augmentation successfully separates the foreground and background, producing sharp results for the entire scene (Figure 7e ).In Figure 7 Left side, we show the importance of the Halo region supervision for the Erase stage.Without it, our model lacks important nearby spatial information, and thus cannot successfully generate the background scene. 
conclusion: 
 We introduce RAM3D, a text-guided 3D object replacement method for 3D scenes, offering a potential editing tool for VR/MR, gaming, and film production. With an Erase-and-Replace approach, RAM3D can effectively replace objects with significantly different contents that blends seamlessly with the original 3D scene. Our method can also add new objects while maintaining realistic appearance and multi-view consistency. We demonstrate the effectiveness of RAM3D in various realistic 3D scenes (including human avatar, forward-facing and 360 • scenes), and superior synthesis quality compared to current state-of-the-art NeRF and Gaussian Splatting based methods.For future work, our Bubble-NeRF method could be extended to other representations such as 3D Gaussian splats [36] , similar to DreamGaussian [40] . Other interesting future directions include disentangling geometry and appearance to enable more fine-grained control for scene editing, addressing multi-face problems using prompt-debiasing methods [51] or models that are pre-trained on multiview datasets [52, 53] , and developing amortized models for faster editing, similar to [54] .In Figure 8 we show Replace-stage results using simple SDS loss (with random diffusion timestep sampling), adding a corgi in the statue scene. The results show slightly less detailed texture synthesis as the RGB-space HiFA loss component is removed. Nevertheless, note that our pipeline still synthesises the new object in the correct position, which is orthogonal to synthesis quality. We additionally provide quantitative ablation results in Table 5 , which show that using simple SDS loss leads to only a slight drop in CLIP Text-Image Direction Similarity, whilst removing background augmentation or training our model in a single stage also lead to worse performance.In Figure 9 we show the performance of our model on the Erase-stage task, with Halo supervision and Depth loss components removed, and compare these variants to our full model. We run a state-of-the-art image segmentation model [55] on each model variant's results, to detect the main segmentation mask inside a bounding box around the statue region. As shown in Figure 9 , (purple region), the original statue segmentation mask is still detected for our No Halo and No Depth loss model variants. However, for our full model, the statue mask is not detected; SAM instead correctly detects the hedge region behind the statue. This implies that our full method has successfully removed the statue, and realistically filled in the background, including the disoccluded region of the hedge.For a quantitative comparison of these model variants, see Table 4 .No Halo No Depth loss Full CLIP Sim. ↑ 0.020 0.087 0.145 Table 4 : Erase-stage ablation results. We report CLIP Text-Image Direction Similarity scores for all model variants, using the prompt "A white plinth in a park, in front of a path", on the STATUE scene. Note that our full model performs best.  
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: We provide comprehensive details on data preprocessing, implementation, and necessary infrastructure for our method in the Appendix section I.6.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: Our method is applied for each scene instead of large datasets of multiple scenes.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We include an Impact Statements section in our Appendix (section B).Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: We do not release models or datasets.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We utilize MIT-licensed datasets and open-source models, all of which have been widely used in prior publications.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: our paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [Yes] Justification: We provided the questions that were asked in our User Study in Appendix Sec. C, whilst the relevant videos are shown in the video on our project page. Respondents participated voluntarily and were not compensated.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [No] Justification: Our User Study (Appendix Sec. C) did not incur risks to study participants and does not require specific approval.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: In addition to text prompts, RAM3D enables users to replace or add their own assets to 3D scenes. This is achieved by first fine-tuning a pre-trained inpainting diffusion model with multiple images of a target object using Dreambooth [56] . The resulting fine-tuned model is then integrated into RAM3D to enable object replacement in 3D scenes. As shown in Figure 10 , after the fine-tuning stage, RAM3D can effectively replace or add objects to new 3D scenes.Replace Add Figure 10 : Users can personalize a 3D scene by replacing or adding their own assets using a fine-tuned RAM3D. We achieve this by first fine-tuning an inpainting diffusion model with five images of the target object (left), and then combining it with RAM3D to perform object replacement and addition with custom content.In Figure 14 , we compare our approach with a naive 2D baseline where each image is processed individually. For each image in the training set (first column), we mask out the foreground object (statue) and replace it with a new object (corgi) using a pre-trained text-to-image inpainting model (Figure 14 -second column). We then train a NeRF scene with these modified images. As shown in Figure 14 -third column, this results in a corrupted, inconsistent foreground object since each view is very different from each other, in contrast to our multi-view consistent result.In Figure 11 , we compare RAM3D with Instruct-NeRF2NeRF [3] , a general scene-editing framework. We note that this method struggles to handle cases where the new object is significantly different from the original one (for example, replace a vase and flowers with a pineapple or a chess piece in Figure 11 second and third column). RAM3D can also generate objects with challenging texture such as the tartan pattern on the jacket, while Instruct-NeRF2NeRF struggles due to its naive reliance on inconsistent dataset updates from Instruct-Pix2Pix [1] . More importantly, Instruct-NeRF2NeRF significantly changes the global structure of the scene even when the edit is supposed to be local (for example, only change the clothing). This is undesirable, especially for human face editing where the rest of the identity should be maintained after the edits. Finally, RAM3D is capable of removing objects from the scene completely, whilst Instruct-NeRF2NeRF cannot (Figure 11 column 1 ).In Figure 12 we additionally show FACE dataset results obtained using 3 followup works to Instruct-NeRF-to-NeRF [31, 32, 30] . These methods also rely on InstructPix2Pix [1] to provide a 2D image editing prior, and could not successfully edit the scenes in our experiments, resulting in poor image quality as shown in Figure 12 . We note that all of these methods are limited by the paired training dataset of text editing instructions and images before/after the edit that was used to train InstructPix2Pix. GaussianEditor, when used in general editing mode, also faces the same limitation as it relies on InstructPix2Pix to update the selected gaussians, as demonstrated in the FACE dataset 5 ). In contrast to these methods, RAM3D distills a state-of-the-art image-inpainting diffusion model to provide a 2D generative prior, which has been finetuned for inpainting real images, and consequently provides better 2D guidance for the object replacement scene editing task.In Figure 13 Left side, we present qualitative comparison with RepaintNeRF [34] , for object replacement using the NeRF Fortress scene. Note that the Erase-and-Replace approach and Bubble-NeRF formulation adopted by RAM3D can generate higher fidelity novel content, with a more complete shape for the apple. In contrast, note that RePaint-NeRF generates an incomplete shape which appears to be partially cut off by the boundary of the fortress in the input scene. We hypothesise that this artifact is a consequence of the training procedure used by RepaintNeRF, in which the entire input scene is modified directly with diffusion guidance.In Figure 13 Right side, we demonstrate our model's competitive performance with DreamEditor [10] . It is important to note that DreamEditor has limitations in terms of handling unbounded scenes due to its reliance on object-centric NeuS [57] . Additionally, since DreamEditor relies on mesh representations, it is not clear how this method will perform on editing operations such as object removal, or operations that require significant changes in mesh topologies. We use the official, publicly available GaussianEditor code implementation to obtain results on the GARDEN and FACE scene shown in Figure 5 . When obtaining results for the GARDEN scene (top row) we first use the Delete functionality to remove the vase from the table. However, we found that we were unable to replicate the quality of the object removal results reported in the GaussianEditor paper [35] . In particular, we found that the mask-dilation and hole-fixing refinement phase of the Delete method resulted in a cloudy semi-transparent artifact in the area occupied by the removed gaussians (we show the before/after results for this refinement stage in Figure 15 ). Therefore, in order to compare with the strongest possible baseline, we opted to use the Deletion result from before this refinement step (shown on the left side of Figure 15 ).We subsequently use the GaussianEditor Add functionality to incorporate the new object into the scene. We found that the new object was initially placed far from the table by GaussianEditor, which provides a user interface for repositioning the object along the z-values of a camera ray passing through the centre of the reference image. We show the before/after results of this manual depth adjustment in Figure 16 , when adding the pineapple object onto the table.We use an Instant-NGP [58] based implicit function for the RAM3D NeRF architecture, which includes a memory-and speed-efficient Multiresolution Hash Encoding layer, together with a 3-layer Figure 14 : Qualitative comparisons between our method RAM3D (last column) with a naive 2D baseline method, which produces view-inconsistent results (third column). This is because each input image is processed independently and thus vary widely from each other (second column).Delete result before re nement Figure 15 : Results obtained using GaussianEditor to remove the vase object from the GARDEN scene. Note that GaussianEditor's proposed mask dilation and hole-fixing refinement stage causes a cloudy artifact to appear above the table, where the vase was originally placed. We therefore use the result on the left, prior to refinement, when adding new objects to replace the vase. MLP, hidden dimension 64, which maps ray-sample position to RGB and density. We do not use view-direction as a feature. NeRF rendering code is adapted from the nerf-pytorch repo [59] .We use a 2-stage Erase-and-Replace training schedule for the STATUE, RED-NET and GARDEN scenes. For the FERN scene, we use user-drawn object masks which cover a region of empty space in the scene, therefore object removal is redundant. In this case, we perform object addition by providing the input scene-images as background compositing images to RAM3D. For the FACE scene, the torso region in which we aim to generate new content overlaps entirely with the mask region -we therefore train a monolithic RAM3D to inpaint new content in the mask in a single training stage. We obtain inpainting masks for object removal by passing dataset images to an off-the-shelf text-tomask model [38] , which we prompt with 1-word descriptions of the foreground objects to remove. The prompts used are: STATUE scene: "statue", GARDEN scene: "Centrepiece", FACE scene: "Clothes", RED-NET scene: "Bag". We dilate the predicted masks to make sure they fully cover the object.For the Erase stage, we compute nearby pixels to the exterior of the inpainting mask, and use them as the Halo region (see Figure 3 ). We apply reconstruction supervision on the Halo region as detailed in I.5. For the object-addition experiments in the FERN scene, we create user-annotated masks in a consistent position across the dataset images, covering an unoccupied area of the scene.In Figure 17 , we show an example of the input masks, which can be obtained through either using LangSAM and a text prompt to describe the object to be replaced or by manual user placement to add more objects to the 3D scene.Mask for replacing "tree trunk" using LangSAM (e.g: "Add strawberry")User-defined mask for object adding (e.g: "Add bunny") The LDM denoising U-net takes input images of size 512×512. In contrast, RAM3D outputs are of equal resolution to the input scene images, which can be non-square. To ensure size compatibility, we need to crop and resize the RAM3D outputs to 512×512 before passing them to the denoiser (see Figure 2 ). For the STATUE, FACE and GARDEN scenes, we resize all images to height 512 and take a centre-crop of 512×512, which always contains the entire object mask region. For the RED-NET scene, the object mask is positioned on the left side of the images; we therefore select the left-most 512 pixels for cropping.For the FERN scene, input images are annotated with small user-provided masks. We find that the previous approach provides too small of a mask region to the LDM's denoiser. In this case, we train RAM3D using the original dataset downsampled by a factor of 2 to a resolution of 2016×1512, and select a rectangular crop around the object mask. We compute the tightest rectangular crop which covers the mask region, and then double the crop-region height and width whilst keeping its centre intact. Finally, we increase the crop region height and width to the max of the height and width, obtaining a square crop containing the inpainting mask region. We apply this crop to the output of RAM3D and then interpolate to 512×512 before proceeding as before.During the Erase training stage, we find it necessary to backpropagate reconstruction loss gradients through pixels close to the inpainting mask (See Ablation Figure 7 ), to successfully reconstruct the background scene. We therefore additionally render pixels inside the Halo region (Section I.3, Figure 3 ), and compute reconstruction loss L recon and perceptual loss L vgg on these pixels, together with the corresponding region on the input images. Note that the masked image content does not fall inside the Halo region in the input images -therefore L recon and L vgg only provide supervision on the scene backgrounds. For the reconstruction loss, we use mean-squared error computed between the input image and RAM3D's RGB output. For perceptual loss, we use mean-squared error between the features computed at layer 8 of a pre-trained and frozen VGG-16 network [60] . In both cases, the loss is calculated on the exterior of the inpainting mask and backpropagated through the Halo region. During the Replace training phase, following [2] , we apply L BGT+ loss between our rendered output x, and the LDM denoised output x, obtaining gradients to update our NeRF-scene weights towards the LDM image prior (see HiFA Loss in Figure 2 , eqn 11 [2] ). No other loss functions are applied during this phase, thus loss gradients are only backpropogated to the pixels on the interior of the inpainting masks. For memory and speed efficiency, RAM3D only renders pixels which lie inside the inpainting mask at this stage (Figure 3 ), and otherwise samples RGB values directly from the corresponding input image.Finally, following [42] , we apply depth regularisation using the negative Pearson correlation coefficient between our NeRF-rendered depth map, and a monocular depth estimate computed on the LDM-denoised RGB output. The depth estimate is obtained using an off-the-shelf model [43] . This loss is backpropogated through all rendered pixels; i.e the union of the inpainting mask and Halo region shown in Figure 3 . We do not apply this regularisation during the Replace stage. In summary, the total loss function for the Replace stage is:EQUATIONwith loss weights as follows: λ recon = 3, λ vgg = 0.03, λ depth = 3.We use the Adam optimiser [61] with a learning rate of 1e-3, which is scaled up by 10 for the Instant-NGP hash encoding parameters.Each dataset is downsampled to have a shortest image side-length (height) equal to 512, so that square crops provided to the LDM inpainter include the full height of the input image. The FERN scene is an exception, in which we sample a smaller 512 image crop within dataset images with a downsample factor of 2.Following [13, 2] , we find that classifier-free guidance (CFG) is critical to obtaining effective gradients for distillation sampling from the LDM denoiser. We use a CFG scale of 30 during the Replace stage, and 7.5 during the Erase stage. We also adopt the HiFA noise-level schedule, with t_min = 0.2, t_max = 0.98, and use stochasticity hyperparameter η = 0. In the definition of L BGT+ loss (see eqn 11 in [2] ), we follow HiFA and choose a λ rgb value of 0.1. We render the RAM3D radiance function using a coarse-to-fine sampling strategy, with 128 coarse and 128 fine raysamples. During the Replace training stage, we swap the composited background image with a randomly chosen plain training is a set of multiview images which match the input scene images on the visible region, and contain inpainted content on the interior of the masked region which is consistent across views. To obtain novel views, we train NeRFs with standard novel view synthesis methods using RAM3D edited images and the original scene cameras poses as training datasets. We use nerf-pytorch [59] for LLFF scenes (STATUE, FERN, RED-NET SCENES), the nerf-studio [62] Nerfacto model for the FACE scene and Gaussian Splatting [36] for the GARDEN scene.As noted in Appendix 4.1, RAM3D is capable of adding multiple objects to scenes by using multiple prompts. However, we note in Figure 18 a ) that adding multiple objects using a single edit prompt can sometimes result in an implausible composition which lacks a coherent spatial relationship. Furthermore, as ReplaceAnything3D is based on text-to-image model distillation techniques, our method suffers from similar artifacts to these methods, such as the Janus multi-face problem, as shown in Figure 18 b ).Our Erase-and-Replace approach might remove important structural information from original objects, so it is not suitable for editing objects' properties such as appearance or geometry (for example, turning the statue gold, see Figure 19 a ).). Our method generates a Bubble NeRF from scratch inside the edit region, meaning that the structural information regarding the original statue is lost. Consequently, our model generates an entirely new statue, with gold appearance.We note that our method works best when Erasing and Replacing objects of similar size. A significant size mismatch between Erased and Replaced objects may lead to degraded synthesis quality. This can be observed in Figure 19 b ), (replacing the statue with a bus).NeurIPS Paper Checklist 


Decision: The paper introduce ReplaceAnything3D (RAM3D), a method that allows users to replace an object in 3D scene (formed from multi-view images) by using text to describe the object to replace, and the object to add.  To accomplish this, the authors proposes a three-stage method that 1) detect and segment the object to be removed (using LangSAM), and then use 3D inpainting (LDM with HiFA) to 2) fill in the background and 3) add the requested object.  The paper includes some comparisons on selected scenes and prompts that shows the proposed method is better at replacing objects than alternative baselines. 

The submission received divergent ratings with two reviewers (MnXs, MzLe) leaning toward accept, and two reviewers (NhGj, ZFSz) voting to reject.

The main concerns expressed by reviewers are:
1. Fairness of comparison against prior work
   - Missing ablation of HiFA vs key contributions of this work [NhGj]
   - Lack of systematic comparisons with same scene and same prompt [ZFSz]
   - Potential cherry picking of results (based on comparison to results show in other papers originally) [ZFSz]    
2. Limited novelty with a straight-forward approach for a very specific targeted text-based editing [ZFSz]
3. Some details need to be clarified and discussion added, including:
  - clarification of specific parts of HiFA that was used [NhGj]
  - more motivation justifying the specific focus on replacement and discussion when comparing against baselines about how the baselines are more general instruction-based editing while the proposed method is designed for object replacement. 
  - more discussion of the limitations regarding object size [MzLe,MnXs] and replacement when only style changed desired [NhGj,MnXs,ZFSz]
  - replace Fig. 10 with results without `--downscale-factor 2` so that generation from prior work is artifact-free [NhGj]

From the AC's perspective, the main concerns are in the fairness of comparison to prior work (1).  During the rebuttal period, the authors provided additional ablations and an user study comparing the proposed methods to baselines.  The AC believes the user study provides a reasonably systematic comparison of prior work. Reviewer NhGj also did a deep dive to investigate the artifacts shown in Fig. 10 for prior work, and determined that while the figure should be replaced to more accurately reflect the performance of prior work, the proposed method does indeed produce more accurate replacements than prior work.  Thus the AC believes concern (1) to have been mostly addressed by the author response.

For the other concerns, (3) can be addressed by improvements to the exposition, and for novelty (2) most reviewers did not have an issue with the novelty of the approach. In addition, the fact that the approach is straight-forward and intuitive does not lessen the contribution.  All reviewers also recognized the approach is very targeted, but it can be practically useful.  

Due to the above, the AC believes that by incorporating the additional results and discussions from the author response period, the work would be appropriate to be presented at NeurIPS.

The authors should make sure to include the following:
- User study (from rebuttal).  It would be ideal if the the user study can be made to be more systematic (for instance, BlendedNERF and Reference-Guided Inpainting also compared against for GARDEN and FACE scenes).  
- Inclusion of ablation of HiFA loss vs SDS (from rebuttal)
- Include quantitive results for ablations (from rebuttal)
- Replace Fig. 10 with results without `--downscale-factor 2` so that generation from prior work is artifact-free
- Add clarification of specific parts of HiFA that was used
- Add discussion (in experiments) of differences between the more general text-based editing that is supported by methods that are compared against vs the very specific editing supported by the proposed method
- Improved discussion of limitations, including limitations on object size, replacement when there is a style change while preserving the object shape, etc.
- Other minor improvements to writing and presentation
  - Improve notation and typesetting of math equations  (see comments from NhGj)
  - Try to clean up the references (make sure paper venues are included, consistent names for conferences, remove extra information [51], paper titles are properly capitalized).   Alphabetic ordering is recommended.

Reviews: review: paper_summary: paper_summary: This paper studies instruction-guided scene editing. They introduce ReplaceAnything3D (RAM3D, not sure why using this abbreviation), a two-stage method to first erase the object to be replaced and in-paint the background in the scene, and then generate the replace-to object and compose it to the scene after stage one. Some relatively good visualization results are shown in the paper.
strength_weakness: strength_weakness: - The method highly utilized HiFA's loss and design in their pipeline, but no ablation study was provided about HiFA to indicate whether it is HiFA or the proposed method contributes most to the good results. More specifically, I would like to understand:
    - If adding HiFA's loss and design with Instruct-NeRF2NeRF or GaussianEditor, can they achieve comparable results as the proposed method?
    - If the proposed method removes HiFA's loss and uses simple loss - so it is natural that the results are worse than the full model, will it still get better results in GaussianEditor?
- Some of the editing results are not good enough. 
    - For clothes editing, there is always an obvious cutting edge (sometimes looks like the collar of a sweater) on the neck part.
- The replacement task is easier than standard instruction-guided editing, which explicitly indicates the replace-from and replace-to objects instead of requiring the diffusion model to infer. 
    - This makes some comparisons unfair, especially the ones in the model that struggle to infer the replace-from region, e.g., Fig.9. 
    - On the other hand, this setting also disables some editing, like more implicit instructions like "turn him into a bald" (i.e., remove hair).
    - Besides, removing an object and regenerating one prevents the model from keeping sufficient identities from the replace-from objects, which is not acceptable in some cases, e.g., color editing of an object may result in highly different objects.
    - The authors need to argue why this setting is reasonable and more/comparably reasonable compared with traditional instruction-guided editing tasks.
- The paper has multiple presentation deflects, especially the math formulas. 
    - For example, the subscripts should not be italicized. $\lambda_{HiFA}$ is used in the paper, which means $\lambda_{H\times i\times F\times A}$ instead of $\lambda_{\mathrm{HiFA}}$.
    - The same variable should be referenced consistently, but the paper contains $y_{\mathrm{replace}}$ at L127/129/203 and $y_{replace}$ (which means $y_{r\times e\times p\times l\times a\times c\times e}$) at L174/185.
questions: questions: Please see "Weaknesses".
limitations: limitations: The authors have included broader impacts and limitations in the paper.
paper_summary: paper_summary: The authors introduce a method for replacing 3D objects within 3D scenes based on text descriptions. They do this by using in-painting approaches for the set of  images that are used for down-stream novel-view synthesis.
strength_weakness: strength_weakness: One big assumption from the way the method works is that the object you're attempting to replace is LARGER than the object you're replacing it with. SAM cannot necessarily anticipate the size difference between the two objects, and so during the erase stage the best it will be able to do is mask out the small area, in which case the area that you can inpaint over is very small. For instance, replacing the statue in Figure 2 with something larger, like a bus, will not work.

The opposite problem may also be true -- when the object being replaced is TOO much larger than the size of the object that is being added, the replace stage can end up over-sizing the object being added (perhaps encouraged by your foreground/background disentanglement, mentioned in Line 188 ). This can be seen in figure 3, where the cookies look (good but) all too gigantic in the scene.

Additionally, the Halo technique assumes that longer range interactions (further apart spatially within the image) between objects reflections don't exist, though this would easily not be true for reflective surfaces.

A table/quantitative results for Ablation studies would be more convincing. I'm especially curious about the impact of the depth reconstruction loss.

As you say in line 245, 3D scene editing is highly subjective -- in which case, I think a user study might make your claims stronger.
questions: questions: Will the trade-offs of using BubbleNerf representation harm the final stage of training a nerf according to multi-view images synthesized by BubbleNerfs? I'm assuming that's the reason why you want to use Nerfs for your final representation, and not the BubbleNerf representations you use for the erase and replace stages. Do you trade speed/expressive power for quality when you use BubbleNerfs?

How were the language prompts chosen for objects to add?
limitations: limitations: Yes.
paper_summary: paper_summary: This paper introduces a novel method that utilizes the Erase-and-Replace strategy for text-guided object replacement in 3D scenes. Given a collection of multi-view images, camera viewpoints, text prompts to describe the objects to be replaced and to be added, this method first optimizes the background scene with the original object erased, and then optimizes the foreground object to be added. Specifically, in the erase stage, it uses the HiFA distillation technique and optimizes a localized NeRF region, which covers the mask region of the original object and the surrounding nearby pixels. In the replace stage, it optimizes the foreground NeRF within the mask of the original object while alternatively substituting the background with constant RGB tensor during the distillation. The proposed approach enables localized 3D editing and obtains more realistic and detailed edited results.
strength_weakness: strength_weakness: The limitation of the proposed approach is also obvious, as described in the limitation section. Is is only suitable for the remove, add, replace editing. The added objects should be within the pre-specified mask region. And the training is time-consuming.
questions: questions: Questions:
(1) Figure 5 shows that the GaussianEditor makes less geometric modification compared to the proposed method. Is the main reason the Gaussian or NeRF representation?
(2) Is the proposed method able to replace the original object with a larger one? What if specifying a mask much larger than the original object?
(3) I’m curious about the property modification mentioned in the limitation. What if erasing the object with the original property and replacing it with the object with new property? 
(4) It’s better to clarify the meanings of \theta_{bg} and \theta_{fg} in Section 3.2. Line 150 says “optimise RAM3D parameters \theta_{bg}”. It makes me assume that \theta_{bg} is all the parameters of RAM3D at this moment.
limitations: limitations: The paper includes a discussion about the limitations.
paper_summary: paper_summary: This paper introduces a method that can add a novel generated or customized object into a 3D reconstructed scene. To enable this, it proposes a Erase-and-Replace strategy. The first step is to remove an object queried by a text prompt: it segment out the target object using existing language based model and fine-tune existing NeRF model based on the inpainted images. The second step is to replace the object where it fine-tune existing NeRF model using the generated or customized images. The model compared with existing instruction based gaussian splatting and NeRF methods.
strength_weakness: strength_weakness: ]The main weakness of this paper is coming from the lack of novelty.]
- The scope of the previous works highly within the previous works, and it actually does not push and advance well the boundary in the field of the research area of the 3D editing field. More specifically, it does not address any challenge which has not been addressed yet. Since the challenge is blurry, the contributions are highly suppressed.
- The proposed method is quite obvious and too straightforward. The impression from this paper, this paper is just to combine existing 3D inpainting and replacement methods. Most components inherit from previous works (or more advanced LLM components) where this paper optimize the engineering parts of the framework (e.g., effectively combine the previous modules, and better and marginally handling the text prompt, data, and mask inputs). Therefore, it also leads to the impression of the product engineering documents not research paper. 
- Finally, the problem itself is not that novel and the proposed method does not even open extra-functionality in the field of 3D editing, which further suppresses the novelty of this paper.

[Potential unfairness]
- The way how existing methods use the prompt is different from the way how the authors used. Previous works have been using the prompts in a way of instruction style, e.g., turn the face to the hulk.
- To be more fair way, it would be nice if the authors can show the results from the same data with the same prompt as shown in the previous works. Since quantifying the performance of the 3D generative works is indeed challenging, this could be one of the ideal and most fair way for apple-to-apple comparison.

[Marginal quality improvement]
- Overall, the quality improvement from the previous works is not that promising. Based on the visual results, it is quite difficult to say which one is showing the better results. Based on the prediction results from the “original papers” (not this paper), it is hard to tell which ones are better. It also leads to the impression that the results shown in the paper is highly cherry-picked.
questions: questions: Based on the current Erase-and-Replace pipeline, it is not fully clear how the authors perform only style changes, e.g., the results in Figure 5-(bottom row). During the inpainting time, if the model fully removes the objects that belong to the dedicated categories in the text prompt, there should be some artifacts such as shape distortion. However, it seems the shape of the man’s torso is perfectly preserved. In such case, how could the shape be perfectly preserved while only changing the style?
limitations: limitations: This paper discussed about the limitations.

Comments: title: Follow-Up Of Baseline Reproducibility Concerns
Follow-Up Of Baseline Reproducibility Concerns
Response to new concerns
Response to follow-up questions
Follow-up questions
comment: I sincerely thank the authors for providing more context about the command settings. The authors do not need to worry about being penalized for not using the actual best commands any longer, either it is a mislead from official instructions or an unintentional misuse of the authors.
Dear reviewer NhGj,

We are very grateful for your thorough investigation into these baselines! 

> “I do not know why the authors added `--downscale-factor 2` instead of using default arguments”

We would like to clarify that **using `–downscale-factor 2` is the default argument for the given resolution.** Please note that **the official codebases specifically suggest this.** To quote from the EN2N README.md:

> "**Important** Please note that training the NeRF on images with resolution larger than 512 will likely cause InstructPix2Pix to throw OOM errors. Moreover, it seems InstructPix2Pix performs significantly worse on images at higher resolution. We suggest training with a resolution that is around 512 (max dimension), so add the following tag to the end of both your nerfacto and in2n training command: `nerfstudio-data --downscale-factor {2,4,6,8}` to the end of your ns-train commands."

Regarding ViCA-NeRF, the README section “Other tips for hyper-parameters” links to the InstructNeRF2NeRF repo which contains the exact same quote as above. Furthermore, on the closed github issues  “Problem of results - not the same as presented in the paper #1” and  “ I cannot reproduce the results :) #3”  which discuss quality issues when training on the exact same Face dataset, the owner of the code repository commented, recommending adding `nerfstudio-data --downscale-factor 2` to the training command.

As reviewer NhGj pointed out, **“The original resolution of the 'face' scene is 994x738”**. As you can see from the above extracts from the official codebases, we simply followed the official documentation to the letter, when adding `nerfstudio-data --downscale-factor 2` to the training command. In other words, **reviewer NhGj has uncovered that the official documentation for reproducing results from EN2N and ViCA-NeRF is erroneous.** We hope that we will not be penalised for precisely following the given instructions for the baselines. Therefore, we believe that this issue should not be regarded as “an unintentional misuse of the baseline codes” but rather as caused by “erroneous documentation provided with baseline codes”.

We are extremely grateful to reviewer NhGj for discovering this issue. Therefore, we will be happy to update Fig.10 in our revision, as the reviewer asked. Now that the baseline methods are generating reasonable results, the advantages offered by our proposed method are made more clear. As reviewer NhGj remarked, the updated baseline results are still not able to synthesise the correct ‘checkered’ texture pattern (unlike ours), but instead only produce “scaly textures”. 

> “I would highly suggest the authors add (timestep annealing) information in the revision, to improve the reproducibility ….  I would kindly suggest that the authors mention fewer "adopt HiFA," "combining HiFA," and "utilize HiFA" in their revision … Instead, they may directly mention the name of the components, like "HiFA's loss," and "timestep annealing strategy inspired by HiFA" to focus more on the components they adopted from HiFA, instead of HiFA itself.”

We are very grateful for these suggestions, which we are very happy to adopt in the revision.
Dear reviewer NhGj,

Thank you for your quick response and providing the images.  I agree that Figure 10 should be updated with results from the original settings as they are artifact free.  

R-MzLe, R-MnXs, R-ZFSz: a quick summary
- There was some initial concerns regarding the artifacts in Fig. 10 which is now resolved
- The artifacts as been identified as due to the use of `--downscale-factor 2` when running the baselines
- We will request the authors update Fig. 10 with the results from the original settings, which while better than the current ones in Fig. 10, and is not really "checkered" while the proposed method produced large checkered pattern. 
- Please check out the images provided by R-NhGj to judge for yourself

We will discuss the paper more during the AC-reviewer discussion phase, but there is no need to be concerned about the comparison against baselines in Fig. 10.
Dear AC,

Thank you for following up and acknowledging my efforts. Your understanding is correct - the degrading of baseline results from an unintentional misuse (in my opinion) of `--downscale-factor 2`, while the optimal command for a fair comparison is the default one without it.

I provide the results through this Anonymous Github Repo: https://anonymous.4open.science/r/neurips24_15922_review_NhGj-5684. As I do not have much time to render them as videos (also, some checkpoints are not saved due to crashing), I provide the results as Wandb validation images, where the left part is the original view and the right part is the edited view. These views are randomly sampled from all the validation views, so they might be different in different runs, but they should be sufficient for us to see the quality and artifact patterns. I apologize for the inconvenience here.

Here, I would like to provide some results in images from the same viewpoint. Unfortunately, OpenReview does not support image markdowns `![]()`, and I can only provide the links. All the results except for the "Tartan jacket" one are "checkered jacket".

- EN2N
   - [EN2N Default](https://anonymous.4open.science/r/neurips24_15922_review_NhGj-5684/en2n_default/3.png)
   - [EN2N w/ Downscaling Run1](https://anonymous.4open.science/r/neurips24_15922_review_NhGj-5684/en2n_downscale2_run1/2.png)
   - [EN2N w/ Downscaling Run2](https://anonymous.4open.science/r/neurips24_15922_review_NhGj-5684/en2n_downscale2_run2/3.png)
   - [EN2N w/ Downscaling "Tartan Jacket"](https://anonymous.4open.science/r/neurips24_15922_review_NhGj-5684/en2n_downscale2_Tartan/2.png)
- ViCA-NeRF
   - [ViCA Default](https://anonymous.4open.science/r/neurips24_15922_review_NhGj-5684/vica_default/2.png)
   - [ViCA w/ Downscaling](https://anonymous.4open.science/r/neurips24_15922_review_NhGj-5684/vica_downscale2/5.png)

It can be seen that the pattern roughly matches the ones shown in Fig.10, and there is noticible difference between w/ and w/o downscaling. Even though the results w/ default (i.e. optimal) commands are still not better than the proposed method's (proposed method produces checkered patterns in common sense, but baselines can only produce scaly textures), it is still necessary for the authors to update their Fig.10 in their revision.

I hope these results could help the AC, other reviewers, and authors to understand the difference between the two settings. Thanks.
Dear reviewer NhGj,

Thank you for your careful review of the paper and your efforts to reproduce the baselines.   Based on my understanding:
- The extra `--downscale-factor 2` causes the results from EN2N and ViCA-NeRF to be degraded (similar to Fig. 10)
- Without the extra `--downscale-factor 2`, the results are more reasonable.  Based on the description, it seems that the quality of the checkered jackets is not as good as the proposed methods.
- There may have been some rationale for the authors to use the `--downscale-factor 2`, but a more fair comparison to the baseline would be the one without.

To help me and the other reviewer's understanding, please post the images through an anonymized link.
As a follow-up, I have completed the reproduction with the additional `nerfstudio-data --downscale-factor 2` argument. Namely, I am using the following commands:
```
# Nerfacto
ns-train nerfacto --data <face dataset> nerfstudio-data --downscale-factor 2
# EN2N
ns-train en2n --data <face dataset> --load-dir <nerfacto checkpoint> --pipeline.prompt "give him a checkered jacket" --pipeline.guidance-scale 7.5 --pipeline.image-guidance-scale 1.5 nerfstudio-data --downscale-factor 2
# ViCA-Ne
ns-train vica --data <face dataset> --load-dir <nerfacto checkpoint> --pipeline.prompt "give him a checkered jacket" nerfstudio-data --downscale-factor 2
```

The results are as follows in the language description:
- EN2N: I tried this experiment twice, but both crashed in the middle. Before crashing:
    - Run 1: Though the scaly textures are also added to the jacket, the whole image was highly reddish.
    - Run 2: The result is like the one without `nerfstudio-data --downscale-factor 2`, but more blurred, and the scaly textures are less obvious.
    - Though both these runs' results do not perfectly match those in Fig.10, due the the randomness and unstability of the model itself,  it is not impossible that it may produce some results like those in Fig.10.
    - For academic rigor, I also reproduced the results of "Tartan jacket", and the results is quite similar to Fig.10.
- ViCA-NeRF: Very blurred results with flocs everywhere. These artifacts match the results as shown in Fig.10.

From the results above, I will not regard the results as unreliable any longer. I do not know why the authors added `--downscale-factor 2` instead of using default arguments, but I tend to believe that there is a valid reason for the authors to do that, given the following facts:
- The original resolution of the "face" scene is 994x738.
- Given that IP2P was trained on a low resolution of 256x256, it looks reasonable to downscale the images to 497x369, although IP2P claimed to generalize well at 512x512 resolution (appendix A.3) and can actually work at 768 resolution (Fig.6).
- On the other hand, the pipeline of EN2N and ViCA-NeRF may also scale the image before inputting to IP2P, this might be the key to make it work for default settings, but may also result in failures if downscale in advance.

Considering all these points, I would regard this issue as an unintentional misuse of the baseline codes due to an unintentional misunderstanding. In this case, I would not penalize the authors for not providing the baselines at their optimal settings in the original paper. However, the authors should still update Fig.10 in their revision for a fair comparison.

Again, I appreciate for the detailed response from the authors. Finally, Considering all the strengths and weaknesses of this paper, I will not further decrease the ratings. At this point, I tend to keep my original rating, but I am also open to increase the rating, and may do so after engaging in the reviewer-AC discussion.
I sincerely thank the authors for the detailed follow-up discussion.

***
> On the contrary, SDS loss and IDU are quite different, as discussed in the InstructNeRF2NeRF paper, Ablation Study section. 

Again, I appreciate the authors for their detailed responses and arguments on this topic. Though I still do not fully agree with some arguments on the difference between SDS and IN2N - I also know and understand that the *practical* or *empirical* performance might be (even significantly) different, but I do not think there is a large difference in *theoretical* or *semantic* (like using different coefficients for L2 regularization may lead to success or failure in training but they do not have much difference) - I will not ask for more discussion on this point, the authors are correct that these are a little out of scope.

I also appreciate the authors for mentioning which parts of HiFA to adopt in their model: _"We therefore adopt both HiFA loss (equation 4 in HiFA) and timestep annealing (equation 5 in HiFA). Nevertheless, we validate this choice with the new ablation studies in the rebuttal PDF. "_ However, I would like to point out that the point of "timestep annealing" is *never* mentioned in the main paper, while this, may contribute a lot to the generation quality. I understand that this method is widely used and is already a part of the model design, and I will not ask for an ablation study on this part. I would highly suggest the authors to add this information in the revision, to improve the reproducibility.

Also, for the revision, I would kindly suggest that the authors mention fewer "adopt HiFA," "combining HiFA," and "utilize HiFA" in their revision - these may mislead the audiences (like me) that "the pipeline is highly utilizing the whole HiFA." Instead, they may directly mention the name of the components, like "HiFA's loss," and "timestep annealing strategy inspired by HiFA" to focus more on the components they adopted from HiFA, instead of HiFA itself.

***
As for the baseline issues, in fact, these days **I have also tried to reproduce the baseline results** with all default settings:

```
# train a nerfacto model to start, with default settings
ns-train nerfacto --data <face dataset> 
# EN2N: default arguments of EN2N provided in their Github repo
ns-train en2n --data <face dataset> --load-dir <nerfacto checkpoint> --pipeline.prompt "give him a checkered jacket" --pipeline.guidance-scale 7.5 --pipeline.image-guidance-scale 1.5
# ViCA-NeRF: default arguments of ViCA-NeRF
ns-train vica --data <face dataset> --load-dir <nerfacto checkpoint> --pipeline.prompt "give him a checkered jacket"
```
And I obtained the results. As the policy about whether reviewers can post external links is not clear, I will instead describe the contents of images here. If AC permits, I will also post some images through an Anonymized Github's repo.
- EN2N: the appearance is like IN2N, but slightly more blurred. The jacket was modified to add some scaly texture (or checkered texture with relatively small grids). I did not observe the fully unreasonable results like in Fig.10.
- ViCA-NeRF: the appearance is that the jacket was changed to a suit with a scaly texture, while the collar part of the person has some blurred, checkered-like patterns. I did not observe any artifacts similar to those in Fig.10, either.

I noticed that the authors use a different command with an additional `nerfstudio-data --downscale-factor 2`. Perhaps these are some settings inherited from IN2N or some other baselines? I am not sure whether this is the reason why the results are worse and unreasonable. 

In this case, I will not regard the credibility level of the results in this paper as unreliable at this point. However, I will try to reproduce the results with additional `nerfstudio-data --downscale-factor 2` during reviewer-AC discussions, and then decide whether and how to adjust the rating. 

I would also humbly encourage and request other reviewers that, if you have time and spare GPUs, you may also try to reproduce these results to check them from your side.

Finally, again, I sincerely thank the authors for the detailed follow-up feedback. I understand that my previous comment pointed out an issue not observed in the original review and was not sent in a very timely manner, and I sincerely apologize for this. I would also suggest the authors revise their Fig.10 with the results without `nerfstudio-data --downscale-factor 2` in their revision, to achieve a fair comparison with the baselines - I agree that the proposed method is still better than the baselines in this version, and this will not change any rating about the quality.
We are not sure which part of Reviewer NhGj's followup comment the reviewer is referring to. If it's regarding the baseline comparisons, we have now provided the training commands and details, and point to issues posted in the ViCA-NeRF github repo indicating that other users have encountered similar problems reproducing the published results. Regarding SDS/HiFA loss vs iterative dataset updates, this stems from misconceptions regarding the InstructNeRF2NeRF IDU method, which is explained in the ablation section of InstructNeRF2NeRF.

Finally, we would like to reiterate that these are all **global** scene editing methods, which we have made our best effort to reproduce, and include for completeness in the appendix (including EN2N which is not published work). **The main focus of our paper is localised scene editing,** which is why we report comparisons with the state-of-the-art localised methods in our main paper in Table 1, and Figures 4 and 5. Please let us know if there are any further clarifications we can provide to help the reviewer with their assessment!
Many thanks for your useful feedback, which has helped us to improve our paper further!
Regarding EN2N, please note that the codebase linked by the official project site is marked as “Unofficial” - but is the only available codebase as far as we know. (Note that we are unable to post links here). We use the default command provided:

`ns-train {en2n,en2n-small} --data <face dataset>--load-dir <nerfacto checkpoint> --pipeline.prompt {"give him a checkered jacket"} --pipeline.guidance-scale 7.5 --pipeline.image-guidance-scale 1.5 nerfstudio-data --downscale-factor 2`

We tried both variants en2n and en2n-small (en2n-small uses a half-precision IP2P model). We obtain better results using en2n-small (shown in figure 10), while en2n completely collapsed.

Similarly for ViCA-NeRF, we follow the default command provided in the official repo:

`ns-train vica --data <face dataset> --load-dir <nerfacto checkpoint> --pipeline.prompt {"give him a tartan jacket"}   nerfstudio-data --downscale-factor 2`

Regarding artefacts in the VICA-NeRF results, note that multiple users have encountered very similar issues on this dataset, as seen on the closed issues tab of the github repo. For example, please see the issues named **‘Problem of results - not the same as presented in the paper’, ‘I cannot reproduce the results :)’** and **‘Incorrect result’**. The reason why users have struggled to replicate the results is still an open question, best addressed by the authors. Nevertheless, we would be delighted if the reviewers/AC can run the code and help us (and other users!) identify the problem, and happy to update this in the camera-ready version.

Finally, **note that these global editing results are not part of our main comparison. However, we made our best effort to reproduce them (including EN2N which is not published work), and include them for completeness in the appendix.** Please be aware of the main paper and project scope (localised editing), and let us know any questions that you have on those comparisons.

 >  “In fact, the formulas (2)(3) in HiFA's paper have actually proven the equivalence between SDS loss and iterative DU's supervising rendered image with edited image … Given that iterative DU's loss is between the original image and the generated image (in multiple denoising steps), they are actually equivalent.”

On the contrary, **SDS loss and IDU are quite different, as discussed in the InstructNeRF2NeRF paper, Ablation Study section**. The key difference is that SDS loss (and HiFA loss) requires rendering full images to obtain encoded latents. In contrast, IN2N training randomly samples rays across all viewpoints, which precludes VAE encoding. Therefore, incorporating HiFA into the IN2N training step would require modifying IN2N to render full images. However, this was already tried in the IN2N ablation section under the heading SDS + InstructPix2Pix. As noted, this approach “results in a 3D scene with more artifacts … We largely attribute this to the fact that the standard SDS samples rays from a small collection of full images, which makes optimization more unreliable than sampling rays randomly across all viewpoints”.

 >  “As we are unable to discuss on new results, I would expect the authors to provide some high-level analysis of how the techniques in HiFA, e.g., formula (4)(5)(6)(7)(8)(9) improve the results. It will be better to provide some of these ablation studies in revision.”

Please note that only equations 4 and 5 in HiFA are relevant to our method. Equations 6 to 9 are concerned with z-variance regularisation which we do not use.  Equation 3 and 4 shows how HiFA loss differs from SDS loss only by an RGB reconstruction term. As noted in the ablation section 5.2 of HiFA, the RGB term “contributes to a more natural appearance and enhanced texture details”. This section also shows that the HiFA timestep annealing scheme yields superior visual quality to standard random timestep-sampling - the reasons were analysed in section 4.1 under “a timestep annealing approach”.

We therefore adopt both HiFA loss (equation 4 in HiFA) and timestep annealing (equation 5 in HiFA). Nevertheless, we validate this choice with the new ablation studies in the rebuttal PDF. In Figure 19a) (and Table 6), we show our model’s output using standard SDS loss and random timestep sampling. As mentioned in our rebuttal above, “the results show slightly less detailed texture synthesis as the RGB-space loss is removed”. Nevertheless, we still obtain reasonable outputs which suggests that these 2 techniques adopted from HiFA are not critical to our model’s performance. We are happy to add these new ablation studies to the camera-ready paper.

Please note that whilst we have validated and adopted HiFAs loss function and timestep annealing as one component of our model, our core contributions are our localised Bubble-NeRF rendering, Erase-and-Replace strategy, and compositional scene representation. We hope the reviewer will take a holistic view of our work and not penalise us for using HiFA.
This addresses my original concerns, but Reviewer NhGj's followup comment is concerning -- please respond to that instead. For the same reason, I've changed my certainty to 3 (but leave the rating unchanged).
I thank the authors for their rebuttal. Most of my concerns are addressed, but more concerns are raised.

***
> “Some of the editing results are not good enough. For clothes editing, there is always an obvious cutting edge on the neck…” We only claim that our model outperforms the existing state-of-the-art on this task. 

I double-checked Fig.10 in the paper after receiving this response, but **more concerns** are raised about the provided results: The artifacts in baseline EN2N and ViCA-NeRF's results are *counter-intuitive*, even *unlikely* to exist in a valid setting of these baselines.

- In Fig.10, EN2N generates high-quality Tartan jacket results but unreasonable results in the "checkered jacket" task. However,
    - In Fig.9, IN2N generates failed but reasonable results of the "checkered jacket" task, at least with a clear face.
    - Given EN2N is an *improvement* of IN2N with *same* diffusion model IP2P, I wonder **why EN2N cannot produce at least the same face-clear results as IN2N**.
- In Fig.10, ViCA-NeRF generates two images with lots of artifacts
    - I observed that the two images have very similarly structured artifacts, e.g., in the right images of both rows in ViCA-NeRF's results, the top-left part of the head has some hair-like artifacts.
    - Given that ViCA-NeRF is mainly a *warp-based* method (based on NeRF-predicted depth) while both editing tasks do *not* change any geometry of the *head* part, I wonder **why similar artifacts appear in different edits** - were these artifacts already existing in the *original scene* input to ViCA-NeRF?

I would like the authors to provide some analysis of these counter-intuitive artifacts. Given the code of EN2N and ViCA-NeRF is publicly available, I would also like the authors to **provide the commands that they used to obtain the results in Fig.10**, e.g., `ns-train en2n ...` or `ns-train vica ...`, so that I, along with other reviewers and AC, can reproduce these results on our side.

I am paying close attention to this newly-noticed concern, since it is related to whether the baselines are compared in a fair setting, e.g., at least with reasonable hyperparameters.

***
> “If adding HiFA’s loss and design with Instruct-NeRF2NeRF or GaussianEditor, can they achieve comparable results?” - this is based on a misunderstanding; there is no SDS loss or other distillation loss in those methods. 

In fact, the formulas (2)(3) in HiFA's paper have actually proven the *equivalence* between SDS loss and iterative DU's supervising rendered image with edited image. In (3) in HiFA's paper, the LHS is SDS loss, and RHS is a weighted term of $\|z - \hat{z}\|$, which is exactly the MSE loss between the latents of the original image and generated image (in one denoising step). Given that iterative DU's loss is between the original image and the generated image (in multiple denoising steps), they are actually equivalent.

As we are unable to discuss on new results, I would expect the authors to provide some high-level analysis of how the techniques in HiFA, e.g., formula (4)(5)(6)(7)(8)(9) improve the results. It will be better to provide some of these ablation studies in revision.
Thanks for the response. The rebuttal has addressed my concerns. I don't have any follow-up questions.
Many thanks for your response! 

1. Yes, in the Replace stage we learn an object-specific Bubble-NeRF, which corresponds to the edit prompt $y_{replace}$ (Section 3.4, main paper). In the Erase stage (Section 3.3, main paper), we learn a scene-specific Bubble-NeRF, which represents the disoccluded background region.

2. We note that we are the first to show multi-object addition results for 3D scenes. However, we recognise that Gaussian Editor’s Add functionality can also in principle be applied multiple times. Nevertheless, as noted in our main paper lines 222 to 229, this method simply generates new objects in isolation using an image-to-3D method, which are manually positioned by the user, (see appendix F). This results in noticeable artifacts at the point of contact between the new object and its surroundings; we expect that these quality issues would be exacerbated in the multiple object case.
Regarding Instruct-NeRF2NeRF and other closely related **global** scene-editing methods, these methods struggle to add entirely new objects into scenes due to their reliance on inconsistent dataset updates, as stated in the limitations section of InstructNeRF2NeRF: “adding entirely new objects to the scene (such as adding a cup on a table) is challenging for various reasons”.
In comparison to existing **localised** NeRF-editing methods, note that our model differs due to its use of a compositional representation (see lines 174 - 183, and eqn 5 in our main paper). As a result of this, the bunny and strawberry results shown in figure 6 (main paper) can be represented by 2 separate Bubble-NeRFs, which are optimised iteratively. The strawberry is added to the scene first, which is then composited to become the new background model. We now apply the same process to add the bunny, ensuring scene harmonisation, whilst fixing the appearance of the strawberry. In contrast, existing localised-editing methods use a single NeRF representation for the whole scene. It is unclear whether this combined representation can be updated to insert a second object nearby to the first one as we do, without impacting the synthesis quality of the first object.

3. Yes, we use the exact same prompts when comparing with baseline methods in figure 4, figure 5 (top 2 rows), and figure 11, directly describing the new object e.g “a strawberry”. However, the Instruct-Pix2Pix based methods (figure 5 3rd row, figures 9 and 10) are conditioned by instruction-style prompts. In this case we use instruction-style prompts such as “give him a checkered jacket” for the baseline method (as described in the original IN2N paper) and “checkered jacket” for our method. Please note that whilst GaussianEditor (Replace mode) uses instruction-style prompts (3rd row, figure 5), the edits are nonetheless localised using a segmentation mask obtained from Segment-Anything, exactly as in our method.

4. We report CLIP-directional similarity to evaluate our model’s prompt adherence, following Instruct-NeRF2NeRF (and also reported by GaussianEditor, VicaNeRF, Collaborative Score Distillation, EfficientNeRF2NeRF, DreamEditor, and BlendedNeRF). Nevertheless, we have additionally conducted a User Study in tables 3 and 4 (rebuttal PDF), to further validate our results. Regarding the synthesis of high frequency details, we note that the “checkered jacket” prompt is a reported failure case in figure 9 of the InstructNeRF2NeRF paper. As noted in the figure caption, InstructPix2Pix updates “fail to consolidate in 3D” in this case, due to view-inconsistent high frequency texture details. Nevertheless, unlike the Instruct-pix2pix-based methods, our model’s distillation-based approach synthesises the correct texture, as shown in figures 5 and 9 of our paper. The challenging texture prompts “Hawaiian Shirt” and “Tartan Jacket” were similarly chosen to highlight our model’s detailed texture synthesis on the bottom row of figure 5.
Dear authors,

While the summary of the paper is not the major source for my decision, I am highly thankful to correct some of my understanding (fine-tuning vs. training from scratch) through the rebuttals! 
I have a follow-up question: 

1) I would like to confirm with the authors, the learned NeRF is still object-specific, right?
2) Regarding the multi-object addition to the scene, why do you think it is not possible to perform adding multiple objects into scene using existing methods (that have shown single object insertion)?
3) Questions about the prompt: I would like to confirm with the authors, do you apply the same prompt used in your method for other baseline methods?
4) In the review, the quality I’ve mentioned is about the “spatial quality” which is highly correlated to the high-frequency perceptual details. Why do you think Clip-matching score can measure the spatial quality of the rendering?

Best,