3D Gaussian Editing with A Single Image

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-19 13:00:10

Abstract: The modeling and manipulation of 3D scenes captured from the real world are pivotal in various applications, attracting growing research interest. While previous works on editing have achieved interesting results through manipulating 3D meshes, they often require accurately reconstructed meshes to perform editing, which limits their application in 3D content generation. To address this gap, we introduce a novel single-image-driven 3D scene editing approach based on 3D Gaussian Splatting, enabling intuitive manipulation via directly editing the content on a 2D image plane. Our method learns to optimize the 3D Gaussians to align with an edited version of the image rendered from a user-specified viewpoint of the original scene. To capture long-range object deformation, we introduce positional loss into the optimization process of 3D Gaussian Splatting and enable gradient propagation through reparameterization. To handle occluded 3D Gaussians when rendering from the specified viewpoint, we build an anchor-based structure and employ a coarse-to-fine optimization strategy capable of handling long-range deformation while maintaining structural stability. Furthermore, we design a novel masking strategy that adaptively identifies non-rigid deformation regions for fine-scale modeling. Extensive experiments show the effectiveness of our method in handling geometric details, long-range, and non-rigid deformation, demonstrating superior editing flexibility and quality compared to previous approaches.

Main paper: 
 introduction: 
 3D scene modeling and editing emerge as crucial tools across diverse applications such as film production, gaming, and augmented/virtual reality, offering exceptional advantages. They enable efficient iteration and rapid prototyping, serving as a canvas for creative expression and effective problem-solving. Due to the high laborious cost of traditional mesh-based scene modeling, implicit neural representations, such as neural radiance fields (NeRF), have recently received increasing attention for their lower cost. Although considerable efforts have been made to address the challenge of establishing interpretable connections between visual effects and implicit representations [8, 37, 51, 53, 55] , NeRF-based methods still face practical limitations in various applications due to their implicit representation's inability to facilitate explicit manipulation. To significantly enhance the efficiency and quality of 3D scene editing, we represent and edit 3D scenes using the emerging 3D Gaussian Splatting (3DGS) method [22] , given its explicit representation and promising reconstruction quality.Prior neural scene editing methods focus on directly manipulating geometry [51, 53, 55] with the assistance of 3D software, such as Blender. These methods follow a pipeline that extracts meshes from the learned radiance fields and utilizes the geometric structure to guide the deformation of the 3D scene. Due to the imperfect reconstructed geometry, these methods struggle to handle non-rigid deformation and fine-grained editing. Other attempts leverage text-to-image models [2, 18] to edit both the geometry and the texture with text prompts, which are extended to support the manipulation of 3DGS scenes [10, 13] . However, they have a clear limitation: users cannot control the details of the objects in the scene. Unlike previous efforts, our approach is inspired by the way humans observe and perceive the 3D world through 2D images. We introduce a single-image-driven approach to editing the 3D scene, aligning with the philosophy of "what you see is what you get. "In a single-image-driven editing task, the user needs to provide an edited image based on a rendering from a specified viewpoint for the 3D scene. In our work, the 3D scene is reconstructed using 3D Gaussian Splatting [22] , and is therefore represented by a set of 3D Gaussian functions. The edited image serves as the target to guide the alignment and manipulation of the 3D content. This process may imply long-range and non-rigid deformation and texture change of 3D objects. We formulate the editing problem as a gradient-based optimization process utilizing 3D Gaussian representation. One trivial solution is to employ photometric losses used in 3DGS [22] to adjust the 3D Gaussians to minimize the difference between the rendered image and the target image. However, these loss functions can only produce intrinsically local derivatives, making them inadequate for handling long-range deformations. Drawing inspiration from DROT [50] , we introduce optimal transport into 3D Gaussian optimization to model long-range correspondence explicitly. We propose a positional loss to drive long-range motions and make the overall process differentiable by reparameterization. To ensure the geometric consistency of the objects after editing, we adopt a novel as-rigid-as-possible (ARAP) regularization scheme that operates on a few anchor points to capture the 3D deformation field in a more efficient way. We also design a coarse-to-fine optimization strategy to enhance the fidelity of the edited results. Furthermore, motivated by the observation that objects in the same scene may have different levels of rigidity, we introduce a novel masking strategy to adaptively identify non-rigid deformation parts and release ARAP regularization, enabling more precise modeling of geometric details for real-world scene editing. The contributions of this paper are summarized as follows:• We propose the first single-image-driven 3D Gaussian scene editing method, realizing "what you see is what you get". • We introduce positional derivatives into 3DGS to capture long-range deformation and enable gradient propagation through reparameterization. • We propose an anchor-based as-rigid-as-possible regularization method and a coarse-to-fine optimization strategy to maintain object-level geometry consistency. • We introduce an adaptive masking strategy to identify nonrigid deformation parts during optimization to ensure more precise modeling. 
related work 2.1 differentiable rendering: 
 Differentiable rendering aims to develop differentiable rendering methods, allowing the computation of derivatives with respect to scene parameters for 3D reconstruction. However, the discontinuities around the object silhouettes pose a significant challenge. To address this issue, [27] introduces an edge sampling method handling Dirac delta functions. SoftRas [29] blurs triangle edges with a signed distance field, aiding gradient back-propagation. [1, 31] approximates boundary terms via reparameterized integrals. The most relevant work to our method is DROT [50] , which integrates Optimal Transport into differentiable rendering, explicitly modeling 3D motions through pixel-level correspondence in screen space.Leveraging the correspondence, DROT extends RGB losses with positional loss, ensuring robust convergence in global and long-range object motions.NeRF [34] and its variants [3-7, 15, 35, 47] , and 3DGS [22] have gained increasing attention due to their superior view synthesis quality. There is a growing demand for human-friendly editing tools to interact with this representation. [28, 53, 55] proposes to extract meshes from a pre-trained NeRF and edit the 3D scene by manipulating the mesh vertices. [20, 26, 37, 51] simplify the geometry structure by cages and employ a cage-based deformation pipeline for 3D editing. [53] proposes to encode the neural implicit field with disentangled geometry and texture codes on mesh vertices. However, these methods are limited by the quality of the reconstructed geometry and struggle to model non-rigid deformation. [8] mitigates this issue by manipulating feature points, but it is laborious to deal with a large number of feature points. On the other hand, [16, 24, 25] decouple color bases and modify them to achieve texture change, while failing to provide fine-grained editing guidance. [48] adopts a teacher-student knowledge distillation scheme to achieve multiview appearance consistency. It only supports rigid transformations like rotation and scaling. With the advancement of text-to-image models [38] [39] [40] [41] , some works [2, 9, 12, 17-19, 33, 42, 43, 46] propose to edit both the geometry and the texture by incorporating CLIP or Diffusion Models to fine-tune NeRF with text instructions. [56] leverages attention maps to locate editing regions. Subsequently, [10, 13, 36, 49] extend semantic editing on NeRFs to 3D Gaussians. However, these methods cannot perform detailed geometry and texture editing. Other works on 3D Gaussian editing [30, 54, 57] involve binding Gaussians to the mesh surface and using the mesh to drive the 3D Gaussians, which are still limited by the quality of the reconstructed meshes. [52] proposes to disentangle geometry and texture for highly efficient texture editing. 
preliminaries: 
 3D Gaussian Splatting (3DGS) [22] is a recent innovation in neural scene representation, which achieves real-time rendering via splatting 3D Gaussians instead of volumetric rendering. Specifically, it represents the scene as a set of 3D anisotropic Gaussians {𝐺 𝑖 } 𝑁 𝑖=1 , each of which is defined by its center position 𝜇 𝑖 ∈ R 3 , 3D covariance matrix Σ 𝑖 ∈ R 3×3 defined in world space, opacity 𝑜 𝑖 ∈ R 1 We address the single-image-driven editing task by an iterative gradient descent process that optimizes the 3D Gaussians to align with the reference image. To model long-range object deformation, we introduce the positional loss. To preserve the geometric consistency of the objects, we propose an anchor-based as-rigid-as-possible regularization scheme, a coarse-to-fine optimization strategy, and an adaptive masking strategy to identify the non-rigid deformation parts.and RGB color 𝑐 𝑖 ∈ R 3 as spherical harmonics (SH). An anisotropic Gaussian filter 𝐺 𝑖 (𝑥) can be written asEQUATIONTo ensure that Σ 𝑖 is always a positive semi-definite matrix during optimization, 3DGS formulates the covariance matrix as Σ 𝑖 = 𝑅 𝑖 𝑆 𝑖 𝑆 𝑇 𝑖 𝑅 𝑇 𝑖 , with a 3D rotation matrix 𝑅 𝑖 ∈ R 3×3 represented by a quaternion 𝑞 𝑖 ∈ R 4 and a scaling matrix 𝑆 𝑖 represented by a 3D vector 𝑠 𝑖 ∈ R 3 .When rendering an image of a specific view, 3DGS employs the EWA splatting method [58] to splat 3D Gaussians 𝐺 𝑖 (𝑥) to 2DGaussians 𝐺 ′ 𝑖 (𝑥) = exp -1 2 (𝑥 -𝜇 ′ 𝑖 ) 𝑇 Σ ′-1 𝑖 (𝑥 -𝜇 ′ 𝑖 ) onto the image plane. 𝜇 ′𝑖 is the center projection on the image plane and the 2D covariance matrix Σ ′ 𝑖 of the splatted 2D Gaussian is given byEQUATIONHere, 𝐽 ∈ R 2×3 is the Jacobian of the affine approximation of the perspective transformation. 𝑊 ∈ R 3×3 represents the viewing transformation. Subsequently, 3DGS employs the alpha-blending method to aggregate the colors of Gaussians that cover the same pixelEQUATIONwhere 𝑁 𝑢 is the number of overlapping Gaussians, and the alpha value 𝛼 𝑖 is formulated as𝛼 𝑖 = 𝑜 𝑖 • 𝐺 ′ 𝑖 (𝑢). 
method: 
 Given the 3D Gaussian-based representation of a static scene and an edited image from a given viewpoint as the reference, the objective is to obtain the optimal 3D Gaussian parameters to align with the reference image. The involved editing operations may include translation, rotation, non-rigid geometric deformation, and texture change. A trivial approach is to use the gradient descent method to optimize the scene parameters, where the derivative of the pixel colors with respect to the 3D Gaussian parameters is given by the pixel-wise 𝐿 1 loss and the structure similarity (SSIM) loss as in the original 3DGS method [22] . However, these losses only generate intrinsically local derivatives, thus becoming less effective for optimizing long-range object translation and deformation and constraining the editing capability.We draw inspiration from the success of DROT [50] in inverse rendering and introduce positional derivatives into the 3D Gaussian editing problem to capture long-range object motion. Leveraging the results of optimal transport (OT), we design a positional loss to explicitly capture long-range motions and guide 3D Gaussians movements. We back-propagate the positional derivatives to scene parameters via reparameterization, as detailed in Section 4.1. Some 3D Gaussians may be occluded when rendering the scene from the given viewpoint. To regularize the geometry of those occluded parts, we propose an anchor-based as-rigid-as-possible (ARAP) regularization method and adopt a coarse-to-fine optimization strategy for better convergence in Sec. 4.2. Furthermore, we design a novel adaptive masking scheme to identify and model non-rigid deformation parts in Sec. 4.3, thereby enabling better modeling of fine-grained details. We summarize the loss functions in Sec. 4.4. Fig. 2 illustrates the overview of our method.To address potential long-range object translation and deformation, our key idea involves capturing the inherent 3D deformation field of the scene during editing. Therefore, we can explicitly guide the deformation and translation of 3D Gaussians during the optimization process. However, the 3D dense correspondence between the initial scene and those of the edited scene is unknown, and thus we cannot directly acquire the motion vector of a 3D point 𝑝. Inspired by DROT [50] , we project the 3D field onto the image plane and leverage optimal transport to estimate 2D motion vectors.Specifically, let 𝑢 ∈ R 2 denotes the 2D position on the image plane, and 𝑐 ∈ R 3 is its color. The vanilla 3DGS optimizes the learnable parameters 𝜃 of 3D Gaussians with the photometric loss L 𝑐 , written asEQUATIONWe extend the photometric loss L 𝑐 with a positional loss L 𝑢 defined on the 2D position 𝑢 to capture the motion of its corresponding local geometry in the inherent 3D space, and reformulate Eq. 4 byEQUATIONHere, L 𝑢 is defined as the difference between the 2D of reaching the state that matches the target image, while 𝜕𝑢/𝜕𝜃 , which can be further decomposed into 𝜕𝑢/𝜕𝑝 • 𝜕𝑝/𝜕𝜃 , enables the differentiable optimization of scene parameters. We treat the pixel centers as samples 𝑢 of the 3D field projected to the 2D image plane and leverage optimal transport to estimate the 2D correspondence. Then we define the transportation cost 𝑤 𝑢,𝑣 from pixel 𝑢 of the rendered image to pixel 𝑣 of the target image as a weighted sum of their color distance and positional distance.EQUATION)where 𝜆 is used to balance the two terms. After obtaining the dense 2D correspondences by optimal transport, the positional loss L 𝑢 is reformulated as the positional distance between pixel 𝑢 and its corresponding target 𝑣. At this point, the derivatives 𝜕L 𝑢 /𝜕𝑢 can be directly deduced from the definition of L 𝑢 , leaving 𝜕𝑢/𝜕𝑝 and 𝜕𝑝/𝜕𝜃 for us to calculate. For the first term, according to Eq. 3, the color of pixel 𝑢 is computed by aggregating the colors of multiple Gaussians that cover the pixel, where the weight coefficient 𝛼 𝑖 𝑖 -1𝑗=1 (1 -𝛼 𝑗 ) measures the contribution of each 2D Gaussian 𝐺 ′𝑖 on the pixel 𝑢. To reduce computational costs, we reuse the intersection point 𝑝 𝑢,𝑖 of a 2D Gaussian 𝐺 ′ 𝑖 and a pixel 𝑢 as a sampling point when modeling the motion field of local geometry. We subsequently calculate the effect of positional derivatives 𝜕L 𝑢 /𝜕𝑢 on the sampling point 𝑝 𝑢,𝑖 byEQUATIONIn the second term, note that the sampling operation that associates the intersection point 𝑝 𝑢,𝑖 and the properties of 2D Gaussian 𝐺 ′ 𝑖 is not differentiable, breaking the back-propagation of gradients. To back-propagate the gradients, we adopt the reparameterization method when drawing samples from the Gaussian distributions. Considering 𝑝 𝑢,𝑖 denotes a sample from a 2D Gaussian 𝐺 ′ 𝑖 with its center 𝜇 ′ 𝑖 and covariance matrix Σ ′ 𝑖 , we can view the sampling operation as a deterministic transformation of parameters 𝜇 ′ 𝑖 , Σ ′ 𝑖 and a random variable 𝜖 ∼ N (0, 𝐼 )EQUATIONHence, the positional derivatives with respect to the center 𝜇 ′ 𝑖 and covariance matrix Σ ′ 𝑖 of 2D Gaussian 𝐺 ′ 𝑖 can be given byEQUATION)where 𝜕𝑝 𝑢,𝑖 /𝜕Σ ′ 1𝑖 can be calculated using the reparameterization in Eq.8. 𝜕Σ′ 1 2 𝑖 /𝜕Σ ′𝑖 can be obtained in closed form. Inspired by 3DGS, which uses a tile-based rasterizer to achieve fast rendering, we propose a tile-based optimal transport matching to achieve high efficiency. Specifically, we split the screen into 16 × 16 tiles, average the colors of pixels within the same tile, and use Sinkhorn [11] divergence to approximate the positional derivatives between the downsampled images. Then, we can update the parameters of Gaussians using Eq. 7 and Eq. 9.To demonstrate the influence of positional loss on long-range object deformation, we visualize the derivatives with respect to the centers of Gaussians and show the results in Fig. 3 . Compared with the photometric losses adopted in 3DGS, our method can accurately determine the gradient descent direction to drive the blade of the Bulldozer downward.In Eq. 7, the positional derivatives vanish as the weight coefficients go zero, thus failing to regularize occluded Gaussians at the reference view. As a result, only the visible parts of the involved objects are affected by the edited image, leading to structural discontinuity and breakdown. Motivated by the observation that the involved editing operations for real-world tasks are often sparse, spatially continuous, and locally rigid, we regularize the motions of 3D Gaussians with a local as-rigid-as-possible (ARAP) assumption as follows.EQUATIONHere, 𝜇 𝑖 denotes the initial position of Gaussian 𝐺 𝑖 . 𝜇 𝑖 and 𝑅 𝑖 present the position and rotation at the current iteration, respectively. K 𝑖 represents the K-nearest neighbors (KNN) of 𝐺 𝑖 and regularization weight 𝜅 𝑖 𝑗 is defined by the relative distance 𝑑 𝑖 𝑗 between two Gaussians, 𝐺 𝑖 and 𝐺 𝑗 , using Radial Basis Function (RBF), formulated as𝜅 𝑖 𝑗 = κ𝑖 𝑗 𝑗 ∈ N 𝑖 κ𝑖 𝑗, where κ𝑖 𝑗 = exp(-𝛾𝑑 2 𝑖 𝑗 )where 𝛾 is a hyper-parameter. However, the ARAP term is defined within a small local region, generating non-zero gradients only when neighboring Gaussians undergo rotation or translation. Consequently, a substantial number of iterations is required to propagate regularization gradients to all occluded parts according to the movements of the neighboring visible parts. This can result in undesired deformation and suboptimal convergence during optimization. To address this issue, we propose to derive sparse anchor points from 3D Gaussians and then leverage them to capture the underlying 3D deformation field, substantially reducing the number of iterations compared to directly using 3D Gaussians.Specifically, we voxelize the 3D scene and then compute the mass centers of 3D Gaussians in each voxel to extract a dense point cloud that covers the scene. We apply farthest point sampling (FPS) on the dense point cloud to downsample 𝑁 𝑎 points and treat them as the initial anchor points {𝑎 𝑗 } 𝑁 𝑎 𝑗=1 , where 𝑎 𝑗 ∈ R 3 denotes the learnable positions of anchor point 𝑗 and 𝑁 𝑎 is the number of anchor points. Each anchor point 𝑎 𝑗 is also associated with a learnable rotation matrix 𝑅 𝑎 𝑗 ∈ R 3×3 represented by a quaternion 𝑟 𝑎 𝑗 ∈ R 4 , which can be locally interpolated to yield a dense deformation field of the Gaussians. Instead of directly optimizing the position and rotation of Gaussians in each iteration, we optimize the parameters of anchor points to model the deformation field. After obtaining the anchor points, we can derive the deformation field of the Gaussians using linear blend skinning (LBS) [44] by locally interpolating the transformations of their neighboring anchor points. More details can be found in our supplementary materials.Coarse Fine Coarse Fine StretchLeveraging a set of sparse anchor points to model the complex deformation space may not faithfully align the scene with the target image. Therefore, we propose a coarse-to-fine optimization strategy to enhance visual quality. In the coarse stage, we utilize an anchorbased structure to optimize the position and rotation of anchor points, effectively capturing long-range changes. Subsequently, in the fine stage, we discard the anchor points and directly optimize both geometric and color parameters of each Gaussian. This approach helps mitigate artifacts such as noise on object boundaries and enhances the modeling of fine texture details. We employ the as-rigid-as-possible loss function on the anchor points during the coarse stage and on the 3D Gaussians during the fine stage.In Eq. 10, ARAP assumes equal rigidity among the neighboring Gaussians of each Gaussian. However, in the real world, different parts of the 3D scene typically exhibit varying degrees of rigidity. Consider a T-pose human model: if we treat the rigidity of its joints and bones equally, undesired bending of bones may occur during deformation. Based on this observation, we incorporate an adaptive rigidity masking mechanism to help identify the extent of non-rigid deformation and mitigate the effects of rigid regularization.Formally, we introduce a learnable mask 𝑚 𝑖 𝑗 ∈ R to each regularization weight 𝜅 𝑖 𝑗 ∈ R and rewrite Eq. 11 asEQUATION)where 𝜎 is the sigmoid function. Notably, the ARAP loss combines both relative rotation and relative distance regularization between Gaussians or anchor points. However, real-world object changes sometimes involve only one of these aspects. For instance, when we lower the blade of a Lego bulldozer, there is a relative rotation between Gaussians near the joint, while their relative geodesic distance remains unchanged. Therefore, we propose a rotation loss and a distance loss to provide explicit supervision on the rotations and positions of Gaussians, respectively. We employ adaptive weights on the regularization terms in non-rigid regions, formulated as:EQUATION)EQUATION)Here, 𝑚 𝑑 𝑖 𝑗 ∈ R and 𝑚 𝑟 𝑖 𝑗 ∈ R denote the learnable weight mask applied on the Gaussians for rotation and distance regularization, respectively.Notably, the optimization process may fall into a trivial solution when the rigidity mask 𝑚 𝑖 𝑗 , 𝑚 𝑑 𝑖 𝑗 , 𝑚 𝑟 𝑖 𝑗 approaches negative infinity. Thus, we periodically reset the weight masks 𝑚 𝑖 𝑗 , 𝑚 𝑑 𝑖 𝑗 , 𝑚 𝑟 𝑖 𝑗 by taking the maximum value between the weight and a hyper-parameter 𝜂.EQUATIONWe visualize the learnable rigidity masks in Fig. 4 , the masks of distance regularization term for the stretched material balls, and the masks of ARAP for the joint of microphone adaptively approach zero after optimization, illustrating the non-rigid deformation part in the scene.In addition to the positional loss L 𝑝 described in Sec. 4.1, we also employ the photometric losses in 3DGS [22] to define the matching loss L match . We use L match to generate gradients from the differences between the rendered image and the target image, written asL match = L p (I, I ref ) + 𝜆||I -I ref || 1 + 𝜆 SSIM L SSIM (I, I ref ) (16)For the learnable masks that adaptively identify the extent of the non-rigid deformation of each part, we apply an L1 regularization term to prevent degradation to zero.EQUATIONThe final loss of the coarse stage can be written asEQUATION)For the fine stage, we additionally regularize the scales of each Gaussian in geometric editing and the colors of each Gaussian in texture editing, written byEQUATION 
experiment: 
 Due to the lack of publicly available benchmarks, we conducted quantitative experiments on the NeRF Synthetic (NS) Dataset [34] and the 3D Biped Cartoon Dataset [32] , both of which contain the ground truth meshes of the reconstructed scenes. Specifically, we chose a viewpoint as the reference view to render an image for each scene in the NS dataset and the MipNeRF360 dataset. We edited them using Adobe Photoshop to construct a reproducible benchmark for reference view alignment evaluation. The 3DBiCar dataset contains 1,500 3D Biped Cartoon Characters, each of which has a T-pose mesh and a posed mesh. We selected 52 characters for evaluation and generated 50 random views of the T-pose mesh for training 3DGS. For testing, we rendered eight surrounding images of the posed mesh, reserving one image for editing, while the others were utilized for novel view synthetic evaluation. We used Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS) as the metrics. To demonstrate the effectiveness of our method on realworld data, we also evaluated it on 5 scenes from the Mip-NeRF 360 Dataset [4] and the Tanks & Temples Dataset [23] for qualitative experiments. We performed single-view video tracking on 2 scenes from the Panoptic Studio Dataset [21] , given that our method can drive the inherent 3D world to temporally consistently align with the frame image once the initial 3D Gaussians model is provided.We conducted two toy experiments to demonstrate the effectiveness and necessity of positional derivatives in handling long-range editing operations. We initialized the first scene containing 3 objects and adjusted the content to align with the reference image. We used the original 3DGS with the ARAP term as the baseline, where the ARAP term maintains the structural stability. The optimization process is shown in Fig. 5 . Leveraging the positional loss, our method can drive objects to their target positions even if there is no overlap between their initial states and target states, such as 1 : Comparisons with other methods on geometric editing. We show the average PSNR/SSIM/LPIPS for reference view alignment on the NS dataset and novel view synthesis on the 3DBiCar dataset. ARAP denotes the as-rigid-as-possible regularization.the microphone and the toy tiger. In contrast, the baseline moves the microphone outside the screen, leading to sub-optimal convergence. We also tested the robustness of our method to non-rigid deformation under different scales. As shown in Fig. 6 , for shortrange deformation, both 3DGS and our method can recover the deformation correctly. However, only our method can capture large deformations well.We compared our method with DROT [50] , which optimizes the position of mesh vertices obtained from NeRF2Mesh [45] , and Deforming-NeRF [51] , which models deformation by manually adjusting the deformable cage extracted from NeRF. As shown in Fig. 7 , our method achieves precise alignment with the reference image, maintaining 3D consistency through the anchor-based structure and the two-stage optimization strategy. However, for DROT, the occluded parts require more iterations to back-propagate gradients from visible parts, leading to structural instability and undesired deformation, such as in the back of the drums. Deforming-NeRF faces limitations due to the resolution of deformable cages, particularly struggling with tasks like stretching objects such as hot dogs.We also demonstrate the results of scene-level editing in Fig. 8 . For scene-level editing, we first select a region of interest and render the image from a specific perspective. Then we can apply various 2D edits and back-propagate to the underlying 3D to align with these edits.Since Deforming-NeRF requires manual adjustment of the cage, which is impractical to test on a large dataset, we quantitatively compare our method with vanilla 3DGS and DROT, and provide the results of reference view alignment and novel view synthesis in Tab. 1. Our method outperforms other methods in both tasks, exhibiting a consistent and significant improvement in metrics.Fig. 9 illustrates hybrid editing cases where we move the black pillar of the LEGO forward, elongate the cockpit, draw an MM logo on the side, stretch the chair horizontally, and draw an ACM logo on the back of it. We optimize the position and rotation of the anchors in the coarse stage to model long-range deformation, while in the fine stage, we refine the parameters of each Gaussian, including both geometry and color parameters. It can be observed that even for complex editing scenarios, our method consistently delivers promising results, demonstrating its robustness. Figure 8 : Geometric editing on Mip-NeRF 360 Dataset. We wavy the edges of the table in the garden and slope the planks of the truck. Our method aligns well with the reference image while maintaining 3D consistency and structural stability.Figure 9 : Hybrid geometry and texture editing. Our method enables simultaneous editing of geometry and textures in a single optimization process.Given the initial 3D Gaussian scene reconstructed from multi-view images, our method enables us to use a single-view video to track the underlying dynamic 3D scene by aligning the rendered image with the subsequent video frames. We only use the coarse stage and optimize the position and rotation of the anchors for fast convergence. We show the reference video frame and two novel views in Fig. 10 . Our method can capture the long-range object motion and maintain both spatial and temporal consistency, producing promising novel view synthesis results.We conducted ablation studies on positional loss, two-stage optimization, adaptive rigidity masking, and explicit supervision of relative rotation (Eq. 13) and distance (Eq. 14). The results are summarized in Table 2 , providing quantitative insights into the effectiveness of each component. Apart from the explicit regularization of relative rotations and distances, the addition of any other components consistently leads to noticeable improvements in target view alignment and novel view synthesis. Moreover, explicit regularization helps maintain structural stability, prevents overfitting to the reference view, and enhances the rendering quality from other perspectives.Fig. 11 presents the optimization results of the coarse stage and fine stage to provide a better understanding of anchor-based deformation and coarse-to-fine optimization. The coarse stage captures long-range deformation during editing and aligns the 3D scene roughly with the reference image, while the fine stage reduces artifacts on the object boundaries and models fine texture details, thereby achieving better alignment.Additionally, we offer a visual comparison of ablating the explicit regularization term of the positions and rotations in Fig. 12 . Notably, explicitly regularizing the relative rotation and position between two neighboring Gaussians can effectively address needlelike problems and reduce structural errors from a new perspective. 
conclusion and limitation: 
 We present a single-image-driven 3D scene editing approach that enables intuitive and detailed manipulation of 3D scenes. We address the problem through an iterative optimization process based on 3D Gaussian Splatting. To handle long-range object translation and deformation, we introduce positional loss into 3D Gaussian scene editing and differentiate the process through reparameterization on 2D Gaussians. To maintain the geometric consistency of the occluded Gaussians in the edited image, we propose an anchorbased As-Rigid-As-Possible (ARAP) regularization and a coarse-tofine optimization strategy. Additionally, we design a novel rigidity masking strategy to achieve precise modeling of fine-grained details. Experiments demonstrate our superior editing flexibility and quality compared to previous approaches.Our method has the following limitations. Since our method leverages optimal transport to calculate the positional loss, it is limited by the accuracy of pixel matching. In areas with weak texture information, where most of the rendered pixels are similar, the Sinkhorn divergence [14] may fail to provide a correct match, thus affecting the optimization of the underlying 3D scene. Additionally, since our method prefers driving 3D Gaussians rather than growing and pruning, it limits the resolution in texture editing. Disentangling geometry and texture, as proposed in [52] , may improve the quality of texture editing.  
 
 Appendix:  


Decision: This paper presents a new single-image-driven 3D scene editing approach based on 3D Gaussian Splatting, enabling intuitive manipulation via directly editing the content on a 2D image plane. Both the method and task look interesting and new to some extent. All reviewers have positive final ratings, are satisfied with the response, and recommend accepting the paper. I agree with their recommendation. Thanks for the authors' effort and rebuttal.

Reviews: review: paper_summary: paper_summary: This paper proposes a method to edit a 3D Gaussian representation of a scene by editing an image rendered from a single viewpoint and constraining the Gaussuans to adapt to the edited image with acoarse to fine optimization scheme with ARAP constraints to capture long range deformations.
To adapt to the presence of non rigid parts a masking strategy is adopted
strength_weakness: strength_weakness: The method is able to transfer reasonably shapes and textures edited in 2D to the 3D Gaussians set allowing to obtain a coherent scene.The method is compared for local editng of NERFs.
The methods seem to correct artifacts that a straightforward optimization of the rendering loss with the edited image create.
limitations: limitations: The description of the adaptive rigidity mask is not clear.
Actually the whole editing pipeline seems limited. The first step in editing both from 2D single views and 3D editors would be to perform a segmentation of the object in the scene allowing to decouple the modified Gaussians from the unchanged ones. This is fundamental and not addressed. It should also be not too difficult, as it is possible to do it manually or automatically on the 3D Gaussian set (or starting on the image and propagating on the Gaussians). 
Actually the editing actions demonstrated would be easier to perform directly on the 3D Gaussians rather than in 2D. Typically in 3D editors, users selects 3D parts to be transformed on a 2D interface and the selection is possibly propagated in 3D. So for a 3D Gaussian editor I would expect that the interactive editing could start with the propagation of the selection of objects to be edited, pissibly smartly segmenting the object corresponding to the 3D part using shape and color information and then introducing editing actions mapped from 2D and 3D. It's hard for me to tink that is better to edit the raster image rendered and the optimize the whole Gaussian set to adapt with rigidity constraints that it is hard to generalize without having a semantic segmentation of the object.
The fact that only limited modifications on mostly segmented parts seems to confirm this.
paper_summary: paper_summary: This paper propose a single-image-driven 3D scene editing approach based on 3D Gaussian Splatting. 3D Gaussian can be deformed through mapping an image deformation back to the 3D scene.
strength_weakness: strength_weakness: It is an interesting and effective way to deform 3D Gaussian by manipulating a 2D viewpoint. It is able to manipulate the 3D model from different 2D views.

The paper is well-written. The experiments demonstrates various applications, such as geometry editing, long range deformation and hybrid editing.
limitations: limitations: The time performance is not discussed. Does the method run in real time? The time performance is critical for an editing task.

In the video, I can see artifacts during editing. The reasons are not clear.

I would like to see some geometric detail editing. There should be some stress tests to illustrate the upper bound for details deformation. This could strengthen the work.
paper_summary: paper_summary: This paper proposes a novel 3D scene editing approach for 3D Gaussian Splatting using a target image as a reference. Specifically, it aligns 3D Gaussians with the target image through gradient-based optimization, introducing optimal transport and a positional loss to capture long-range motions. The paper also designs an anchor-based as-rigid-as-possible regularization method and a coarse-to-fine strategy to ensure geometric consistency. Furthermore, a masking mechanism is introduced to extend non-rigid deformation. Experimental results demonstrate that the proposed method achieves significant quality improvements when geometrically editing 3D scenes.
strength_weakness: strength_weakness: 1.	The paper is well-written and easy to follow, with a good structure.
2.	The proposed method is technically sound and thoroughly demonstrated.
3.	The experiments are sufficient and fully demonstrate the performance of the proposed methodology.
limitations: limitations: 1.	It would be better to showcase the time complexity, as the proposed method introduces several terms to restrict the optimization process.
2.	In section 4.2, it is not clear when to derive sparse anchor points from 3D Gaussians. Furthermore, the operation to “compute the mass centers of 3D Gaussians in each voxel” is not clearly explained.
3.	Considering editing operations like non-rigid geometric deformation, can the proposed method outperform editing methods using text-to-image models? This would be an interesting comparison to explore.
paper_summary: paper_summary: The paper proposes a 3DGS editing method given a single user-specified image, with experiments of geometry editing (especially for long-range deformation), and texture editing.
strength_weakness: strength_weakness: The paper introduces motion vectors with position loss in the image domain, providing a novel approach to GS editing. An ARAP loss regularizes GS to maintain structural consistency, while an adaptive rigidity mask identifies the extent of non-rigid deformation. These losses are designed in an explainable and intuitive manner, resulting in a complete pipeline.
limitations: limitations: 1. **Lacking Related Works in NeRF Editing**: The paper should include more references to related works in NeRF editing. A series of studies [1-6] incorporate UV mapping to enable local and pixel-level editing in NeRF. Including these references would provide a more comprehensive background and context for the current work.

2. **Issues with Single Image-Based Editing**: The single image-based editing method struggles with consistent novel image synthesis, as seen in the supplementary video (geometry editing/3dbicar/39, 2075). This observation undermines the claim that other methods heavily rely on accurate mesh manipulation, which guarantees novel view synthesis. This issue raises concerns about the robustness of the proposed method when editing objects with more complex geometry. However, I still believe this approach is novel and offers significant guidance for the GS editing domain, as using a single image specified by users is an easy and intuitive way to perform editing.

3. **Comparison with Other Methods**: While some GS-editing works are still under review, it would be better to compare the proposed method with them (e.g., GaussianEditor, CVPR'24). If not, comparisons with NeRF-based editing methods such as NeuMesh and Seal-3D, which also use a single image as input, would help evaluate the performance of the proposed method. These comparisons would provide a better understanding of the proposed method's strengths and weaknesses relative to existing approaches.

4. **Time efficiency is also a concern.**.  While GS achieves real-time rendering, it is important to assess whether your editing method allows for interactive manipulation and real-time rendering of novel images after editing. 

5. Can the authors provide visual examples for their ablation studies on the loss terms? While the numbers in Table 2 are metrics of supervised edited image, the metric alone is not convincing. Comparative images from novel and different views would better illustrate the contribution of each loss. Additionally, showing some failure cases due to textureless areas or mismatched pixels would be beneficial. This would help in understanding the limitations and robustness of the proposed method.

Reference:
[1] NeuTex: Neural Texture Mapping for Volumetric Neural Rendering

[2] Neural Parameterization for Dynamic Human Head Editing

[3] Parameterization-driven Neural Implicit Surfaces Editing

[4] Nuvo: Neural UV Mapping for Unruly 3D Representations

[5] Learning an Isometric Surface Parameterization for Texture Unwrapping

[6] Texture-GS: Disentangling the Geometry and Texture for 3D Gaussian Splatting Editing

[7] GaussianEditor: Swift and Controllable 3D Editing with Gaussian Splatting