3DitScene: Editing Any Scene via Language-guided Disentangled Gaussian Splatting

## Abstract

Abstract. Scene image editing is crucial for entertainment, photography, and advertising design.
Existing methods solely focus on either 2D individual object or 3D global scene editing.
This results in a lack of a unified approach to effectively control and manipulate scenes at the 3D level with different levels of granularity.
In this work, we propose 3DitScene , a novel and unified scene editing framework leveraging language-guided disentangled Gaussian Splatting that enables seamless editing from 2D to 3D, allowing precise control over scene composition and individual objects.
We first incorporate 3D Gaussians that are refined through generative priors and optimization techniques.
Language features from CLIP then introduce semantics into 3D geometry for object disentanglement.
With the disentangled Gaussians, 3DitScene allows for manipulation at both the global and individual levels, revolutionizing creative expression and empowering control over scenes and objects.
Experimental results demonstrate the effectiveness and versatility of 3DitScene in scene image editing.
Code and online demo can be found at our project homepage: https://zqh0253.github.io/3DitScene/ .

## 1. Introduction

Editing scene images is of great importance in various fields, ranging from entertainment, professional photography and advertising design.
Content editing allows to create immersive and captivating experiences for audiences, convey the artistic vision effectively and achieve the desired aesthetic outcomes.
With the rapid development of deep generative modeling, many attempts have been made to edit an image effectively.
However, they have encountered limitations that hindered their potential.

Previous methods primarily concentrate on scene editing in 2D image space.
They commonly rely on generative priors, such as GANs and Diffusion Models (DM), and employ techniques like modification of cross-attention mechanisms (Hertz et al., , ), and optimization of network parameters (Kim et al., ; Kawar et al., ; Ruiz et al., ; Gal et al., ; Chen et al., )
to edit the appearance and object identity within scene images.
While some efforts have been made to extend these methods to 3D editing, they ignore 3D cues and pose a challenge in maintaining 3D consistency, especially when changing the camera pose.
Moreover, these approaches typically focus on global scenes and lack the ability to disentangle objects accurately, resulting in limited control over individual objects at the 3D level.

In order to edit any scene images and enable 3D control over both scene and its individual objects, we propose 3DitScene, a novel scene editing framework which leverage a new scene representation, language-guided disentangled Gaussian Splatting.
Concretely, the given image is first projected into 3D Gaussians which are further refined and enriched through 2D generative prior (Rombach et al., ; Poole et al., ).
We thus obtain a comprehensive 3D scene representation that naturally enables novel view synthesis for a given image.
In addition, language features from CLIP are distilled into the corresponding 3D Gaussians to introduce semantics into 3D geometry.
These semantic 3D Gaussians help disentangle individual objects out of the entire scene representation, resulting in language-guided disentangled Gaussians for scene decomposition.
They also allow for a more user-friendly interaction *i.e.,* users could query specific objects or interest via text.
To this end, our 3DitScene enables seamless editing from 2D to 3D and allow for modifications at both the global and individual levels, empowering creators to have precise control over scene composition, and object-level edits.

We dub our pipeline as 3DitScene.
Different from previous works that focus on addressing a single type of editing, 3DitScene integrates editing requirements within a unified framework.
Our teaser figure demonstrates the versatility of 3DitScene by showcasing its application to diverse scene images.
We have conducted evaluations of 3DitScene under various settings, and the results demonstrate significant improvements over baseline methods.

## 2. Related work

Figure: Figure 2. 3DitScene training pipeline. Given input view, we first initialize 3DGS by lifting pixels to 3D space and then expand it over novel views by RGB and depth inpainting. Semantic features are then distilled into 3D Gaussians to achieve object-level disentanglement.
Refer to caption: x2.png

Figure: Figure 3. 3DitScene Inference pipeline. User can query object of interest via language prompt. Enabled by the disentangled 3D representation, user can change camera viewpoint, and manipulate the object of interest in a flexible manner.
Refer to caption: x3.png

Image Editing with Generative Models.
The field of 2D image synthesis has advanced significantly with the development of generative models such as GANs (Karras et al., , ) and diffusion models (Rombach et al., ; Song et al., ; Ho et al., ).
Many studies capitalize on the rich prior knowledge embedded in generative models for image editing. Some endeavors utilize GANs for various image editing tasks, including image-to-image translation, latent manipulation (Shen et al., ; Yang et al., ; Zhu et al., ; Xu et al., ; Jahanian et al., ), and text-guided manipulation (Patashnik et al., ).
However, due to limitations in training on large-scale data, GANs often struggle to perform well on real-world scene images. As diffusion models make notable progress, the community is increasingly focusing on harnessing the potent text-to-image diffusion model for real image editing (Kim et al., ; Kawar et al., ; Ruiz et al., ; Gal et al., ; Chen et al., ; Hertz et al., , ; Meng et al., ; Su et al., ).
However, these methods are confined to the 2D domain and are limited in editing objects within a 3D space.
Concurrently, other research efforts (Yenphraphai et al., ) attempt to address 3D-aware image editing, but they introduces inconsistency in the editing process, and cannot change the camera perspective of the entire scene.
In contrast, our model leverages an explicit 3D Gaussian to convert 2D images into 3D space while disentangling objects with language guidance. This approach enables our model not only to consistently perform 3D-aware object editing but also facilitates scene-level novel-view synthesis.

Single-view 3D Scene Synthesis.
Among 3D scenes generation (Zhang et al., ; Höllein et al., ; Chung et al., ; Chen et al., , ; Mao et al., ; Epstein et al., ), conditional generation on a single-view presents an unique challenge.
Previous approaches address this challenge by training a versatile model capable of inferring a 3D representation of a scene based on a single input image  (Wiles et al., ; Tucker and Snavely, ; Hu et al., ; Han et al., ; Flynn et al., ; Li et al., ; Hong et al., ; Yu et al., ).
However, these methods demand extensive datasets for training and tend to produce blurry textures when confronted with significant changes in camera viewpoints.
Recently, several works have embraced diffusion priors (Liu et al., ; Chan et al., ; Xu et al., ; Gu et al., ; Tang et al., ; Qian et al., ) to acquire a probabilistic distribution for unseen views, leading to better synthesis results.
Nevertheless, these methods often concentrate on object-centric scenes or lack 3D consistency.
Our approach connect 2D images and 3D scenes with explicit 3D Gaussians and incorporate diffusion knowledge, which overcome the mentioned challenges.

## 3. Method

Our target is to propose a 3D-aware scene image editing framework that allows simultaneous control over the camera and objects.
To accomplish this, Sec. 3.1 introduces a novel scene representation called language-guided disentangled Gaussian splatting.
In order to achieve object-level control, Sec. 3.2 further distills language features into the Gaussian splatting representation, achieving disentanglement at the object level.
We elaborate the optimization process in Sec. 3.3 and demonstrate the flexible user control enabled by our framework during inference in Sec. 3.4.

### 3.1. 3D Gaussian Splatting from Single Image

Preliminary. 3D Gaussian Splatting (3DGS) (Kerbl et al., ) has been proved effective in both reconstructive (Luiten et al., ; Yang et al., ) and generative setting (Zou et al., ; Tang et al., ).
It represents a 3D scene via a set of explicit 3D Gaussians.
Each 3D Gaussian describes its location by a center vector $\mathbf{x}\in\mathbb{R}^{3}$, a scaling factor $\mathbf{s}\in\mathbb{R}^{3}$, a rotation quaternion $\mathbf{q}\in\mathbb{R}^{4}$, and also stores an opacity value $\alpha\in\mathbb{R}$ and spherical harmonics (SH) coefficients $\mathbf{c}\in\mathbb{R}^{k}$ ($k$ represents the degrees of freedom of SH) for volumetric rendering.
All the above parameters can be denoted as $\Theta=\{\mathbf{x}_{i},\mathbf{s}_{i},\mathbf{q}_{i},\alpha_{i},\mathbf{c}_{i
}|i\in[0,\cdots,N-1]\}$, where $N$ is the number of 3D Gaussians.
A tile-based rasterizer is used to render these Gaussians into 2D image.

Image-to-3DGS initialization. Given an input image $\mathbf{I}\in\mathbb{R}^{3\times H\times W}$, an off-the-shelf depth prediction model is applied to estimate its depth map $\mathbf{D}\in\mathbb{R}^{H\times W}$.
Then, we could transform image pixels into 3D space, forming the corresponding 3D point clouds:

$$ (1) $\mathcal{P}=\phi_{2\to 3}(\mathbf{I},\mathbf{D},\mathbf{K},\mathbf{T}),$ $$

where $\mathbf{K}$ and $\mathbf{T}$ are camera intrinsic and extrinsic matrices respectively.
Such point clouds $\mathcal{P}$ are then used to initialize the 3DGS
by directly copying the location and color values, with other GS-related parameters randomly initialized.
To refine the 3DGS’s appearance, we adopt a reconstruction loss:

$$ (2) $\mathcal{L}_{\text{recon}}=\|\mathbf{I}-f(\mathcal{P},\mathbf{K},\mathbf{T})\| ^{2}_{2},$ $$

where $f$ is the rendering function.

We further enhance the rendered quality by leveraging prior knowledge from image generative foundation model, namely Stable Diffusion (Rombach et al., ).
It provides update direction to the images rendered by the current 3DGS in the form of Score Distillation Sampling (Poole et al., ) loss, denoted as $\mathcal{L}_{\text{SDS}}$.

3DGS expansion by inpainting.
When camera perspectives changes, rendered views will contain holes due to occlusion or new region outside the original view frustum.
We use Stable Diffusion to inpaint the uncovered regions.
Then, the newly added pixels need to be accurately transformed into 3D space to align seamlessly with the existing 3D Gaussians.

Previous methods (Chung et al., ; Höllein et al., ; Yu et al., ) first predict the depth values, and then use heuristic methods to adjust the values to align with the existing 3D structure.
However, relying on heuristic methods often overlooked various scenarios, leading to artifacts such as depth discontinuities or shape deformations.

Instead, we propose a novel method to lifted novel contents to 3D while ensuring seamless alignment without any heuristic procedures.
The key insight is to treat the problem as an image inpainting task, and utilize state-of-the-art diffusion-based depth estimation models (Fu et al., ; Yang et al., ) as a prior to solve the task.
During denoising steps, rather than using models to predict the noise over the entire image, we employ the forward diffusion process to determine the value of fixed areas (Meng et al., ).
This approach guarantees the final result, after denoising, adheres to the depth of original fixed parts, ensuring smooth expansion.

After smooth 3DGS expansion via depth inpainting, we take the imagined novel views as reference views, and apply reconstruction loss $\mathcal{L}_{\text{recon}}$ to supervise the updated 3DGS. SDS loss $\mathcal{L}_{\text{SDS}}$ is adopted for views rendered from camera perspectives that are interpolated between the user-provided viewpoint and the newly imagined views.

### 3.2. Language-guided Disentangled Gaussian Splatting

Based on the 3DGS built from single input image, users can generate novel views.
In this section, we further distill CLIP (Radford et al., ) language feature to 3D Gaussians.
This introduce semantics into 3D geometry, which helps disentangle individual objects out of the entire scene representation.

Language feature distillation.
We augment each 3D Gaussian with a language embedding $\mathbf{e}\in\mathbb{R}^{C}$, where $C$ denotes the number of the channels.
Similar to RGB image $\mathbf{I}$, a 2D semantic feature map $\mathbf{E}\in\mathbb{R}^{C\times H\times W}$ can also be rendered by the rasterizer.
To learn the embedding, we first use Segment Anything Model (SAM) (Kirillov et al., ; Zhang et al., ) to get semantic masks $\mathbf{M}_{i}$.
Then, we can obtain embedding of each object $\textbf{I}\odot\textbf{M}_{i}$ and supervise the corresponding region on rendered feature map $\mathbf{E}$, according to the distillation loss:

$$ (3) $\mathcal{L}_{\text{distill}}=\sum_{i}\big{\|}\big{(}\mathbf{E}-g\left(\mathbf{ I}\odot\mathbf{M}_{i}\right)\big{)}\odot\mathbf{M}_{i}\big{\|}_{2}^{2},$ $$

where $g$ is the CLIP’s image encoder, and $\odot$ denotes element-wise multiplication.
Following LangSplat (Qin et al., ), we additionally train an autoencoder to compress the embedding space to optimize the memory consumption of language embedding $\mathbf{e}$.

Scene decomposition.
After distillation, we can decompose the scene into different objects.
This enables user to query and ground specific object, and perform editing over single object (*e.g.* translation, rotation, removal, re-stylizing).

It is worth noting that such scene decomposition property not only enables more flexible edits during inference stage, but also provides augmentation over scene layouts during the optimization process.
Since now we can query and render each object independently, we apply random translation, rotation, and removal over objects.
This augmentation over the scene layout leads to a significant improvement in the appearance of occluded regions, ultimately enhancing the overall quality of the edited views (see Sec. 4.4).

### 3.3. Training

The overall training objective can be expressed as:

$$ (4) $\mathcal{L}=\lambda_{\text{recon}}\mathcal{L}_{\text{recon}}+\lambda_{\text{ SDS}}\mathcal{L}_{\text{SDS}}+\lambda_{\text{distill}}\mathcal{L}_{\text{ distill}},$ $$

where $\lambda_{\text{recon}}$, $\lambda_{\text{SDS}}$ and $\lambda_{\text{distill}}$ are coefficients that balance each loss term.

### 3.4. Inference

Due to the disentangled nature of our representation, users can now interact with and manipulate objects in a flexible manner.
Here, we mainly discuss prompting objects via two different modalities:

Text prompt.
Users can query an object through text prompts as shown in Fig. 3.
Following LERF (Kerr et al., ) and LangSplat (Qin et al., ), we calculate the relevancy score score between the language embedding $\mathbf{e}$ in the 3D Gaussians and the embedding of the text prompt $\mathbf{e}_{l}$ as:

$$ (5) $\texttt{score}=\min_{i}\frac{\exp(\mathbf{e}\cdot\mathbf{e}_{l})}{\exp(\mathbf {e}\cdot\mathbf{e}_{l})+\exp(\mathbf{e}\cdot\mathbf{e}^{i}_{\text{canon}})},$ $$

where $\mathbf{e}^{i}_{\text{canon}}$ is the CLIP embeddings of canonical phrases including “object”, “things”, “stuff”, and “texture”.
Gaussians that have relevance scores below a predefined threshold are excluded. The remaining part is identified as the object of user interest.

Bounding box. Users can also select an object by drawing an approximate bounding box around it on the input image.
3D Gaussians within the bounding box are first identified, followed by K-Means clustering based on their language embeddings $\mathbf{e}$.
Assuming the object is the most significant one within the bounding box, clusters whose number of Gaussians does not exceed a threshold proportion will be discarded.

In the meantime, user can also adjust the camera viewpoint by specifying intrinsic and extrinsic parameters.

## 4. Experiments

**Table 1. User study result. We report the percentage of favorite users for the consistency and quality of images edited by each method**
|  |  | AnyDoor | Ojbect 3DIT | Image Scuplting | Ours |
| --- | --- | --- | --- | --- | --- |
| Consistency | Human | 5.1 | 16.8 | 12.7 | 65.4 |
| GPT4-v | 0.0 | 6.7 | 31.3 | 62.0 |  |
| Quality | Human | 10.4 | 0.5 | 25.1 | 64.0 |
| GPT4-v | 6.7 | 13.3 | 39.2 | 40.8 |  |

Figure: Figure 4. Visualization of rendered images and feature maps. For each sample, we show three views of rendered images and feature maps. To demonstrate the disentangled scene representation, we use the language embedding to select a foreground object and render it exclusively.
Refer to caption: x4.png

### 4.1. Settings

Implementation details. To lift an image to 3D, we use GeoWizard (Fu et al., ) to estimate its relative depth.
Stable Diffusion (Rombach et al., )’s inpainting pipeline is adopted to generate new content for 3DGS’s expansion.
We leverage MobileSAM (Zhang et al., ) and OpenCLIP (Ilharco et al., ) to segment and compute rendered views’ feature maps, which are further leveraged to supervise the language embedding of 3D Gaussians.
We use Stable Diffusion to perform Score Distillation Sampling (Poole et al., ) during optimization.
Given the already decent image quality at the start of optimization benefited from explicit 3DGS initialization, we adopt a low classifier-free guidance (Ho and Salimans, ) scale.

Baselines. We compare our method with following scene image editing works:
(1) AnyDoor (Chen et al., ) is a 2D diffusion-based model that can teleport target objects into given scene images. It leverages Stable Diffusion’s powerful image generative prior by finetuning upon it.
(2) Object 3DIT (Michel et al., ) is designed for 3D-aware object-centric image editing via language instructions. It finetunes Stable Diffusion over a synthetic dataset containing pairs of original image, language instruction, and edited image.
(3) Image Sculpting (Yenphraphai et al., ) is also designed for 3D-aware object-centric image editing. It estimates a 3D model from an object in the input image to enable precise 3D control over the geometry. It also uses Stable Diffusion to refine the edited image quality.
(4) AdaMPI (Han et al., ) focuses on the control over camera perspective. It leverages monocular depth estimation and color inpainting with learned adaptive layered depth representations.
(5) LucidDreamer (Chung et al., ) tackles novel view synthesis by querying Stable Diffusion’s inpainting pipeline with dense camera trajectories.

Figure: Figure 5. Comparison results of object-centric manipulation. We apply translation, resizing, and removal over foreground objects. Two different edit results are shown for each method.
Refer to caption: x5.png

Figure: Figure 6. Comparison results of camera control. We show two views with different camera perspectives for each method.
Refer to caption: x6.png

Figure: Figure 7. Ablation results for layout augmentation during optimization. To evaluate the degree of object-level disentanglement, we conduct object removal for each sample. The top row displays the input image, while the next two rows showcase the edited scene
Refer to caption: x7.png

### 4.2. Quantitative results

We conduct a user study to compare the edited results by our method with the established baselines.
We generate 20 samples for each method and request users to vote for their preferred method based on consistency with the original image and quality for each sample.
We collect feedback from 25 users, and report the result in Tab. 1.
Our method consistently outperforms previous baselines in terms of both consistency and image quality.
As recommended in a previous study (Wu et al., ), GPT4-v has the ability to evaluate 3D consistency and image quality. Therefore, we include GPT-4v as an additional criterion.
The preference of GPT4-v is well aligned with human preference, which once again demonstrates the superiority of 3DitScene.

### 4.3. Qualitative results

Fig. 4 showcases the generated novel views with their respective feature maps produced by our framework.
The feature maps demonstrate remarkable accuracy in capturing the semantic content of the images. This ability to distinctly separate semantic information plays a crucial role in achieving precise object-level control
In the following, we demonstrate flexible editing over scene images enabled by our framework, and also compare with baseline methods.

Object manipulation.
Since different methods define object manipulation, particularly translation operations, in different coordinate systems(^1^11AnyDoor, Object 3DIT and Image Sculpting respectively employs 2D masks, language prompts, and image coordinates for control. We use coordinates in 3D space instead.), it becomes challenging to evaluate them under a unified and fair setting.
Therefore, we evaluate each method under its own specific setting to achieve the best possible result.
As shown in Fig. 5, AnyDoor struggles to maintain object identity and 3D consistency when manipulating object layouts, primarily due to the absence of 3D cues.
Object 3DIT, trained on synthetic datasets, exhibits limited generalization ability to real images.
By leveraging a 3D model derived from the input image, Image Sculpting achieves better results.
Nonetheless, it encounters issues with inconsistency when manipulating objects. This arises from the fact that they solely rely on the 3D model for providing rough guidance, resulting in the loss of finer details during optimization.

In contrast, our method delivers satisfactory 3D-aware object-level editing results. It maintains accurate 3D consistency of edited objects after rearranging their layout. Additionally, it preserves occlusion relationships within the scene, such as moving the girl to be partially occluded by a foreground object in the last row example.

Figure: Figure 8. Ablation results for loss terms. We show rendered novel views under different loss settings. The left column lists the input image. In right columns, two views are shown for each configuration. The quality degrades when reconstruction or SDS loss term is discarded
Refer to caption: x8.png

Figure: Figure 9. Ablation results for depth inpainting. Row (a): images and corresponding depth map (only available in the left half); Row (b): depth map predicted by heuristic alignment; Row (c): depth map predicted by our depth inpainting method.
Refer to caption: x9.png

Camera control.
We compare our methods with AdaMPI and LucidDreamer for camera control.
As illustrated in Fig. 6, AdaMPI only focuses on scenarios where the camera zooms in, and does not consider novel view synthesis.
Therefore, this approach is not suitable for 3D-aware image editing when large camera control is required.
LucidDreamer also leverages Stable Diffusion’s inpainting capacity for novel view synthesis. However, it suffers from sudden transitions in the content within the frame (see sample in the bottom line). It also requires dense camera poses.
In contrast, our method only needs as few as three camera poses and enables smooth transitions from the input view to novel views, enhancing user control over the camera perspective.

### 4.4. Ablation study

Layout augmentation during optimization.
As our representation disentangles at object level, we could perform layout augmentation during optimization.
Here, we investigate whether disentanglement property benefits the optimization process.
We use the task of removing objects to evaluate the degree of disentanglement.

As illustrated in Fig. 7, when layout augmentation is disabled during optimization, floating artifacts can be observed.
We discover that these Gaussians lie inside the object.
They are occluded by Gaussians at the surface.
As they do not contribute to the rendering result, they are consequently not updated by gradient descent during optimization, leaving their language embeddings unsupervised.

In contrast, when applying layout augmentation during optimization, such Gaussians will be exposed when the foreground object is moved away, and hence updated.
With this ablation, it is concluded that the disentanglement property of the proposed representation not only enables more flexible inference, but also contributes to the optimization process.

Loss terms.
During optimization, we adopt three loss terms: $\mathcal{L}_{\text{recon}}$, $\mathcal{L}_{\text{SDS}}$, and $\mathcal{L}_{\text{distill}}$.
$\mathcal{L}_{\text{distill}}$ plays a critical role in distilling language embedding into 3D.
The remaining two terms focus on enhancing the visual quality of images.
Here, we investigate the contribution of these two items through an ablation study.
As input images can provide guidance of overall structure and detailed appearance, there is no need of applying a large classifier free guidance (CFG) value for SDS loss.
Thus, by default, we choose 5 as the CFG value.

As illustrated in Fig. 8, the image quality degrades severely without $\mathcal{L}_{\text{recon}}$ or $\mathcal{L}_{\text{SDS}}$.
Without $\mathcal{L}_{\text{recon}}$, the image is only refined by the SDS loss, which creates discrepancies with the original image.
When the CFG value is set low, 5 as default, the image appears lacking in details and exhibits unusual texture patterns. Increasing the CFG value introduces more details, yet leads to inconsistencies with the original image, while the issue of strange texture patterns persists.
Additionally, only applying $\mathcal{L}_{\text{recon}}$ results to floating artifacts and blurriness across the entire image.
In conclusion, both SDS and reconstruction loss are crucial for achieving decent image quality.

Depth inpainting.
When expanding 3DGS at novel views, we need to estimate the depth map of unseen regions.
Here, we compare our inpainting-based depth estimation with heuristic-based method.
Fig. 9 show images with depth map available in the left part. The task is to predict the depth map of the right part.
Method relying on heuristic alignment results to artifacts like depth discontinuity.
In contrast, our proposed method is capable of producing accurate depth maps that align well with the left known part.

## 5. Conclusion and Discussion

We present a novel framework, 3DitScene, for scene image editing.
Our primary objective is to facilitate 3D-aware editing of both objects and the entire scene within a unified framework.
We achieve this by leveraging a new scene representation, language-guided disentangled scene representation.
This representation is learnt by distilling CLIP’s language feature into 3D Gaussians.
The semantic 3D Gaussians effectively disentangle individual objects out of the entire scene, , thereby enabling localized object editing.
We test 3DitScene under different settings and prove its superiority compared to previous methods.

Figure: Figure 10. Image pairs edited by 3DitScene.
Refer to caption: x10.png

Figure: Figure 11. Image pairs edited by 3DitScene.
Refer to caption: x11.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: * What is the runtime of the proposed method and other alternatives? While the 3D representation seems to improve quality, it seems unlikely that this multi-stage method would be fast enough for real use.
* The teaser figure seems to suggest that this process is "hands-off," taking in a text prompt about manipulating the scene and then generating, but I'm having trouble telling if this is actually true. Could the authors please clarify this?
* To what extent does the method begin to fail if there are large baseline camera movements? SDS at low CFG has very limited generation ability, for instance.
* Are there other downstream tasks where this pipeline could be useful? Maybe other instances of image editing?
soundness: 2
strengths: Strengths: 
* The authors leverage a broad variety of methods to extend image features to 3D, all of which are well-justified inclusions.
* Authors ablate the main technical decisions in the paper, specifically the different loss terms, CFG scale and object removal.
* Uses GPT4o LLM-as-critic evaluations from past works to automatically evaluate 3D generation quality.
* Writing is focused and practical, highlighting different elements of the design space and clearly explaining each decision the authors made.
* Method is training-free, so it should improve over time as base models improve.
* Figures are well-presented and understandable.
* The importance of the proposed task is clear and the method seems to be a reasonable improvement over existing works.
summary: The authors present a new image editing pipeline that lift 2D data priors, including CLIP, mono-depth, SAM and Stable Diffusion, into a 3D Gaussian Splatting representation for manipulation. This decomposes the scene into a camera and queryable objects that can be rotated and translated in 3D space. The process introduces discontinuities at novel views, which the authors propose to fill in following an image inpainting procedure and using the SDS loss from Dreamfusion. The resulting scene can be rendered to produce an edited image.
weaknesses: Weaknesses:
* The main weakness of this paper is that it mostly combines existing techniques in a pipeline to improve an application rather than introducing new technical insights. For instance:
  * LucidDreamer used monocular depth to unproject an image into a Gaussian Splatting scene
  * LERF introduced language-embedded 3D representations
  * Dreamfusion introduced the SDS objective
  * Inpainting techniques are already broadly explored in the image diffusion model literature
* Evaluations are very limited, with the user study not being well-explained and including a small number of participants (25) and generated samples (20). This is a known challenge in this problem space, but it would have been nice to see more discussion or contribution in this direction.

---

confidence: 3
contribution: 4
initial_rating: 6
presentation: 3
questions: How long does it take for training one scene?
How long does it take for rendering one image?
I don't understand why “Remove the headscarf then move the camera” case in the teaser can generate the hair of the woman. cause in inference you just manipulate the gs, is ther anything generative that enables this capability?

I pretty much love the paper, if the author can provide more details, I am happy to raise my score.
soundness: 3
strengths: The quality is very well. The proposed methods are solid

Authors also conduct a lot of  experiments to compare with multiple baselines.
summary: This paper provides a method for highly precise 3d scene editing.
The author starts from a single image input, first using estimated depth to init the 3dgs. Use inpainting for both novel view and its depth map. Authors also introduce a language feature distillation loss, so that we will be able to use text to query the gs during inference
weaknesses: I am afraid that the SAM mask is not accurate, the distill loss may cause some artifacts. Like some of the gaussians in the edge may not be able to match the image prompt. Is it true?
Current method cannot deal with shadow and lightning, like the Rotate the camera then delete the dog case in the teaser
Need more visual results.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Please refer to the weaknesses section.
soundness: 3
strengths: The proposed technique is well-founded, and the experimental results are impressive.
summary: This paper presents a scene editing framework that leverages language-guided disentangled Gaussian Splatting. Initially, the input image is projected into 3D Gaussians, which are subsequently refined and enriched through a 2D generative prior. Language features from CLIP are then embedded into the corresponding 3D Gaussians. This representation enables view changes and text-guided editing of the scene.
weaknesses: The novelty appears limited, as the proposed method largely combines existing techniques. Projecting a 2D image into 3D Gaussians based on depth information has been demonstrated in works like LucidDreamer, among others. Additionally, embedding language features into 3D Gaussian Splatting (3DGS) and enabling text-guided editing have been explored in methods such as Grouping Gaussian, LangSplat, and GaussianEditor.

It is unclear how the proposed method outperforms prior techniques, such as LucidDreamer and AdaMPI, in Figure 6. While LucidDreamer and AdaMPI exhibit some out-of-range areas, simple inpainting could potentially address these issues.

Since the method relies on depth maps to generate 3D Gaussian splats, it appears limited in its ability to handle significant changes in view angle.

The method seems to only function with generated images rather than real-world inputs.

---

confidence: 2
contribution: 2
initial_rating: 5
presentation: 3
questions: See what is written in the weaknesses section.
soundness: 3
strengths: 1. 3DitScene effectively integrates various editing requirements into a single framework, facilitating comprehensive manipulation of both scenes and objects.
2. The framework employs language-guided disentangled Gaussian Splatting to produce a detailed 3D scene representation, which enables novel view synthesis and improves object decomposition.
summary: The authors introduce 3DitScene, a unified framework that employs language-guided disentangled Gaussian Splatting, allowing seamless transitions from 2D to 3D editing. This approach utilizes refined 3D Gaussians informed by CLIP’s language features, enabling precise control over both individual objects and overall scene composition. Experimental results validate the effectiveness and versatility of 3DitScene, demonstrating its superiority over existing methods.
weaknesses: The implementation details are not clearly articulated. Specifically, the training and optimization specifics for 3D Gaussian Splatting (3DGS) are lacking, including the number of iterations during initialization and for novel view optimization. Furthermore, the parameters of the loss function, such as λrecon, are not specified. Additionally, the experimental results do not appear to be satisfactory.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. The paper lacks clarity on the editing process, despite mentioning text, bounding boxes, and camera controls in Sec.3.4.
The detailed operation steps are still ambiguous, and there is no clear editing procedure outlined in paper or video demo.

    From Sec.3.4, it appears that editing primarily involves "text-guided" editing by querying specific object + object manipulation?

    Is "camera control" achieved always by changing camera instrincs and extrinsics parameters? But in Fig.1, it seems that text prompts are also used for control.

    In addition, there's no examples of bounding box usage in the paper.

    I also suggest more editing details such as text prompts in Fig.5 and 6 and video, similar to what is shown in Fig.1.

2. Since the 3D-aware editing is based on explicit 3DGS, and the disenangled object manipulation does not involve whole image re-generation. Accordingly, lighting inconsistencies arise after editng, e.g., shadows do not change appropriately as object moves. Can the lifted semantics help resolve this problem, or do you have any ideas can handle this obvious artifact?

3. Language-semantics 2D-to-3D lifting is increasingly common nowadays, e.g., LERF, LangSplat, Gaussian Grouping. What are the advantages of proposed semantics lifting schemes over these existing works? Additionally, would other language-embedded 2D-to-3D lifting methods should be compatible with this proposed editing framework?
soundness: 3
strengths: 1. The work proposed a 3D-aware image editing pipeline by establishing semantic 3D representation via 2D-to-3D inpainting for novel view completion and language semantics lifting for semantic disenanglements. Under this language-guided disenangled 3D representation, the editing supports object-level maipulatiion and camera controls.

2. Evaluation experiments are extensive for performance and comparisons with baselines. The method designs are validated through conducting ablation studies, exploring various alternative designs, including layout augmentation, losses, depth inpainting strategies.
summary: The paper proposed a unified framework, 3DitScene, for 3D-aware 2D image editing with different level of granulairies and diverse editing tasks, such as object level editing and camera adjustments.
By converting a single image into 3D Gaussian splatting (3DGS) representation, 3DitScene generates additional 3D content in previously unseen areas via RGB and depth inpainting from diffussion priors. Additionally, it distills CLIP language semantics into the expanded 3D. With the recontructed 3D scene with disentangled 3D semantics, the framework enables editing of both viewing camera and objects in the the image by camera, bounding box and textual controls.
weaknesses: 1. While the paper proposed a practical and accurate image editing system, it does not demonstrate the editing procedure for practical use. Also see Questions 1 for further details.

2. Some details are missing for method reproduction and training, such as trainining parameter configuration. It is also unclear about camera configuration, e.g. whether K and T in Eq.1 are always preset and fixed for given single image? More details are needed to facilitate reproduction and editing controls.

Additional concerns can be found in the Questions.

Meta-review: The paper introduces a new framework for 3D scene editing that leverages language-guided Gaussian splatting. Reviewers generally praised the quality of the editing results and the potential of the approach, but also raised concerns about the limited novelty, as the method primarily combines existing techniques. 

Although scores remain mixed, the main complaints seem to be about technical novelty---and in my eyes, combining a set of existing methods into a novel system/application seems like a sufficiently significant contribution for publication.

Decision: Accept (Poster)