MVDrag3D: Drag-based Creative 3D Editing via Multi-view Generation-Reconstruction Prior

Authors: Honghua Chen, Yushi LAN, Yongwei Chen, Yifan Zhou, Xingang Pan
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: 3D Editing, 3D Drag, Multiview Diffusion Model

Abstract:
Drag-based editing has become popular in 2D content creation, driven by the capabilities of image generative models. However, extending this technique to 3D remains a challenge.  Existing 3D drag-based editing methods, whether employing explicit spatial transformations or relying on implicit latent optimization within limited-capacity 3D generative models, fall short in handling significant topology changes or generating new textures across diverse object categories. To overcome these limitations, we introduce MVDrag3D, a novel framework for more flexible and creative drag-based 3D editing that leverages multi-view generation and reconstruction priors.
At the core of our approach is the usage of a multi-view diffusion model as a strong generative prior to perform consistent drag editing over multiple rendered views, which is followed by a reconstruction model that reconstructs 3D Gaussians of the edited object. While the initial 3D Gaussians may suffer from misalignment between different views, we address this via view-specific deformation networks that adjust the position of Gaussians to be well aligned. In addition, we propose a multi-view score function that distills generative priors from multiple views to further enhance the view consistency and visual quality. Extensive experiments demonstrate that MVDrag3D provides a precise, generative, and flexible solution for 3D drag-based editing, supporting more versatile editing effects across various object categories and 3D representations.

Decision: reject
Reviewer scores: 8, 5, 5, 5

Reviews and discussion:
<thread>
creation date:
1730352342605

thread_type:
official_review

<thread_object>
creator: reviewer_bkuz

<content>
summary:
The paper proposed a new algorithm called MVDrag3D, which is a framework for drag-based 3D editing. The method first generates multi-view images, followed by mechanisms introduced to enforce consistencies in the generation process, before reconstructing a 3D model. In the experiments, it looks like the proposed approach provides a more precise 3D reconstructed results comparing to other state of the art approaches

strengths:
The paper is a system paper but t does has some interesting novel ideas that might be beneficial to the research community. There are several notable strengths
- The proposed approach is sensible in that leverating multi-view to control the dragging behavior is reasonable, and the overall formulation of the algorithm is intuitive.
- The introduced multi-view guidance score and the refinement approach is novel in that it helps reduce artifacts and addressed softspots in the algorithm. There are several mini-ideas in the approach that are interesting, such as adding random noises in the inversion, etc.
- Experimental results looks competitive, especially some of the visual comparisons.

weaknesses:
I think there are merits in the paper but i've got some questions or confusions while reading the paper
- In figure 3, the differences of the final reconstruction results are very minor to me. I couldn't fully appreciate the need of the random noise unless I zoom into the pictures. Technically none of those results matches the input very well.
- Experimental results are conducted on vey simple 3D objects. Maybe this is a common issue for all state of the art approaches. Similarly, in fig. 6 -- many appraoches perform quite competitively (such as the boots, flower, and the suited animal case), and I struggle to tell if the approach is truly groundbreaking or incremental
- The descriptions of the technical formulations of the approach requires more details. For example, i struggled to understand the "Gaussian appearnace optimization" section -- "Note that all Gaussian properties are optimized during this process, with densification and pruning operations enabled." -- which densification and pruning approach exactly, and how is this done? The "shoe" example in fig.5 is also showing tiny improvements with the optimizations.
- It'd be good if the paper could provide human evaluations. The automatically computed metrics might not tell the full story of the effectiveness of the approach.

questions:
I have raised questions in the weakness section
</content>
</thread_object>

</thread>
<thread>
creation date:
1730593961024

thread_type:
official_review

<thread_object>
creator: reviewer_mdmj

<content>
summary:
This paper proposes MVDrag3D, which utilizes a deformation network and SDS loss from multi-view diffusion models to edit 3D objects. The results show that the proposed method can generate 3D creations with the corresponding drag operation while creating new textures.

strengths:
* MVDrag3D is capable of user-friendly drag operations. Extending the idea of 2D drag to 3D makes sense and shows good qualitative results.
* MVDrag3D shows the ability to generate new textures, which makes it different from the standard 3D deformation works.

weaknesses:
* Low-quality 3D creations: Most of the results are based on simple 3D creations or generated 3D creations, which are pretty low-quality. For example, the shoes and the fox show a blurred texture. Such evaluation dramatically limits the application of the proposed method. In addition, the edited results in the video tend to be blurred compared with the "Drag on meshes" illustration. These observations raise two concerns: (1) will this method blur the original 3D content? (2) will this method work for high-quality 3D creations?
* Failure cases of drag operations: The drag operation does not usually seem successful in the examples. The drag on the owl is not correctly optimized since the cloth doesn't reach the target position. In addition, the texture of the tie is degenerated during this process. Such a result even sees worth than the baselines like APAP. The use of SDS loss probably hurt the results to some extent.

questions:
* Listed in the wakenesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730631762119

thread_type:
official_review

<thread_object>
creator: reviewer_xmm9

<content>
summary:
This paper proposes a 3D editing method based on sparse control point dragging, applicable to both neural implicit representations and explicit mesh representation. The proposed method is based on a multi-view diffusion model and a large model for sparse view reconstruction. Specifically, the method renders the 3D representation into images from four specified viewpoints, then uses DDIM inversion to convert these images into Gaussian noise, which is regenerated using the multi-view diffusion model MVDream. During the generation process, control point movements are incorporated as a constraint to define multi-view guided energy optimization for intermediate denoising features. The generated edited images from the four viewpoints will obtain a 3D GS representation using the large pre-trained model, LGM. However, the 3D Gaussians obtained from the four views exhibit geometric deviations and texture artifacts. The former is addressed by introducing an MLP to shift the positions of Gaussian spheres, while the latter is resolved through multi-view SDS loss optimization. The paper compares the proposed method with other similar drag-based editing methods.

strengths:
-- The paper has a clear description and is well-structured, providing not only the necessary details for implementing the method but also the motivation for choosing this technical approach.

-- The method proposed in this paper is reasonable and has achieved results superior to other comparative methods.

weaknesses:
-- The method proposed in this paper is straightforward but lacks sufficient technical contribution. The first step generates multi-view edited images, with its optimization method and energy terms resembling those of Dragondiffusion. The main difference is merely substituting a single-view diffusion model with a multi-view one, which is a simple and direct change requiring minimal adjustments. In the second step, the LGM is used to regenerate the 3DGS model by directly utilizing an existing pre-trained model. Finally, the adjustments made through the deformation MLP and multi-view SDS loss optimization are common practices in 3D modeling and generation. Therefore, the proposed method primarily combines existing approaches without significant improvements, leaning more towards engineering rather than showcasing technical contributions.

-- Although the proposed method outperforms existing approaches in terms of deformation effects, it is still limited by the 3DGS representation, resulting in some artifacts and blurriness at the model edges in the final output. For instance, this is evident in the green leaves of the flowers, the open mouth of the crocodile, and the open mouth of the lion in the video.

questions:
-- Does the proposed method ultimately produce a 3DGS representation for editing the input mesh? What if we still want a mesh-based output? While 3DGS can extract the mesh, the quality of the extracted mesh is not very good, and it is necessary to use higher-quality representations like 2DGS for better reconstruction results.

-- While the deformation MLP and multi-view SDS loss optimization do have some effectiveness in fine-tuning the final results, they seem to be suitable primarily for relatively minor issues. For more significant geometric inconsistencies and texture artifacts, it remains unclear whether they can adequately address these challenges. And could the video include results from ablation experiments related to these two aspects?

After rebuttal:
1. Though this work is a good engineering effort, I still believe it lacks sufficient technical contribution and relies heavily on the multi-view reconstruction method. As I mentioned in the previous review, it mainly combines some existing methods with some appropriate improvements, but these improvements are not substantial enough to be considered an independent technical contribution. While this combination can solve certain problems that existing methods struggle with, it does not mean that the approach is technically novel or provides enough insight.

2. Furthermore, the artifacts in the results indicate that the proposed method is too dependent on the robustness of existing methods. The authors mention that more advanced methods can be substituted, but this further suggests that the proposed approach is just a combination of existing methods, and each of these methods needs to have robust results on its own.

3. While the proposed editing method can handle some situations that existing deformation methods cannot, editing methods generally need to preserve the original representation. Although the introduction of 2DGS can further obtain meshes from the editing results, it does not fully preserve the details and topology of the original mesh. Therefore, the proposed method’s applicable editing objects might need to be reconsidered.

Taking these points into consideration along with reviews from other reviewers, I still have concerns about accepting this paper. Hence, I maintain my original score.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730648752957

thread_type:
official_review

<thread_object>
creator: reviewer_dkc5

<content>
summary:
In this paper, the authors present a framework named MVDrag3D for drag-based 3D editing. It first renders a 3D object and the drag points into 4 orthogonal views, and introduce a multi-view guidance energy to achieve consistent multi-view score-based image editing. Multi-view Gaussian reconstruction is then performed on the edited images, followed by Gaussian position adjustment by view-specific lightweight deformation networks. Finally, an image-conditioned multi-view SDS optimization is applied to further enhance view consistency and visual quality.

strengths:
+ The idea of casting 3D drag-based editing into multi-view 2D drag-based editing sounds novel and feasible.
+ The use of multi-vew diffusion model to ensure consistent multi-view image editing sounds novel and feasible.
+ The lighweight view-specific deformation networks appear to be effective in improving the geometric alignment of the 3D Gaussians with the image.
+ The use of image-conditioned multi-view SDS optimzation for enhancinmg view consistency and visual quality sounds logical.
+ Both qualitative and quantitative results demonstrate SOTA results.

weaknesses:
- There is no disucssions on how to choose the optimal 4 orthogonal views. Theoretically, the 4 orthogonal should be chosen such that the drag directions should be as far away from the view directions as possible. In this paper, the authors simply choose orthogonal azimuths (0 deg, 90 deg, 180 deg, 270 deg) and a fixed elevation (0 deg).
- By rendering only 4 orthogonal views, some details (shape and texture) of the 3D object may be lost.
- In 3D Gaussian reconstruction step, partial 3D Gaussians are regressed for each view, which are then fused into a unifed representation. It is not clear why the authors do not perform multi-view Gaussian reconstruction instead.
- It is also not clear wheather the Gaussian position optimization step is necessary. Why can't the image-conditioned multi-view SDS optimization be carried out directly on the initial 3D Gaussians?

questions:
- Why do the authors regress a partial 3D Gaussians for each view and fuse them afterwards instead of performing a multi-view reconstruction?
- Referring to Table 1, is there any explanations why SDS optimization produces better results for Gaussians than for meshes in terms of DAI?
- For 3D meshes, are the final outputs still 3D Gaussians? Are the quantitative results all computed on 3D Gaussians?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732507517977

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear Reviewers,

Thank you for your valuable feedback and insightful comments.

As the discussion period concludes on November 26th, we kindly ask if our responses have effectively addressed your questions. Please don’t hesitate to reach out with any additional questions, concerns, or requests for clarification.

Warm regards, The Authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1732848361856

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Response to common concerns of Reviewer bkUz and MdMJ

comment:
**Differences and improvements with baselines.** To better highlight the differences and improvements between the methods, we categorized drag editing tasks into three types: **local deformation**, **large structural changes**, and **new content generation**. We analyzed and compared the results of four methods—APAP, PhysGaussian, DiffEditor, and Ours—across these categories, as summarized in the table below. 

| Editing Category | Local Deformation  | Large Structural Changes | New Content Generation |
| :---- | :---- | :---- | :---- |
| Examples | *Yoda* in Fig. 1, *black shoe* and *axes* in Fig. 6 | *Dog* in Fig. 1, *wolf*, *elephant*, *chair* in Fig. 6, *tiger* and *giraffe* in Fig. 13 | *Sparrow* and *lion* in Fig.1, *crocodile* in Fig. 2,  *lotus bud* in Fig. 6, and *eagle* in Fig. 13 |
| APAP | √ | less realistic shape | × |
| PhysGaussian | √ | less realistic shape | × |
| DiffEditor | Inconsistent | Inconsistent | Inconsistent |
| Ours | √ | √ | √ |

For **local deformation** tasks, such as deforming *Yoda* in Fig. 1 or shortening black shoes and stretching axes in Fig. 6, APAP, PhysGaussian, and our method handle these tasks successfully, while DiffEditor exhibits inconsistencies.

For **large structural changes**, such as lifting the leg of the *dog* in Fig. 1 or articulating the *wolf* and *elephant* in Fig. 6 and the *tiger* and *giraffe* in Fig. 13, APAP and PhysGaussian produce unrealistic shapes. For instance, when lifting the wolf’s left leg in Fig. 6, APAP deforms the leg but bends it rather than lifting it, resulting in an unrealistic shape. In contrast, our method produces realistic and natural articulation-like shapes that adhere to the animal’s kinematics.

For **new content generation**, such as opening the *lion*’s mouth in Fig. 1 or opening the *eagle*’s wings in Fig. 13, APAP and PhysGaussian are unable to handle these tasks. While DiffEditor can generate new content from certain views, inconsistencies across multiple views result in degraded reconstruction quality. In contrast, our method handles these cases effectively.

Finally, as summarized in the table, our method outperforms the baselines in most cases (as also noted by Reviewers Dkc5 and xmM9). It enables large-scale structural changes and generates new content through intuitive drag operations, making it highly effective for complex scenarios. We believe the main contribution of this work lies in a new framework to address a very challenging and underexplored task, while the minor artifacts could be addressed by adopting more powerful generation-reconstruction priors in the follow-up works.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes a method for 3D drag editing. Most reviewers gave negative ratings, with the primary concern being the low visual quality and marginal qualitative improvement in the results. The rebuttal addressed the paper’s ability to handle significant structural changes, but it did not sufficiently convince the reviewers that the improvements were substantial enough to justify the method. Despite Reviewer Dkc5 recommending acceptance, he/she did not feel strongly enough to champion the paper, and the results still appeared blurry. After a careful review of the results, the area chair appreciates the significant quantitative improvement (especially in terms of DAI) and recognizes the merit of the work. However, the visual quality remains an issue. Therefore, the area chair recommends rejection at this stage, and suggests the authors to revise the paper by incorporating the suggestions from the reviewers and submit the revised version to a future venue.