3D Object Manipulation in a Single Image Using Generative Models

Authors: Ruisi Zhao, Zechuan Zhang, Zongxin Yang, Yi Yang
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: 3d object manipulation, diffusion models, image editing, image animation

Abstract:
Object manipulation in images aims to not only edit the object presentation but also gift objects with motion. Previous methods encountered challenges in concurrently handling static editing and dynamic motion applications, while also struggling to achieve realism in object appearance and scene lighting. In this work, we introduce OMG3D, a novel framework that integrates the precise geometric control with the generative power of diffusion models, thus achieving significant enhancements in visual performance. Our framework converts 2D objects into 3D, enabling user-directed modifications and lifelike motions at the geometric level. To address texture realism, we propose CustomRefiner, a texture refinement module that pretrain a customized diffusion model to align the style and perspectives of coarse renderings with the original image. Additionally, we introduce IllumiCombiner, an lighting processing module that estimates and adjusts background lighting to match human visual perception, resulting in more realistic illumination. Extensive experiments demonstrate the outstanding visual performance of our approach in both static and dynamic scenarios. Remarkably, all these steps can be done using one NVIDIA 3090. The code and project page will be released upon acceptance of the paper.

Decision: reject
Reviewer scores: 3, 8, 5, 6

Reviews and discussion:
<thread>
creation date:
1730571461096

thread_type:
official_review

<thread_object>
creator: reviewer_ljb3

<content>
summary:
The paper proposes OMG3D for 3D editing of objects. It takes an image and produce 3D assets with high quality texture and lighting estimation. Then 3D editing including motion and shadow effects can be added in common 3d editing softwares. To get high-quality results, it proposes a texture refinement and a lighting estimation module in the image-to-3d stage. The method is compared against image and video generation models using user study and LLM-based evaluations, under which it works better in realism and consistency axis, and works better or on-par in terms of image/text alignment.

strengths:
- The paper is well written and the video demo is well made.
- The method produces high-quality results, and those can be plugged into existing graphics pipelines relatively easily due to the explicit modeling of geometry, texture and lighting.
- The authors did a good job in quantitative evaluations and comparisons, including some recent image/video generation methods.
- The method is lightweight and can run on a single 3090 GPU.

weaknesses:
- Is the comparison to images/video models fair? For instance, the pipeline introduced in this paper involves human editing (e.g., rigging and animation) while the baselines does not. Although I like the fact that the comparisons reflect the final quality, the different level of human involvement can be made clearer and discussed.
- It would be useful to explicitly state the steps that require manual intervention and differentiate those from the automatic component of the method through a table. For instance, intensity enhancement needs manual adjustment but lighting estimation does not.
- Lack of ablations and comparison to image-to-3D methods. Since this is main technical contribution, it would be useful to ablate the design choices that are different from existing methods, and report quantitative results showing their individual contributions to the overall performance. For instance, how much does DDIM inversion help, how much does dreambooth customization help, how much does depth controlnet help, and how much does feature injection helps. Ideally, by ablating each of the designs one after another, it can reach a known baseline.

questions:
- How is depth control incorporated? Is it added to SDXL during Dreambooth finetuning? What does the denoising function look like?

- How to deal with multiple planes (e.g., table and ground) in plane creation process? Since depth from Depth Anything is up to scale, how to adjust the plane to be in contact with the object?

- How is the material property specified during rendering in the editing stage?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730638917116

thread_type:
official_review

<thread_object>
creator: reviewer_ovkz

<content>
summary:
In this paper, the authors present a framework named OMG3D for object manipulation from images. Similar to Image Sculpting (Yenphraphai et al. 2024), OMG3D first segments the object and reconstruct a mesh in 3D. The mesh can then be mainuplated by 3D manipulation software. To address the loss of details in the rendered results, the authors propose a texture refinement module, named CustomRefiner, which performs gradient backpropagation directly to the UV texture map through differentiable rasterization. To achieve realistic lighting and shadow effects, the authors propose a lighting processing module, named IllumiCombiner, which estimates lighting from the background images and renders shadows that align with the scene.

strengths:
+ Compared with Image Sculpting, OMG3D can produce results showing better texture alignment with the original image and achieve realistic light and shadow effects.
+ The idea of gradient backpropagation to the UV texture map through differentiable rasterization sounds novel.
+ The idea of estimating a spherical light from the background image and applying it in the rendering pipeline to achieve realistic shading and shadow sounds logical.
+ The qualitative results look convincing.

weaknesses:
- A large part of the proposed pipeline comes from Image Sculpting. For instance, the "precise geometric control" is made possible by object segmentation followed by image-to-3D. The generative enhancement used in driving the UV-texture optimization also appears to be identical to that in Image Scuplting. This makes this work a bit incremental and lowers its novelty. Overall, this work can be regarded as an integration of Image Scultping (for 3D model manuipulation) and DiffusionLight (for introducing light and shadows effects).
- The key difference between this work and Image Sculpting is the replacing of direct rendered image enhancement with UV-texture optimization. However, the authors fail to discuss/analyze in detail why UV-texture optimization can produce better results than direct image enhancement.
- The light processing and plane creation part is not very clear.

questions:
- In Fig.4, there are two "(ours)". It seems that the one in the 2nd column is a typo.
- How is c_a in (6) used to correct the color of the estimated lighting? 
- What are I_d and I_c in (7)? How to adjust I_d "to maintain the object's saturation while ensuring that shadows remain evenly distributed in all direcionts"?
- How is the normal vector from the depth map being used in rendering shadows?
- Is the motion manually defined for the results of Image Scultping? How come the pumpkin jumping example shows lack of motion?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730678289530

thread_type:
official_review

<thread_object>
creator: reviewer_k8iy

<content>
summary:
This paper proposes a method for inserting 2D objects into 3D and letting users modify them. The method involves training a customized diffusion model added with a lighting processing module.

strengths:
They can combine precise geometric control.
They can handle better texture renderings.
They can handle lighting better.

They offer complete comparison to showcase their results with other state-of-the-art methods.

weaknesses:
They can try comparison with VSD loss. Or provide more visual examples in the supplementary results.

questions:
Have you tried to insert real humans instead of animation? Want to see that result.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730722195185

thread_type:
official_review

<thread_object>
creator: reviewer_ee9t

<content>
summary:
The paper introduces OMG3D, a framework for object manipulation in images that addresses both static editing and dynamic motion generation challenges, while also improving the realism of object appearance and lighting. OMG3D integrates precise geometric control with the generative capabilities of diffusion models, converting 2D objects into 3D for user-directed modifications and lifelike motions. To enhance texture realism, it proposes CustomRefiner, a module that refines textures using a customized diffusion model to match the style and perspective of the original image. Additionally, the authors introduces IllumiCombiner, a module that adjusts background lighting to create more realistic illumination, aligned with human visual perception. The authors conducted extensive experiments show that OMG3D achieves good performance in both static and dynamic applications.

strengths:
- The proposed method utilizes explicit 3D generation capability to ensure both static and dynamic manipulation. While other 2D-based methods fail to do so.
- The utilization of HDRi for realistic lighting.
- Visual quality is good.

weaknesses:
- Lack of technical novelty, most of the presented techniques exist. The proposed method seems to put them together nicely to produce a few good results. E.g., CustomRefiner is a combination of depth-controlnet, dreambooth lora, and differentiable rendering on UV map, IllumiCombiner is a combination of HDRi estimation and virtual-plane rendering.
- The idea of realistic lighting is interesting to me, but I am not convinced by the proposed method. The real light transport is more complex than by linearly modulated two terms. Solving global illumination is a very difficult problem with a single image. The proposed method can only handle simple objects.

Minors:
- L216: IlluminCombiner should be bold.

questions:
See above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732509070759

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Looking Forward to Your Feedback

comment:
Dear Reviewers,

We are highly grateful for your time and effort to review our work! We understand that you may have busy schedules, but we greatly value your feedback. And we are pleased that reviewers are excited about the novel contributions of our work. Reviewers remark that **“the idea of realistic lighting is interesting to me”**[ee9t],  **“they can combine precise geometric control, handle better texture renderings and handle lighting better.”**[k8iy],  **“the idea of UV texture map optimization sounds novel”**[oVkz], **“the paper is well written and the qualitative results are well made”**[LJb3]. We thank the reviewers for the strong praise of our work and contributions.

We now highlight a few important points raised by reviewers in the general response:

### Highlights of New Analyses and Experiments

In response to reviewers' comments, we have included these analyses and experiments in individual comments and revision PDF.  We now briefly describe these responses.

* **Explanation of core technical novelty and analyses of lighting methods** [ee9t]:  In our introduction of revision PDF, we have emphasized that the core contribution of our framework is the proposal of an application framework. Additionally, we have updated the related work section to include comparisons and analyses of existing lighting estimation methods.

* **More visual results and comparisons** [k8iy]: In the supplementary materials, we have added more visual results of human animation and provided additional comparisons with text-to-3D generation methods using VSD loss.

* **More analyses with Image Sculpting**[oVkz]: We have added more analyses in the comments and supplementary materials to highlight the differences in motivation and methodology between our approach and the Image Sculpting method, emphasizing the distinctions of our work and why our OMG3D works better.

* **Clearer descriptions of the lighting process and plane creation** [oVkz]: We have revised the descriptions of the lighting process and plane creation in the updated PDF to make them clearer and more precise.

* **More ablation study of techniques we used**[LJb3]: We have included additional ablation studies in the supplementary materials to better demonstrate the importance of the techniques used in our work.

### Enhanced Figure

We updated Figure 4 and Figure 5 to improve clarity. We updated Figure 7, Figure 8 and Figure 9 to show more ablation studies and visual results.

We thank all reviewers for their thoughtful commentary. We worked hard to improve our paper, and we sincerely hope the reviewers find our responses informative and helpful. If you feel the responses have not addressed your concerns to motivate increasing your score, we would love to hear what points of concern remain and how we can improve our work. Thank you again!


Best regards,

The Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a framework to manipulate single objects in existing images by geometry changes or animations. The method separates the image into foreground and background, before processing each part individually. From the background, the lighting is estimated to obtain an editable representation. The object is transformed into an explicit 3D representation that can be manually modified. In the end, the individual parts are composed into an image by a conditional diffusion model.

The main strengths of this work are the well-designed pipeline and good qualitative results. It seems that the method is able to capture changing lighting and geometry well and produces better results than the used baselines.

The main concern of me and also three out of four reviewers is the missing technical novelty. The individual parts of this work are taken and adapted from previous work. The main contribution of this work is mostly the overall applied system.

The reviews ended up with diverging borderline scores of 3/5/6/8, where the negatively leaning reviewers provided the stronger arguments in discussions. Since I agree with the arguments of lacking technical novelty, I tend to follow with a reject recommendation for this work. In case of rejection I strongly encourage the authors to submit this work to a more application focused venue, where it might be more positively received due to the high quality results.