Tailor3D: Customized 3D Assets Editing and Generation with Dual-Side Images

Authors: Zhangyang Qi, Yunhan Yang, Mengchen Zhang, Long Xing, Xiaoyang Wu, Tong Wu, Dahua Lin, Xihui Liu, Jiaqi Wang, Hengshuang Zhao
Primary area: generative models
Keywords: 3D Object Editing, 3D Generative Models

Abstract:
Recent advances in 3D AIGC have shown promise in directly creating 3D objects from text and images, offering significant cost savings in animation and product design. However, detailed edit and customization of 3D assets remains a long-standing challenge. Specifically, 3D Generation methods lack the ability to follow finely detailed instructions as precisely as their 2D image creation counterparts. Imagine you can get a toy through 3D AIGC but with undesired accessories and dressing. To tackle this challenge, we propose a novel pipeline called Tailor3D, which swiftly creates customized 3D assets from editable dual-side images. We aim to emulate a tailor's ability to locally change objects or perform overall style transfer. Unlike creating 3D assets from multiple views, using dual-side images eliminates conflicts on overlapping areas that occur when editing individual views. Specifically, it begins by editing the front view, then generates the back view of the object through multi-view diffusion. Afterward, it proceeds to edit the back views. Finally, a Dual-sided LRM is proposed to seamlessly stitch together the front and back 3D features, akin to a tailor sewing together the front and back of a garment. The Dual-sided LRM rectifies imperfect consistencies between the front and back views, enhancing editing capabilities and reducing memory burdens while seamlessly integrating them into a unified 3D representation with the LoRA Triplane Transformer. Experimental results demonstrate Tailor3D's effectiveness across various 3D generation and editing tasks, including 3D generative fill and style transfer. It provides a user-friendly, efficient solution for editing 3D assets, with each editing step taking only seconds to complete.

Decision: reject
Reviewer scores: 5, 3, 5, 6

Reviews and discussion:
<thread>
creation date:
1730323335601

thread_type:
official_review

<thread_object>
creator: reviewer_xaru

<content>
summary:
The paper introduced a practical system approach for 3D editing with the help of multi-view generations. The approach Tailor3D by generating the back view, then stitch the front and back view features for fine-tuning the reconstruction results. When testing on several datasets and benchmarks, the user studies showed the effectiveness of the approach, and it was quite competitive.

strengths:
This is a nice system paper with quite some strengths
- The overall presentation of the approach is accompanied by good visualizations and diagrams. I find it useful understanding the whole paper
- The technique is interesting and it nicely integrated several technologies such as LRM, LoRA, and efficient attention mechanisms to edit and combine feature representations of front and back views. 
- Overall, the result is competitive comparing to other approaches, it's nice to show user studies rather than automatically computed benchmark numbers in this paper

weaknesses:
I think this is a nice system paper with good quality results. There are strong merits of this paper but I have a few questions
- I think the approach is heavily bounded by the quality of back-view generation algorithms. I wonder if authors could comment on whether there are empirical evidences or ablation studies on this topic. E.g: if the back-view generation algorithm is not performing well, would the front & back view fusion approach recover the issues?
- The approach is limited to using 2 views (front and back), would using more views increase the complexity of the formulations in this paper?
- In eq. 4, could authors help elaborating on the theory of the equation, or point to sections of the LRM paper that explains this equation? I think it might sound intuitive but I wonder if this is mathematically correct.

questions:
My questions are listed in the weakness section
</content>
</thread_object>

</thread>
<thread>
creation date:
1730350245812

thread_type:
official_review

<thread_object>
creator: reviewer_sfh2

<content>
summary:
This manuscript introduces a framework for generating customized 3D assets, which integrates 2D image editing methods with a Dual-side LRM. Initially, the front view is edited using 2D editing techniques. Subsequently, the back view is generated by a multi-view diffusion model and can also be edited using the same 2D method. Both views are then processed by the Lora Triplane Transformer to create the 3D object. The effectiveness of this method has been validated through extensive applications.

strengths:
1. The proposed Dual-LRM and the editing of front and back views are technically sound. Editing these views without significant overlap effectively avoids inconsistent results. The proposed Dual-LRM can transform the two views into a 3D mesh.
2. The manuscript presents extensive editing results and an ablation study to validate each component within the framework.
3. The proposed LoRA Triplane Transformer and Fuse Double Side Feature effectively reduce the finetuning cost based on a pretrained LRM.

weaknesses:
1. Grammar typo in Line 71.
2. Use \citep for citation.
3. The motivation in the contribution section is not clear. Dual-LRM is proposed to avoid inconsistency in multi-view image editing results. Why is there still inconsistency between the front and back views?
4. Too few comparison methods. InstantMesh[1] and CRM[2] also leverage multi-view images as input for 3D generation.
5. The technical novelty is a minor concern. The 2D image editing and multi-view diffusion model are directly used. The dual-sided LRM can also be directly replaced with InstantMesh. I recommend a comparison with InstantMesh by inputting edited two views into InstantMesh and Dual-side LRM.

[1]Xu J, Cheng W, Gao Y, et al. Instantmesh: Efficient 3d mesh generation from a single image with sparse-view large reconstruction models[J]. arXiv preprint arXiv:2404.07191, 2024.

[2] Wang Z, Wang Y, Chen Y, et al. Crm: Single image to 3d textured mesh with convolutional reconstruction model[J]. arXiv preprint arXiv:2403.05034, 2024.

questions:
1. Compared to editing in a multi-view diffusion model, what are the advantages of editing in a single view? From my perspective, editing in a single view invariably incurs inconsistencies, though only front and back views are used, there is still an overlapped area for some objects.

2. Compared to training an LRM with only two views as input, what are the advantages of the proposed LoRA Triplane Transformer, except for memory cost? For example, how about the performance comparison? Moreover, since the LoRA Triplane Transformer is based on a pretrained LRM, how does the performance of directly training a Dual-side LRM from scratch compare?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730495508676

thread_type:
official_review

<thread_object>
creator: reviewer_5wus

<content>
summary:
This paper proposes Tailor3D, a method for fast 3D generation and editing. 
The core component of Tailor3D is a dual-side image LRM, which takes a dual-side image pair (front and back) as input, and output a 3D representation of the object (in Triplane).
Compared to original LRM, this dual-side LRM takes two images, instead of a single image or four images.
The motivation of using dual-side image is that, using two images, especially front and back views can provide convinience for users to edit while preventing critical inconsistencies from more views (like 4 views). 
This paper show extensive experiments about dual-side image LRM, including generation, editing, and very dense ablations.

strengths:
1.This paper introducing a very interesting sub-problem: given a pretrained LRM, how could we merge the the output triplane of dual-side image into a single triplane. This paper proposes a sound method to do so and shows the effectiveness in the experiments.
2. The ablation study in this paper is quite comprehensive, including the way of merging two triplanes, the rank of LORA, extrinsic parameters, etc.
Also, this paper provides the failure cases and analysis, which is very useful for understanding the model.
3. This paper is easy to read and follow.

weaknesses:
1. My biggest concern about this paper is that many considerations lack clear motivation, which makes me wonder whether some problems addressed are real or artificial.

(1) This paper emphasizes reducing the memory usage of LRM and therefore proposes using LoRA for a pretrained LRM. However, I don't see a clear reason why memory usage would be problematic here. Given that there are even four-view versions of LRM [DMV3D: DENOISING MULTI-VIEW DIFFUSION USING 3D LARGE RECONSTRUCTION MODEL] and this paper uses an A100 GPU, I don't understand why memory usage is a critical concern and whether LoRA is a necessary component for the model.

(2) Similarly, this paper proposes a novel method to merge two triplanes using cross-attention. I like this solution. However, I don't see a clear justification for this approach. In particular, why not simply take the features from both images and fine-tune the LRM model to generate a single triplane? This pipeline seems more natural and easier to implement. I may be missing some important considerations here.

(3) The dual-side image setting also seems somewhat unusual. While I agree that using two images can help eliminate inconsistencies across multiple views, it also constrains general editing capabilities. For example, if users want to edit the side of an object (between the front and back views), the current setting may be inconvenient. This raises the question: is the current setting really necessary? Could we instead use four views, edit in one of them, and fine-tune LRM to propagate these changes into triplanes?

It would be greatly appreciated if the authors could address these questions in their rebuttal. More importantly, the paper would benefit from clearer motivation and more detailed elaboration of these considerations to help readers better understand both the problems and proposed solutions.

2. Table 1 is confusing to me. What's the setting of single-image generation here? How could the proposed method achieve this task as the trained LRM takes dual-side images as input? If just providing single image, why this model outperforms the others so much as it should be similar to the original LRM with single image input?

3. I understand that training LRM may take a long time and many resources. But I still think it might be worthwhile to do some higher-level ablations to support some design considerations, mentioned in the first point of weakness. For example, could we train a LRM taking the image embedding from two images and directly outputs the triplane? Would that be better or worse compared to the current one? I think this paper does a great job on ablating the components in the current design, but some higher-level ablations would be really needed to support some decisions.

questions:
I have some questions listed in the weakness part. I would be very happy to change my rating given the answers of these questions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730531914455

thread_type:
official_review

<thread_object>
creator: reviewer_dnwz

<content>
summary:
This paper presents a method that allows users to customize 3D assets using 2D edits through a dual-side triplane-NeRF generation model that takes both front-view and back-view as input. And they perform thorough studies to evaluate the proposed method.

strengths:
* The input on both front-view and back-view seem reasonable and achieve good quality results.
* The evaluation seems thorough.

weaknesses:
* The method is a combination of existing methods and trained on a new data format that involves both front and back view. For me, it is unclear what is the major technical contribution.
* The quality of the generated 3D results are poor... I think this is due to the backend (LRM), but I still think it is worth pointing out that so many people nowaday relying on these poor 3D generative model for different purpose, but the reality might be that the quality of the 3D content generated by these models are just not good enough for real-world applications.

questions:
* Following the point mentioned in the weakness seciton, I wonder whether it is possible to extend or generalize the proposed method to the future 3D generative model? Since I think if not, then for real-world scenario, the proposed method is not useful at all because of limited 3D content quality bounded by the backbone model.
*
</content>
</thread_object>

</thread>
<thread>
creation date:
1732553869113

thread_type:
official_comment

<thread_object>
creator: area_chair_gm6q

<content>
title:
Please read the rebuttal and reply

comment:
Dear Reviewers,

Thanks again for serving for ICLR, the discussion period between authors and reviewers is approaching (November 27 at 11:59pm AoE), please read the rebuttal and ask questions if you have any. Your timely response is important and highly appreciated.

Thanks,

AC
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a method for rapid 3D object editing. The main idea is to leverage 2D editing models and 3D reconstruction models. Specifically, given a front view image as input, it first edits the image with editing methods, then synthesizes back-view image using MV diffusion models and edit it similarly, finally, the paper proposes a dual-side LRM to reconstruct the edited 3D objects. To extend the LRM to include back-view as input, the paper adopts LoRA as well as a fusing mechanism. 

This paper is well-written and easy to follow, the task is interesting and the authors have done comprehensive experiments to show the effectiveness of the proposed method.

For weakness, the main concerns raised by reviewers are
- the dependency on existing works and the limitation of the technical contribution of the paper. (Reviewer DNWZ, Sfh2, XARU) 
- the motivation of the current setting (e.g., if dual-side setting is reasonable?) (Reviewer 5Wus)
- missing comparisons (Reviewer Sfh2)

The authors have actively answered these questions with new experiments (see the Reviewer Discussion below), unfortunately reviewers are not fully convinced and three out of four reviewers lean to reject this paper. 

After reading the paper, reviews and rebuttals, AC agrees with the reviewers on the following points: the paper is dependent on existing modules and the motivation is not fully clear. Specifically, the editing part is done through off-the-shelf editing methods, making the contributions focus on the dual-side reconstruction method. Based on this, AC agrees with Reviewer 5Wus that the setting is questionable (or at least the claim of the paper may focus more on reconstruction instead of editing) and Reviewer Sfh2 that the reconstruction quality does matters and should be thoroughly compared with existing methods. 

To conclude, the paper is recommended for rejection for this time and the authors are encouraged to improve it based on all feedbacks.