TIP-Editor: An Accurate 3D Editor Following Both Text-Prompts And
  Image-Prompts
Authors: name: Jingyu Zhuang; name: Di Kang; name: Yan-Pei Cao; name: Guanbin Li; name: Liang Lin; name: Ying Shan
Published: 2024-01-26 12:57:05

Abstract: Text-driven 3D scene editing has gained significant attention owing to its
convenience and user-friendliness. However, existing methods still lack
accurate control of the specified appearance and location of the editing result
due to the inherent limitations of the text description. To this end, we
propose a 3D scene editing framework, TIPEditor, that accepts both text and
image prompts and a 3D bounding box to specify the editing region. With the
image prompt, users can conveniently specify the detailed appearance/style of
the target content in complement to the text description, enabling accurate
control of the appearance. Specifically, TIP-Editor employs a stepwise 2D
personalization strategy to better learn the representation of the existing
scene and the reference image, in which a localization loss is proposed to
encourage correct object placement as specified by the bounding box.
Additionally, TIPEditor utilizes explicit and flexible 3D Gaussian splatting as
the 3D representation to facilitate local editing while keeping the background
unchanged. Extensive experiments have demonstrated that TIP-Editor conducts
accurate editing following the text and image prompts in the specified bounding
box region, consistently outperforming the baselines in editing quality, and
the alignment to the prompts, qualitatively and quantitatively.