SHAP-EDITOR: Instruction-guided Latent 3D Editing in Seconds
Authors: name: Minghao Chen; name: Junyu Xie; name: Iro Laina; name: Andrea Vedaldi
Published: 2023-12-14 18:59:06

Abstract: We propose a novel feed-forward 3D editing framework called Shap-Editor.
Prior research on editing 3D objects primarily concentrated on editing
individual objects by leveraging off-the-shelf 2D image editing networks. This
is achieved via a process called distillation, which transfers knowledge from
the 2D network to 3D assets. Distillation necessitates at least tens of minutes
per asset to attain satisfactory editing results, and is thus not very
practical. In contrast, we ask whether 3D editing can be carried out directly
by a feed-forward network, eschewing test-time optimisation. In particular, we
hypothesise that editing can be greatly simplified by first encoding 3D objects
in a suitable latent space. We validate this hypothesis by building upon the
latent space of Shap-E. We demonstrate that direct 3D editing in this space is
possible and efficient by building a feed-forward editor network that only
requires approximately one second per edit. Our experiments show that
Shap-Editor generalises well to both in-distribution and out-of-distribution 3D
assets with different prompts, exhibiting comparable performance with methods
that carry out test-time optimisation for each edited instance.