Differentiable Blocks World: Qualitative 3D Decomposition by Rendering
  Primitives
Authors: name: Tom Monnier; name: Jake Austin; name: Angjoo Kanazawa; name: Alexei A. Efros; name: Mathieu Aubry
Published: 2023-07-11 17:58:31

Abstract: Given a set of calibrated images of a scene, we present an approach that
produces a simple, compact, and actionable 3D world representation by means of
3D primitives. While many approaches focus on recovering high-fidelity 3D
scenes, we focus on parsing a scene into mid-level 3D representations made of a
small set of textured primitives. Such representations are interpretable, easy
to manipulate and suited for physics-based simulations. Moreover, unlike
existing primitive decomposition methods that rely on 3D input data, our
approach operates directly on images through differentiable rendering.
Specifically, we model primitives as textured superquadric meshes and optimize
their parameters from scratch with an image rendering loss. We highlight the
importance of modeling transparency for each primitive, which is critical for
optimization and also enables handling varying numbers of primitives. We show
that the resulting textured primitives faithfully reconstruct the input images
and accurately model the visible 3D points, while providing amodal shape
completions of unseen object regions. We compare our approach to the state of
the art on diverse scenes from DTU, and demonstrate its robustness on real-life
captures from BlendedMVS and Nerfstudio. We also showcase how our results can
be used to effortlessly edit a scene or perform physical simulations. Code and
video results are available at https://www.tmonnier.com/DBW .