FirePlace: Geometric Refinements of LLM Common Sense Reasoning for 3D
  Object Placement
Authors: name: Ian Huang, name: Yanan Bao, name: Karen Truong, name: Howard Zhou, name: Cordelia Schmid, name: Leonidas Guibas, name: Alireza Fathi

Scene generation with 3D assets presents a complex challenge, requiring both
high-level semantic understanding and low-level geometric reasoning. While
Multimodal Large Language Models (MLLMs) excel at semantic tasks, their
application to 3D scene generation is hindered by their limited grounding on 3D
geometry. In this paper, we investigate how to best work with MLLMs in an
object placement task. Towards this goal, we introduce a novel framework,
FirePlace, that applies existing MLLMs in (1) 3D geometric reasoning and the
extraction of relevant geometric details from the 3D scene, (2) constructing
and solving geometric constraints on the extracted low-level geometry, and (3)
pruning for final placements that conform to common sense. By combining
geometric reasoning with real-world understanding of MLLMs, our method can
propose object placements that satisfy both geometric constraints as well as
high-level semantic common-sense considerations. Our experiments show that
these capabilities allow our method to place objects more effectively in
complex scenes with intricate geometry, surpassing the quality of prior work.