PF-LRM: Pose-Free Large Reconstruction Model for Joint Pose and Shape
  Prediction
Authors: name: Peng Wang; name: Hao Tan; name: Sai Bi; name: Yinghao Xu; name: Fujun Luan; name: Kalyan Sunkavalli; name: Wenping Wang; name: Zexiang Xu; name: Kai Zhang
Published: 2023-11-20 18:57:55

Abstract: We propose a Pose-Free Large Reconstruction Model (PF-LRM) for reconstructing
a 3D object from a few unposed images even with little visual overlap, while
simultaneously estimating the relative camera poses in ~1.3 seconds on a single
A100 GPU. PF-LRM is a highly scalable method utilizing the self-attention
blocks to exchange information between 3D object tokens and 2D image tokens; we
predict a coarse point cloud for each view, and then use a differentiable
Perspective-n-Point (PnP) solver to obtain camera poses. When trained on a huge
amount of multi-view posed data of ~1M objects, PF-LRM shows strong
cross-dataset generalization ability, and outperforms baseline methods by a
large margin in terms of pose prediction accuracy and 3D reconstruction quality
on various unseen evaluation datasets. We also demonstrate our model's
applicability in downstream text/image-to-3D task with fast feed-forward
inference. Our project website is at: https://totoro97.github.io/pf-lrm .