What You See is What You GAN: Rendering Every Pixel for High-Fidelity
  Geometry in 3D GANs
Authors: name: Alex Trevithick; name: Matthew Chan; name: Towaki Takikawa; name: Umar Iqbal; name: Shalini De Mello; name: Manmohan Chandraker; name: Ravi Ramamoorthi; name: Koki Nagano
Published: 2024-01-04 18:50:38

Abstract: 3D-aware Generative Adversarial Networks (GANs) have shown remarkable
progress in learning to generate multi-view-consistent images and 3D geometries
of scenes from collections of 2D images via neural volume rendering. Yet, the
significant memory and computational costs of dense sampling in volume
rendering have forced 3D GANs to adopt patch-based training or employ
low-resolution rendering with post-processing 2D super resolution, which
sacrifices multiview consistency and the quality of resolved geometry.
Consequently, 3D GANs have not yet been able to fully resolve the rich 3D
geometry present in 2D images. In this work, we propose techniques to scale
neural volume rendering to the much higher resolution of native 2D images,
thereby resolving fine-grained 3D geometry with unprecedented detail. Our
approach employs learning-based samplers for accelerating neural rendering for
3D GAN training using up to 5 times fewer depth samples. This enables us to
explicitly "render every pixel" of the full-resolution image during training
and inference without post-processing superresolution in 2D. Together with our
strategy to learn high-quality surface geometry, our method synthesizes
high-resolution 3D geometry and strictly view-consistent images while
maintaining image quality on par with baselines relying on post-processing
super resolution. We demonstrate state-of-the-art 3D gemetric quality on FFHQ
and AFHQ, setting a new standard for unsupervised learning of 3D shapes in 3D
GANs.