HyperFields: Towards Zero-Shot Generation of NeRFs from Text
Authors: name: Sudarshan Babu; name: Richard Liu; name: Avery Zhou; name: Michael Maire; name: Greg Shakhnarovich; name: Rana Hanocka
Published: 2023-10-26 00:36:03

Abstract: We introduce HyperFields, a method for generating text-conditioned Neural
Radiance Fields (NeRFs) with a single forward pass and (optionally) some
fine-tuning. Key to our approach are: (i) a dynamic hypernetwork, which learns
a smooth mapping from text token embeddings to the space of NeRFs; (ii) NeRF
distillation training, which distills scenes encoded in individual NeRFs into
one dynamic hypernetwork. These techniques enable a single network to fit over
a hundred unique scenes. We further demonstrate that HyperFields learns a more
general map between text and NeRFs, and consequently is capable of predicting
novel in-distribution and out-of-distribution scenes -- either zero-shot or
with a few finetuning steps. Finetuning HyperFields benefits from accelerated
convergence thanks to the learned general map, and is capable of synthesizing
novel scenes 5 to 10 times faster than existing neural optimization-based
methods. Our ablation experiments show that both the dynamic architecture and
NeRF distillation are critical to the expressivity of HyperFields.