SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Bipasha Sen*
bise@mit.edu
Gaurav Singh*
IIIT, Hyderabad
gaurav.si
Aditya Agarwal*
adityaag@mit.edu
Rohith Agaram
rohith.agaram†
K Madhava Krishna
mkrishna@iit.ac.in
Srinath Sridhar
srinath@brown.edu
Figure 1: We propose HyP-NeRF, a latent conditioning method that learns improved quality NeRF priors using a hypernetwork to generate instance-specific multi-resolution hash encodings along with neural network weights. The figure showcases the fine details preserved in the NeRF generated by HyP-NeRF (green box) as opposed to the NeRF generated by naive conditioning (red box) in which, a hypernetwork predicts only the neural weights while relying on the standard positional encodings.
Neural Radiance Fields (NeRF) have become an increasingly popular representation to capture high-quality appearance and shape of scenes and objects. However, learning generalizable NeRF priors over categories of scenes or objects has been challenging due to the high dimensionality of network weight space. To address the limitations of existing work on generalization, multi-view consistency and to improve quality, we propose HyP-NeRF, a latent conditioning method for learning generalizable category-level NeRF priors using hypernetworks. Rather than using hypernetworks to estimate only the weights of a NeRF, we estimate both the weights and the multi-resolution hash encodings [35] resulting in significant quality gains. To improve quality even further, we incorporate a denoise and finetune strategy that denoises images rendered from NeRFs estimated by the hypernetwork and finetunes it while retaining multiview consistency. These improvements enable us to use HyP-NeRF as a generalizable prior for multiple downstream tasks including NeRF reconstruction from single-view or cluttered scenes, and text-to-NeRF. We provide
*Equal authors (order decided by a coin flip)
$\dagger$  @research.iit.ac.in
Neural fields, also known as implicit neural representations (INRs), are neural networks that learn a continuous representation of physical quantities such as shape or radiance at any given spacetime coordinate [70]. Recent developments in neural fields have enabled significant advances in applications such as 3D shape generation [77], novel view synthesis [32, 2], 3D reconstruction [72, 66, 38, 68], and robotics [52, 51]. In particular, we are interested in Neural Radiance Fields (NeRF) that learn the parameters of a neural network  $f_{\phi}(\mathbf{x}, \theta) = \{\sigma, c\}$ , where  $\mathbf{x}$  and  $\theta$  are the location and viewing direction of a 3D point, respectively, and  $\sigma$  and  $c$  denote the density and color estimated by  $f_{\phi}$  at that point. Once fully trained,  $f_{\phi}$  can be used to render novel views of the 3D scene.
Despite their ability to model high-quality appearance, NeRFs cannot easily generalize to scenes or objects not seen during training thus limiting their broader application. Typically, achieving generalization involves learning a prior over a data source such as image, video, or point cloud distributions [21, 20, 58, 76, 29, 49], possibly belonging to a category of objects [65, 46]. However, NeRFs are continuous volumetric functions parameterized by tens of millions of parameters making it challenging to learn generalizable priors. Previous works try to address this challenge by relying on 2D image-based priors, 3D priors in voxelized space, or by using latent conditioning.
Image-based priors re-use the information learned by 2D convolutional networks [75, 34] but may lack 3D knowledge resulting in representations that are not always multiview consistent. Methods that learn 3D priors in voxelized space [33] suffer from high compute costs and inherently lower quality due to voxelization limitations. Latent conditioning methods [19, 43] learn a joint network  $f(x, \theta, z)$  where  $z$  is the conditioning vector for a given object instance. These methods retain the advantages of native NeRF representations such as instance-level 3D and multiview consistency, but have limited capacity to model a diverse set of objects at high visual and geometric quality. InstantNGP [35] provides a way to improve quality and speed using instance-specific multi-resolution hash encodings (MRHE), however, this is limited to single instances.
We propose HyP-NeRF, a latent conditioning method for learning improved quality generalizable category-level NeRF priors using hypernetworks [15] (see Figure 1). We take inspiration from methods that use meta-learning to learn generalizable representations [55, 48] while retaining the quality of instance-specific methods [35]. Our hypernetwork is trained to generate the parameters—both the multi-resolution hash encodings (MRHE) and weights—of a NeRF model of a given category conditioned on an instance code  $z_{n}$ . For each instance code  $z_{n}$  in the learned codebook, HyP-NeRF estimates  $h_{n}$  denoting the instance-specific MRHE along with  $\phi_{n}$  indicating the weights of an MLP. Our key insight is that estimating both the MRHEs and the weights results in a significant improvement in quality. To improve the quality even further, we denoise rendered views [42] from the estimated NeRF model, and finetune the NeRF with the denoised images to enforce multiview consistency. As shown in Figure 2 and the experiments section, this denoising and finetuning step significantly improves
quality and fine details while retaining the original shape and appearance properties.
Figure 2: Once trained, HyP-NeRF acts as a prior to support multiple downstream applications, including NeRF reconstruction from single or multi-view images and cluttered scene images, and text-to-NeRF. We further improve quality using our denoising network.
Once HyP-NeRF is trained, it can be used as a NeRF prior in a variety of different applications such as NeRF reconstruction from a single view posed or unposed images, single pass text-to-NeRF,
qualitative comparisons and evaluate HyP-NeRF on three tasks: generalization, compression, and retrieval, demonstrating our state-of-the-art results.
3Project page: hyp-nerf.github.io
or even the ability to reconstruct real-world objects in cluttered scene images (see Figure 2). We show qualitative results on applications and quantitatively evaluate HyP-NeRF's performance and suitability as a NeRF prior on the ABO dataset [11] across three tasks: generalization, compression, and retrieval. To sum up our contributions:
1. We introduce HyP-NeRF, a method for learning improved quality NeRF priors using a hypernetwork that estimates instance-specific hash encodings and MLP weights of a NeRF.
2. We propose a denoise and finetune strategy to further improve the quality while preserving the multiview consistency of the generated NeRF.
3. We demonstrate how our NeRF priors can be used in multiple downstream tasks including single-view NeRF reconstruction, text-to-NeRF, and reconstruction from cluttered scenes.
Neural Radiance Fields [32] (NeRFs) are neural networks that capture a specific 3D scene or object given sufficient views from known poses. Numerous follow-up work (see [62, 70] for a more comprehensive review) has investigated improving quality and speed, relaxing assumptions, and building generalizable priors. Strategies for improving quality or speed include better sampling [2], supporting unbounded scenes [3], extensions to larger scenes [69, 60], using hybrid representations [35, 74], using learned initializations [61, 5, 45], or discarding neural networks completely [73, 59]. Other work relaxes assumption of known poses [67, 31, 27, 25, 9, 53], or reduce the number of views [75, 43, 4, 14, 34, 44, 71, 28, 37]. Specifically, PixelNeRF [75] uses convolution-based image features to learn priors enabling NeRF reconstruction from as few as a single image. VisionNeRF [28] extends PixelNeRF by augmenting the 2D priors with 3D representations learned using a transformer. Unlike these methods, we depend purely on priors learned by meta-learning, specifically by hypernetworks [15]. AutoRF [34] and LolNeRF [43] are related works that assume only a single view for each instance at the training time. FWD [5] optimizes NeRFs from sparse views in real-time and SRT [45] aims to generate NeRFs in a single forward pass. These methods produce NeRFs of lower quality and are not designed to be used as priors for various downstream tasks. In contrast, our focus is to generate high-quality multiview consistent NeRFs that capture fine shapes and textures details. HyP-NeRF can be used as a category-level prior for multiple downstream tasks including NeRF reconstruction from one or more posed or unposed images, text-to-NeRF (similar to [40, 18]), or reconstruction from cluttered scene images. Additionally, HyP-NeRF can estimate the NeRFs in a single forward pass with only a few iterations needed to improve the quality. Concurrent to our work, NerfDiff [13] and SSDNeRF [8] achieve high quality novel view synthesis by using diffusion models.
Learning 3D Priors. To learn category-level priors, methods like CodeNeRF [19] and LolNeRF [43] use a conditional NeRF on instance vectors  $z$  given as  $f(\mathbf{x},\theta ,z)$ , where different  $zs$  result in different NeRFs. PixelNeRF [75] depends on 2D priors learned by 2D convolutional networks which could result in multi-view inconsistency. DiffRf [33] uses diffusion to learn a prior over voxelized radiance field. Like us, DiffRF can generate radiance fields from queries like text or images. However, it cannot be directly used for downstream tasks easily.
Our approach closely follows the line of work that aims to learn a prior over a 3D data distribution like signed distance fields [39], light field [55], and videos [48]. We use meta-learning, specifically hypernetworks [15], to learn a prior over the MRHEs and MLP weights of a fixed NeRF architecture. LearnedInit [61], also employs standard meta-learning algorithms for getting a good initialization of the NeRF parameters. However, unlike us, they do not use a hypernetwork, and use the meta-learning algorithms only for initializing a NeRF, which is further finetuned on the multiview images. Methods like GRAF [47],  $\pi$ -GAN [6], CIPS-3D [79], EG3D [7], and Pix2NeRF [4] use adversarial training setups with 2D discriminators resulting in 3D and multiview inconsistency. [40, 64, 16] tightly couple text and NeRF priors to generate and edit NeRFs based on text inputs. We, on the other hand, train a 3D prior on NeRFs and separately train a mapping network that maps text to HyP-NeRF's prior, decoupling the two.
3 HyP-NeRF: Learning Improved NeRF prior using a Hypernetwork
Our goal is to learn a generalizable NeRF prior for a category of objects while maintaining visual and geometric quality, and multiview consistency. We also want to demonstrate how this prior can be
used to enable downstream applications in single/few-image NeRF generation, text-to-NeRF, and reconstruction of real-world objects in cluttered scenes.
Background. We first provide a brief summary of hypernetworks and multi-resolution hash encodings that form the basis of HyP-NeRF. Hypernetworks are neural networks that were introduced as a meta-network to predict the weights for a second neural network. They have been widely used for diverse tasks, starting from representation learning for continuous signals [55, 54, 57, 48], compression [36, 12], few-shot learning [50, 23], continual learning [63]. Our key insight is to use hypernetworks to generate both the network weights and instance-specific MRHEs.
Neural Radiance Fields (NeRF) [32, 2] learn the parameters of a neural network  $f_{\phi}(\mathbf{x}, \theta) = \{\sigma, c\}$ , where  $\mathbf{x}$  and  $\theta$  are the location and viewing direction of a 3D point, respectively, and  $\sigma$  and  $c$  denote the density and color predicted by  $f_{\phi}$  at that point. Once fully trained,  $f_{\phi}$  can be used to render novel views of the 3D scene. NeRF introduced positional encodings of the input 3D coordinates,  $\mathbf{x}$ , to a higher dimensional space to capture high-frequency variations in color and geometry. InstantNGP [35] further extended this idea to instance-specific multi-resolution hash encodings (MRHE) to encode  $\mathbf{x}$  dynamically based on scene properties. These MRHEs,  $h$ , are learned along with the MLP parameters,  $\phi$  for a given NeRF function,  $f$  and show improved quality and reduced training/inference time.
Image Denoising is the process of reducing the noise and improving the perceptual quality of images while preserving important structural details. Recent advancements in deep learning-based image restoration and denoising techniques [26, 24, 10] have demonstrated remarkable success in removing noise and enhancing the perceptual quality of noisy images that may have suffered degradation. Such networks are trained on large datasets of paired noisy and clean images to learn a mapping between the degraded input and the corresponding high-quality output by minimizing the difference between the restored and the ground truth clean image. In our case, we use denoising to improve the quality of our NeRF renderings by reducing artifacts and improving the texture and structure at the image level.
Given a set of NeRFs denoted by  $\{f_{(\phi_n,h_n)}\}_{n = 1}^N$ , where  $N$  denotes the number of object instances in a given object category, we want to learn a prior  $\Phi = \{\Phi_S,\Phi_C\}$ , where  $\Phi_S$  and  $\Phi_C$  are the shape and color priors, respectively. Each NeRF,  $f_{(\cdot)_n}$ , is parameterized by the neural network weights,  $\phi_{n}$  and learnable MRHEs,  $h_i$  as proposed in [35].  $f_{(\cdot)_n}$  takes a 3D position,  $\mathbf{x}$ , and viewing direction,  $\theta$ , as input and predicts the density conditioned on  $\mathbf{x}$  denoted by  $\sigma_n^{\{\mathbf{x}\}}$ , and color conditioned on  $\mathbf{x}$  and  $\theta$  denoted by  $c_n^{\{\mathbf{x},\theta\}}$ . This is given as,
$$
f _ {\left(\phi_ {n}, h _ {n}\right)} (\mathbf {x}, \theta) = \left\{\sigma_ {n} ^ {\{\mathbf {x} \}}, c _ {n} ^ {\{\mathbf {x}, \theta \}} \right\}. \tag {1}
$$
Our proposed method for learning NeRF priors involves two steps. First, we train a hypernetwork,  $M$ , to learn a prior over a set of multiview consistent NeRFs of high-quality shape and texture. Second, we employ an image-based denoising network that takes as input an already multiview consistent set of images, rendered from the predicted NeRF, and improves the shape and texture of NeRF to higher quality by finetuning on a set of denoised images. Our architecture is outlined in Figure 3 and we explain each step in detail below.
Step 1: Hypernetwork for Learning NeRF Prior. We want to design our hypernetwork,  $M$ , with trainable parameters,  $\Omega$  that can predict NeRF parameters  $\{\phi_n, h_n\}$  given a conditioning code  $z_n = \{S_n, C_n\}$ , where  $S_n$  and  $C_n$  are the shape and color codes, respectively, for an object instance  $n$  belonging to a specific category. Here,  $S_n$  and  $C_n$  belong to codebooks,  $S$  and  $C$  that are trained along with  $\Omega$  in an auto-decoding fashion.
As shown in Figure 3 (top), ideally we want  $M$  to learn a prior  $\{\Phi_C, \Phi_S\}$  over  $S$  and  $C$  such that given a random set of codes,  $\{\mathcal{Y}_S \sim \Phi_S, \mathcal{Y}_C \sim \Phi_C\}$ ,  $M$  should be able to generate a valid NeRF with consistent shape and texture for the given category of objects. To achieve this, we train  $M$  by assuming the same constraints as are needed to train a NeRF - a set of multiview consistent images  $\mathbf{I} = \{\{I_{\theta \in \Theta}\}_n\}_{n=1}^N$  for a set of poses,  $\Theta$ . In each training step, we start with a random object instance,  $n$ , and use the corresponding codes  $S_n$  and  $C_n$  from the codebooks as an input for  $M$ . Our key insight is that estimating both the MRHEs and MLP weights results in a higher quality than other alternatives.  $M$  then predicts the NeRF parameters  $\{\phi_n, h_n\}$ , which is then used to minimize the
Figure 3: Architecture Diagram: HyP-NeRF is trained and inferred in two steps. In the first step (top), our hypernetwork,  $M$ , is trained to predict the parameters of a NeRF model,  $f_{n}$  corresponding to object instance  $n$ . At this stage, the NeRF model acts as a set of differentiable layers to compute the volumetric rendering loss, using which  $M$  is trained on a set of  $N$  objects, thereby learning a prior  $\Phi = \{\Phi_{S},\Phi_{C}\}$  over the shape and color codes given by  $S$  and  $C$ , respectively. In the second step (bottom), the quality of the predicted multiview consistent NeRF,  $f_{n}$ , is improved using a denoising network trained directly in the image space. To do this,  $f_{n}$  is rendered from multiple known poses to a set of images that are improved to photorealistic quality.  $f_{n}$  is then finetuned on these improved images. Importantly, since  $f_{n}$  is only finetuned and not optimized from scratch, and thus  $f_{n}$  retains the multiview consistency whilst improving in terms of texture and shape quality. following objective:
$$
\mathcal {L} \left(\Omega , S _ {n}, C _ {n}\right) = \sum_ {\mathbf {r} \in R} \left| \left| \mathbf {V} ^ {\prime} \left(\mathbf {r}, \left\{\sigma_ {n} ^ {\left\{x _ {i} ^ {\mathbf {r}} \right\}}, c _ {n} ^ {\left\{x _ {i} ^ {\mathbf {r}}, \theta \right\}} \right\} _ {i = 1} ^ {L}\right) - \mathbf {V} _ {n} (\mathbf {r}) \right| \right| \tag {2}
$$
$$
\left\{\sigma_ {n} ^ {\left\{x _ {i} ^ {\mathbf {r}} \right\}}, c _ {n} ^ {\left\{x _ {i} ^ {\mathbf {r}}, \theta \right\}} \right\} = f _ {\left(\phi_ {n}, h _ {n}\right)} \left(x _ {i} ^ {\mathbf {r}}, \theta\right) \quad \text {a n d} \quad \left\{\phi_ {n}, h _ {n} \right\} = M _ {\Omega} \left(S _ {n}, C _ {n}\right) \tag {3}
$$
where  $\mathbf{V}'$  denotes the volumetric rendering function as given in [32] eqn. 3 and 5,  $\mathbf{r}$  is a ray projected along the camera pose  $\theta$ ,  $x_{i}^{\mathbf{r}} \in \mathbf{x}$  and  $L$  denote the number of points sampled along  $\mathbf{r}$ , and  $\mathbf{V}_n$  denote the ground truth value for projection of the  $n^{\mathrm{th}}$  object along  $\mathbf{r}$ .
Note that, in this step, the only trainable parameters are the meta-network weights,  $\Omega$ , and the codebooks  $S$  and  $C$ . In this setting, the NeRF functions  $f_{(\cdot)n}$  only act as differentiable layers that allow backpropagation through to  $M$  enabling it to train with multiview consistency loss attained by the volumetric rendering loss as described in [32]. We use an instantiation of InstantNGP [35] as our function  $f_{(\cdot)n}$  consisting of MRHE and a small MLP.
A general limitation of hypernetworks arises from the fact that the intended output space (i.e. the space of valid MLP weight matrices) is a subset of the actual output space, which is unrestricted and can be any 2D matrix. Thus, a hypernetwork trained on loss functions in the weight space can result in unstable training, and might require a lot of training examples to converge. To overcome this issue, we train our hypernetwork end-to-end directly on images, so that it learns the implicit NeRF space along with the category specific prior on it, which simplifies the setting for the hypernetwork and allows for more stable training. As a causal effect of this, HyP-NeRF, when trained on less number of examples, essentially acts as a compressing model.
Step 2: Denoise and Finetune. In the first step,  $M$  is trained to produce a consistent NeRF with high-fidelity texture and shape. However, we observed that there is room to improve the generated
Chairs	Sofa
PSNR↑	SSIM↑	LPIPS↓	FID↓	PSNR↑	SSIM↑	LPIPS↓	FID↓
ABO-512	PixelNeRF [75]	18.30	0.83	0.31	292.32	17.51	0.84	0.28	323.89
CodeNeRF [19]	19.86	0.87	0.298	-	19.56	0.87	0.290	-
HyP-NeRF (Ours)	24.23	0.91	0.16	68.11	23.96	0.90	0.18	120.80
w/o Denoise	23.05	0.90	0.16	102.45	23.54	0.90	0.174	121.69
Table 1: Generalization. Comparison of single-posed-view NeRF generation. Metrics are computed on renderings of resolution  $512 \times 512$ . HyP-NeRF significantly outperforms PixelNeRF and CodeNeRF on all the metrics in both the datasets.
NeRFs to better capture fine details like uneven textures and edge definition. To tackle this challenge, we augment  $M$  using a denoising process that takes  $f_{(\cdot)n}$  and further finetunes it to achieve  $f_{(\cdot)_n}^H$ .
As shown in Figure 3 (bottom), we render novel views from the multiview consistent NeRF into  $m$  different predefined poses given by  $\{\theta_1, \theta_2 \dots \theta_m\}$  to produce a set of multiview consistent images  $\{\hat{I}_i\}_{i=1}^m$ . We then use a pre-trained image-level denoising autoencoder that takes  $\{\hat{I}_i\}_{i=1}^m$  as input and produces images of improved quality given as  $\{\hat{I}_i^H\}_{i=1}^m$ . These improved images are then used to finetune  $f_{(\cdot)_n}$  to achieve  $f_{(\cdot)_n}^H$ . Note that, we do not train the NeRFs from scratch on  $\{\hat{I}^H\}$  and only finetune the NeRFs, which ensures fast optimization and simplifies the task of the denoising module that only needs to improve the quality and does not necessarily need to maintain the multiview consistency. While our denoising is image-level, we still obtain multiview consistent NeRFs since we finetune on the NeRF itself (as we also demonstrate through experiments in the supplementary).
For our denoising autoencoder, we use VQVAE2 [42] as the backbone. To train this network, we simply use images projected from the NeRF, predicted by the hypernetwork (lower quality relative to the ground truth) as the input to the VQVAE2 model. We then train VQVAE2 to decode the ground truth by minimizing the L2 loss objective between VQVAE2's output and the ground truth.
3.2 HyP-NeRF Inference and Applications
Training over many NeRF instances,  $M$  learns a prior  $\Phi$  that can be used to generate novel consistent NeRFs. However,  $\Phi$  is not a known distribution like Gaussian distributions that can be naively queried by sampling a random point from the underlying distribution. We tackle this in two ways:
Test Time Optimization. In this method, given a single-view or multi-view posed image(s), we aim to estimate shape and color codes  $\{S_o,C_o\}$  of the NeRF that renders the view(s). To achieve this, we freeze  $M$  's parameters and optimize the  $\{S_o,C_o\}$  using the objective given in Equation (2).
Query Network. We create a query network,  $\Delta$ , that maps a point from a known distribution to  $\Phi$ . As CLIP's [41] pretrained semantic space, say  $\mathbf{C}$ , is both text and image aware, we chose  $\mathbf{C}$ , as our known distribution and learn a mapping function  $\Delta(z \sim \mathbf{C}) \to \Phi$ . Here,  $\Delta$  is an MLP that takes  $z$  as input and produces  $\mathcal{Y}_z \in \Phi$  as output. To train  $\Delta$ , we randomly sample one pose from the ground truth multiview images  $I_\theta^n \in \{I_{\theta \in \Theta}\}_n$  and compute the semantic embedding  $z_\theta^n = \mathrm{CLIP}(I_\theta^n)$  and map it to  $\{\bar{S}_n, \bar{C}_n\} \in \Phi$  given as  $\{\bar{S}_n, \bar{C}_n\} = \Delta(z_\theta^n)$ . We then train our query network by minimizing the following objective:
$$
\mathcal {L} _ {\Delta} = \sum_ {\theta} | | \{\bar {S} _ {n}, \bar {C} _ {n} \}, \{S _ {n}, C _ {n} \} | |. \tag {4}
$$
At the time of inference, given a text or image modality such as a text prompt, single-view unposed (in-the-wild) image, or segmented image, we compute the semantic embedding using CLIP encoder and map it to  $\Phi$  using  $\Delta$ , from which we obtain the shape and color codes as input for the HyP-NeRF.
Note that for N query points in a scene, the forward pass through the hypernetwork (computationally expensive) happens only once per scene. Only the NeRF predicted by the hypernetwork (computationally less expensive) is run for each query point.
We provide evaluations of the prior learned by HyP-NeRF specifically focusing on the quality of the generated NeRFs. We consider three dimensions: (1) Generalization (Section 4.1): we validate whether HyP-NeRF can generate novel NeRFs not seen during training by conditioning on only a