SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 4: Qualitative Comparison of Generalization on ABO. The NeRFs are rendered at a resolution of  $512 \times 512$ . HyP-NeRF is able to preserve fine details such as the legs, creases, and texture even for novel instances. PixelNeRF fails to preserve details and to model the structure.
ABO Chairs	ABO Table	ABO Sofas
PSNR↑	SSIM↑	LPIPS↓	CD↓	PSNR↑	SSIM↑	LPIPS↓	CD↓	PSNR↑	SSIM↑	LPIPS↓	CD↓
[35]	35.43	0.96	0.07	-	34.07	0.95	0.07	-	33.87	0.95	0.08	-
Ours	31.37	0.94	0.1	0.0082	29.52	0.93	0.11	0.0033	30.32	0.94	0.11	0.0118
Table 2: Compression. We randomly sample 250 datapoints from our training dataset and compare the NeRFs learned using InstantNGP [35] on the individual instances against HyP-NeRF that learns the entire dataset. Note, we do not employ the denoising module (see Section 3.1) for this evaluation.
single-posed-view of novel NeRF instances. (2) Compression (Section 4.2): since HyP-NeRF is trained in an auto-decoding fashion on specific NeRF instances (see Equation (2)), we can evaluate the quality of the NeRFs compressed in this process. (3) Retrieval (Section 4.3): as shown in Figure 2, HyP-NeRF's prior enables various downstream applications. We show how to combine our prior with CLIP [41] to retrieve novel NeRFs.
Datasets and Comparisons. We primarily compare against two baselines, PixelNeRF [75] and InstantNGP [35] on the Amazon-Berkeley Objects (ABO) [11] dataset. ABO contains diverse and detailed objects rendered at a resolution of  $512 \times 512$  which is perfect to showcase the quality of the NeRF generated by HyP-NeRF. Rather than use a computationally expensive model like VisionNeRF (on the SRN [56] dataset) on a resolution of  $128 \times 128$ , we show our results on  $512 \times 512$  and compare with PixelNeRF. Additionally, we compare with the other baselines on SRN at  $128 \times 128$  resolution qualitatively in the main paper (Figure 5) and quantitatively in the supplementary. For compression, we directly compare with InstantNGP [35], that proposed MRHE, trained to fit on individual objects instance-by-instance.
Architectural Details. We use InstantNGP as  $f_{(\cdot)n}$ , with 16 levels, hastable size of  $2^{11}$ , feature dimension of 2, and linear interpolation for computing the MRHE; the MLP has a total of 5, 64-dimensional, layers. We observed that a hastable size  $2^{11}$  produces NeRF of high-quality at par with
Figure 5: Qualitative Comparison of Generalization on SRN on the task of single-view inversion (posed in our case) and compare the quality of the views rendered at  $128 \times 128$ . HyP-NeRF renders NeRFs of similar quality to the PixelNeRF and VisionNeRF baselines.
the size of  $2^{14}$ . Hence, we use  $2^{11}$  to speed up our training. Our hypernetwork,  $M$ , consists of 6 MLPs, 1 for predicting the MRHE, and the rest predicts the parameters  $\phi$  for each of the MLP layers of  $f$ . Each of the MLPs are made of 3, 512-dimensional, layers. We perform all of our experiments on NVIDIA RTX 2080Tis.
Metrics. To evaluate NeRF quality, we render them at 91 distinct views and compute metrics on the rendered images. Following PixelNeRF, we use PSNR(↑), SSIM(↑), and LPIPS(↓) [78]. Additionally, we compute Fréchet Inception Distance (FID)(↓) [17] to further test the visual quality. Although these metrics measure the quality of novel-view synthesis, they do not necessarily evaluate the geometry captured by the NeRFs. Therefore, we compute Chamfer's Distance (CD) whenever necessary by extracting a mesh from NeRF densities [30]. Please see the supplementary for additional det
ABO Chairs	ABO Sofa
Top 1	Top 3	Top 1	Top 3
98.72%	99.81%	91.6%	95.27%
Table 3: Retrieval. We design a simple query network (see Section 3.2) to retrieve NeRF instances from HyP-NeRF's prior seen at the time of training and achieve almost  $100\%$  accuracy.
4.1 Generalization
One way to evaluate if HyP-NeRF can render novel NeRF instances of high quality is through unconditional sampling. However, our learned prior  $\Phi$  is a non-standard prior (like a Gaussian distribution) and thus random sampling needs carefully designed mapping between such a known prior and  $\Phi$ . Therefore, we instead rely on a conditional task of single-view novel NeRF generation: given a single arbitrarily-chosen view of a novel object, we generate the corresponding NeRF,  $f_{(\cdot)}$ , through test-time optimization (see Section 3.2). We compare quantitatively with PixelNeRF on ABO at a high resolution of  $512 \times 512$  and qualitatively with the rest of the baselines on SRN at  $128 \times 128$ .
As shown in Table 1, we significantly outperform PixelNeRF on all of the metrics. Further, the qualitative results in Figure 4 clearly show the difference between the rendering quality of HyP-NeRF against PixelNeRF. Specifically, PixelNeRF fails to learn details, especially for the Sofa category. On the other hand, HyP-NeRF preserves intricate details like the texture, legs, and folds in the objects even at a high resolution. Further, we show our results on the widely used SRN dataset at the resolution of  $128 \times 128$  in Figure 5. Here, our quality is comparable with the baselines.
4.2 Compression
Unlike InstantNGP, which is trained on a single 3D instance, HyP-NeRF is trained on many NeRF instances which effectively results in the compression of these NeRFs into the latent space (or the codebook). We evaluate this compression capability by computing NeRF quality degradation compared to single-instance-only method, InstantNGP.
We randomly sample 250 instances from the training set and train InstantNGP separately on each of them. These samples are a subset of the training data used in HyP-NeRF's codebook. We show degradation metrics in Table 2. Note that we do not perform denoising on the generated NeRFs as we want to only evaluate the compression component of HyP-NeRF in this section. As can be seen in Table 2, there is a significant degradation in terms of PSNR (an average of  $11\%$ ), but the overall geometry is preserved almost as well as InstantNGP. However, InstantNGP is trained on a single instance, whereas we train on 1000s of NeRF instances (1038, 783, and 517 instances for ABO Chairs, Sofa, and Tables, respectively). This results in a  $60\times$  compression gain: for ABO Chairs,
with 1038 training instances, HyP-NeRF needs 163MB to store the model, whereas a single instance of InstantNGP needs on average 8.9MB. Note that we use the same network architecture [1] for HyP-NeRF and InstantNGP making this a fair comparison. Moreover, the storage complexity for InstantNGP-based NeRFs is linear with respect to the number of instances, whereas our degradation in visual quality is sublinear.
4.3 Retrieval
A generalizable learned prior has the ability to generate NeRFs based on different input modalities like text, images, segmented and occluded images, random noise, and multiview images. We now demonstrate additional querying and retrieval capabilities as described in Section 3.2.
This experiment's goal is to retrieve specific NeRF instances that HyP-NeRF has encountered during training from a single-view unposed image of that instance. Section 4.3 presents the number of times we could correctly retrieve from an arbitrary view of seen NeRF instances. We achieve almost  $100\%$  accuracy for Chair and Sofa datasets. However, we take this a step further and try to retrieve the closest training instance code corresponding to unseen views of seen instances taken from in-the-wild internet images. Figure 6 (top) shows examples from this experiment in which we are able to retrieve a NeRF closely matching the input query. This demonstrates the ease of design
ing a simple mapping network that can effectively interact with HyP-NeRF's prior.
Along with retrieving a seen instance, we use the query network to generate novel NeRFs of unseen instances as shown in Figure 6 (middle and bottom). In the middle row, we take an image of a cluttered scene, segment it with SAM [22], and pass this as input to the query network, from which we obtain a set of latent codes given as input to HyP-NeRF (see Figure 2). Finally, in the bottom row, we show text-to-NeRF capabilities enabled by HyP-NeRF.
4.4 Ablation
Two key designs of HyP-NeRF include incorporating the MRHE and the denoising network. We present the affect of removing these two components in Table 4 and Figure 1 for MRHE and Table 1, Figure 2, and Figure 4 for denoising. In the first ablation, we change the design of our neural network by using a hypernetwork to predict the parameters of a standard nef with positional encodings [32]. Since we remove the MRHE, we also increase the number of layers in the MLP to match the layers mentioned in [32]. Since there is a significant increase in the view rendering time, we randomly sam
ple 70 training examples for evaluation. As seen in Table 4, the quality of the rendered views lags significantly in all the metrics including the CD (measured against NeRFs rendered on InstantNGP individually). This is showcased visually in Figure 1 and the supplementary. Similarly, we find significant differences between the quality of the NeRFs before and after denoising (Table 1, Figure 2, and Figure 4), particularly in the Chair category with more diverse shapes.
Chairs
PSNR↑	SSIM↑	LPIPS↓	CD↓
HyP-NeRF	29.23	0.94	0.10	0.0075
w/o MRHE	26.42	0.92	0.16	0.0100
Table 4: Ablation of removing MRHE on ABO dataset. Due to the significant rendering time of HyP-NeRF w/o MRHE, we sample 70 object instances from the training dataset to compute the metrics at  $512 \times 512$  resolution.
Figure 6: Qualitative Comparison of Querying (Section 3.2) on HyP-NeRF's prior. In the top, we use an in-the-wild single-view unposed image to retrieve the closest NeRF HyP-NeRF has seen during training. In middle, we take a cluttered scene, and mask out the object of interest using Segment Anything [22] and in the bottom we use a text prompt as an input to our query network,  $\Delta$ . We then obtain the latent codes  $\{S,C\}$  from  $\Delta$ , which are used as an input for HyP-NeRF.
Figure 7: Qualitative results on color and geometry disentanglement: We take two instances from the training set and switch the geometry and color codes to generate novel instances. As can be seen, the geometry and the color are transferred while preserving fine shape details. Even the fine details, like stripes and color-contrast between the chair seats and edges from Chair-2, are accurately transferred to Chair-1 (Chair-2-Geometry + Chair-1-Color). Zoom in for an improved experience.
Figure 8: Latent visualization through TSNE plots on shape and color codes (from the codebooks  $S$  and  $C$ ). As can be seen, the underlying space forms meaningful clusters as shown through examples randomly sampled from two different clusters.
4.5 Color and Shape Disentanglement
We start with two object instances from the train set -  $A$  and  $B$  and denote their corresponding shape and geometry codes as  $A_{s}$ ,  $B_{s}$  and  $A_{g}$ ,  $B_{g}$ . We switch the geometry and shape code and generate two novel NeRFs given by  $A_{s}$ ,  $B_{g}$  and  $B_{s}$ ,  $A_{g}$ . In Figure 7, we can clearly see the disentanglement: geometry is perfectly preserved, and the color is transferred faithfully across the NeRFs. In Figure 8, we show clusters of color and shape codes using TSNE plots and visualize instances from the clusters.
We propose HyP-NeRF, a learned prior for Neural Radiance Fields (NeRFs). HyP-NeRF uses a hypernetwork to predict instance-specific multi-resolution hash encodings (MRHEs) that significantly improve the visual quality of the predicted NeRFs. To further improve the visual quality, we propose a denoising and finetuning technique that result in an improved NeRF that preserves its original multiview and geometric consistency. Experimental results demonstrate HyP-NeRF's capability to generalize to unseen samples and its effectiveness in compression. With its ability to overcome limitations of existing approaches, such as rendering at high resolution and multiview consistency, HyP-NeRF holds promise for various applications as we demonstrate for single- and multi-view NeRF reconstruction and text-to-NeRF.
Limitation and Future Work. One limitation of our work is the need for the pose to be known during test-time optimization (Section 3.2). Although we propose the query network to predict novel NeRFs conditioned on an unposed single view, the result may not exactly match the given view because of the loss of detail in the CLIP embedding. Future work should design a mapping network that can preserve fine details. An iterative pose refinement approach that predicts the pose along with the shape and color codes could also be adopted. A second limitation of our work is the non-standard prior  $\Phi$  that was learned by HyP-NeRF which makes unconditional generation challenging. GAN-based generative approaches solve this problem by randomly sampling from a standard distribution (like Gaussian distribution) and adversarially training the network. However, those methods often focus more on image quality than 3D structure. Future work could address this by incorporating latent diffusion models that can map a standard prior to HyP-NeRF's prior.