HyP-NeRF: Learning Improved NeRF Priors using a HyperNetwork
Authors: Bipasha Sen
Gaurav Singh
Aditya Agarwal
Rohith Agaram
Madhava Krishna
Srinath Sridhar
Venue: NeurIPS 2023 poster (2023)
Keywords: neural radiance field
hypernetwork
multi-hash encoding
NeRF
TL;DR: HyP-NeRF is a latent conditioning method for learning generalizable category-level NeRF priors using hypernetworks to estimate NeRF weights and multi-resolution hash encodings. A denoise strategy further improves the quality of NeRFs.

Abstract: Neural Radiance Fields (NeRF) have become an increasingly popular representation to capture high-quality appearance and shape of scenes and objects. However, learning generalizable NeRF priors over categories of scenes or objects has been challenging due to the high dimensionality of network weight space. To address the limitations of existing work on generalization, multi-view consistency and to improve quality, we propose HyP-NeRF, a latent conditioning method for learning generalizable category-level NeRF priors using hypernetworks. Rather than using hypernetworks to estimate only the weights of a NeRF, we estimate both the weights and the multi-resolution hash encodings resulting in significant quality gains. To improve quality even further, we incorporate a denoise and finetune strategy that denoises images rendered from NeRFs estimated by the hypernetwork and finetunes it while retaining multiview consistency. These improvements enable us to use HyP-NeRF as a generalizable prior for multiple downstream tasks including NeRF reconstruction from single-view or cluttered scenes and text-to-NeRF. We provide qualitative comparisons and evaluate HyP-NeRF on three tasks: generalization, compression, and retrieval, demonstrating our state-of-the-art results.

HyP-NeRF: Learning Improved NeRF Priors using a HyperNetwork
Bipasha Sen MIT CSAIL
Gaurav Singh* IIIT, Hyderabad gaurav.sij
Aditya Agarwal' MIT CSAIL adityaagdmit.cmm edu
Rohith Agaram III,, Hyderabad rohith agaram'
bise@mit.. edu
K Madhava Krishna III,, Hyderabad mkri cchna@iiit ac. in
Srinath Sridhar
Brown University
srinath@brown.ede
HyP-NeRF
Colebbok
Ours
Nalve Latent wowddiiiiooooo soonndllioing
Mull-Rasolltion Hash Encodings
Mub--ayered Perceptron
We propose HyP-NeRF. latent conditioning method that learns improved quality NeRF priors using a hypernetwork to generate instance-speciffe multi-resolution hash encodings along with neural network weights. The figure showcases the fine details preserved in the NeRF generated by HyP-NeRF (green box) as opposed to the NeRF generated by naive conditioning (red box) in which, hypernetwork predicts only the neural weights while relying the standard positional encodings.
Figure 1:
Abstract
Neural Radiance Fields (NeRF) have become increasingly popular representation to capture high-quality appearance and shape of scenes and objects. However, learning generalizable NeRF priors over categories of scenes or objects has been challenging due to the high dimensionality of network weight space. To address the limitations of existing work on generalization, multi-view consistency and improve quality, we propose HyP-NeRFF latent conditioning method for learning generalizable category-level NeRF priors using hypernetworks. Rather than using hypernetworks estimate only the weights of NeRF, we estimate both the weights and the multi-resolution hash encodings 35 resulting significant quality gains. To improve quality even further, we incorporate denoise and finetune strategy that denoises images rendered from NeRFs estimated by the hypernetwork and finetunes it while retaining multiview consistency. These improvements enable to use HyP-NeRF as a generalizable prior for multiple downstream tasks including NeRF reconstruction from single-view cluttered scenes, and text-to-NeRF. We provide
Equual authors (order decided by coin flip)
@ messarch.iiit.cchn
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

qualitative comparisons and evaluate HyP-NeeR on three tasks; generalization, compression, and retrieval, demonstrating our state-of-the-ar results.
Introduction
Neural fields, also known as implicit neural representations (INRs), are neural networks that learn continuous representation of physical quantities such shape radiance any given space- neural fields have enabled significant advances in
time coordinate [70]. Recent developments
applications such as 3D shape generation 777,, novel view synthesis B22 2 3D reconstruction [72 66 [38] (8), and robotics [52 [1]. In particular, we are interested in Neural Radiance Fields (NeRF)
that learn the parameters of neural network [x(x, 0) {o, c). where and are the location and viewing direction of a 3D point, respectively, and a and e denote the density and color estimated by
fa at that point. Once fully trained, f can be used to render novel views of the 3D scene.
Despite their ability to model high-quality appearance, NeRFs cannot easily generalize to scenes objects not seen during training thus limiting their broader application. Typically, achieving generalization involves learning a prior over a data source such as image, video, or point cloud distributions [2] 20
29 491, possibly belonging to category of objects [65] 46]. However, NeRFs are continuous volumetric functions parameterized by tens of millions of parameters making it challenging to learn generalizable priors. Previous works try to address this challenge by relying 2D image-based priors, 3D priors in voxelized space, or by using latent conditioning.
58
76
Image-based priors re-use the information learned by 2D convolutional networks 75 but may lack 3D knowledge resulting in representations that are not always multiview consistent. Methods that learn 3D priors in voxelized space [33] suffer from high compute costs and inherently lower quality due to voxelization limitations. Latent conditioning methods 19 43 learn a joint network f(x, 0, where z is the conditioning vector for given object instance. These methods retain the advantages of native NeRF representations such as instance-level 3D and multiview consistency, but have limited capacity model diverse set of objects high visual and geometric quality. InstantNGP [35] provides a way to improve quality and speed using instance-specific multi reltliirsol hash encodings (MRHE), however, this is limited single instances.
We propose HyP-NeRF, a latent condition- ing method for learning improved quality gen- cralizable category-level NeRF priors using hypernetworks [15] (see Figure [). We take in-
backgraand
unposed
posed
Text: "A chair
spiration from methods that use learnnl to
learn generalizable representations [55] 48) while retaining the quality of instance-specific meth- ods B3 Our hypemneworr is trained to generate the partheters-both the multi-resolution hash en- codings (MRHE) and weights-oo of NeRF model of a given category conditioned on an instance code 2n For each instance code 2n in the learned codebook, HyP-NeRF estimates fi denoting the instance-specific MRHE along with ó., indicat- ing the weights of an MLP. Our key insight is that estimating both the MRHEs and the weights results in significant improvement in quality. To improve the quality even further, we denoise ren- dered views [42 from the estimated NeRF model, and finetune the NeRF with the denoised images to enforce multiview consistency. As shown in Figure 2 and the experiments section, this de- noising and finetuning step significantly improves quality and fine details while retaining the original shape and appearance properties.
Encoder
cocksion
teat-time optimization
HyP-NeRF
Denoiser
Denokes
Figure Once trained, HyP-NeRF acts as prior to support multiple downstream applications, including NeRF reconstruction from single or multi-view images and cluttered scene images, and text-to-NeRF. We fur- ther improve quality using our denoising network.
Once HyP-NeRP is trained, it can be used as NeRF prior in variety of different applications such as NeRF reconstruction from single view posed or unposed images, single pass text-to-NeRF
PProject page: pop-nerffgithub...

---

or even the ability to reconstruct real-world objects in cluttered scene images (see Figure 2). We show qualitative results on applications and quantitatively evaluate HyP-NeRF's performance and suitability as NeRF prior on the ABO dataset [1] across three tasks: generalization, compression, and retrieval. To sum up our contributions:
1. We introduce HyP-NeRF. a method for learning improved quality NeRF priors using a hypernetwork that estimates instance-specifee hash encodings and MLP weights of a NeRF.
2. We propose denoise and finetune strategy to further improve the quality while preserving the multiview consistency of the generated NeRF.
3. We demonstrate how our NeRF priors can be used in multiple downstream tasks including vingleview NeRF reconstruction, text-to-NeRF, and reconstruction from cluttered scenes.
2 Related Work
Neural Radiance Fields 32 (NeRFs) are neural networks that capture a specific 3D scene or object given sufficient views from known poses. Numerous follow-up work (see [2 70 for a more comprehensive review) has investigated improving quality and speed, relaxing assumptions, and building generalizable priors. Strategies for improving quality or speed include better sam- pling 22 supporting unbounded scenes (3), extensions to larger scenes [69 60), using hybrid 74), using learned initializations 66 S 451, or discarding neural networks Other work relaxes assumption of known poses [67 61 25 53], or reduce
representations 33
completely 73 59
[1 28 37 Specifically, PixeINeRF 75 uses convolution- based image features to learn priors enabling NeRF reconstruction from as few as a single image. VisionNeRF (28) extends PixeINeRF by augmenting the 2D priors with 3D representations learned using transformer. Unlike these methods, we depend purely on priors learned by meta-learning, specifically by hypernetworks [[5]. AutoRF [34] and LoINeRF [43 are related works that assume only single view for each instance at the training time. FWD (5) optimizes NeRFs from sparse views in real-time and SRT 45) aims to generate NeRFs in a single forward pass. These methods produce NeRFs of lower quality and are not designed to be used as priors for various downstream tasks. In contrast, our focus to generate high-quality multiview consistent NeRFs that capture fine shapes and textures details. HyP- NeR-N can be used as a category- prior for multiple downstream tasks including NeRF reconstruction from one or more posed or unposed images, text-to-NeRF (similar to 40 18) ), reconstruction from cluttered scene images. Additionally, HyP-NeRF can estimate the NeRFs in a single forward pass with only a few iterations needed to improve the quality. Concurrent to our work, NeriDiff [13] and SSDNeRF [8] achieve high quality novel view synthesis by using diffusion models.
the number of views
43 14
34 44
Learning 3D Priors. To learn category-level priors, methods like CodeNeRF 19 and LoINeRF 43
use a conditional NeRF on instance vectors given as fxx, 8, 2), where different 2s result in different NeRFs. PiXelNeRF (75) depends on 2D priors learned by 2D convolutional networks which could result in multi-view inconsistency. DiffRf 333 uses diffusion to learn a prior over voxelized radiance field. Like us, DiffRF can generate radiance fields from queries like text or images. However, it cannot be directly used for downstream tasks easily.
Our approach closely follows the line of work that aims to learn a prior over a 3D data distribution like signed distance fields (3), light field (55), and videos [48). We use meta-learning specifically hypernetworks 115). to learn a prior over the MRHEs and MLP weights of fixed NeRF architecture. LearnedInit [6., also employs standard meta-learning algorithms for getting a good initialization of the NeRF parameters. However, unlike us. they do not use hypemetwork, and the meta-learning algorithms only for initializing a NeRF, which is further finetumed on the multiview images. Methods like GRAF [47], T-G (6, CIPS-3D (79), EG3D [7]. and Pix2NeRF [4] use adversarial training setups with 2D discriminators resulting in 3D and multiview inconsistency. [40 64 [6] tightly couple text and NeRF priors to generate and edit NeRFs based text inputs. We, on the other hand, train
a 3D prior on NeRFs and separately train a mapping network that maps text to HyP-NeRF's prior, decoupling the two.
3 HyP-NeRF: Learning Improved NeRF prior using a Hypernetwork
Our goal learn a generalizable NeRF prior for a category of objects while maintaining visual and geometric quality, and multiview consistency. We also want to demonstrate how this prior can be
3

---

used to enable downstream applications in tigle/fee-image NeRF generation, text-to-NeRF, and reconstruction of weal-world objects in cluttered scenes.
Background. We first provide a brief summary of hypernetworks and multi-resolution hash encodings that form the basis of HyP-NeRF. Hypernetworks are neural networks that were introduced meta-network to predict the weights for second neural network. They have been widely used
for diverse tasks, starting from representation learning for continuous signals [55] [4 57 48).
compression [6 2], few-shot learning (50] 23 continual learning [63] Our key insight is to use
hypernetworks generate both the network weights and instance- instance-specific MRHEs.
Neural Radiance Fields (NeRF) [3 learn the parameters of a neural network fo 0) (o, c}. where x and 0 are the location and viewing direction of a 3D point, respectively. and and denote the density and color predicted by J& at that point. Once fully trained, So can be used to render novel views of the 3D scene. NeRF introduced positional encodings of the input 31D coordinates, x, to higher dimensional space to capture high-frequency variations in color and geometry. InstantNGP [35] further extended this idea to wistance-specific multi-resolution hash encodings (MRHE) encode dynamically based on scene properties. These MRHEs, h, are learned along with the MLP parameters, for given NeRF function, and show improved quality and reduced
/iriinn/iifeencee time.
Image Denoising is the process of reducing the noise and improving the perceptual quality of images while preserving important structural details. Recent advancements in deep learning- -aasnn image restoration and denoising techniques (26] 24] [0] have demonstrated remarkable success in removing noise and enhancing the perceptual quality of noisy images that may have suffered degradation. Such networks are trained on large datasets of paired noisy and clean images to learn a mapping between the degraded input and the corresponding high-quality output by minimizing the difference between the restored and the ground truth clean image. In our case, we use denoising to improve the quality of our NeRF renderings by reducing artifacts and improving the texture and structure at the image level.
3.1 Method
Given a set of NeRFs denoted by (f(o.....) I )
where N denotes the number of object instances in given object category, want learn prior t { &c), where @s and &e are the shape and color priors, respectively. Each NeRF, fo., is parameterized by the neural network weights, Pn.
and learnable MRHEs, h4 as proposed in [35 fe),. takes 3D position, x, and viewing direction,
(x) and color conditioned and 0 as input and predicts the density conditioned on denoted by of denoted by en {x,0} This is given as,
( 0) (x) (207,((x))) cxx.0).
(x),
Our proposed method for learning NeRF priors involves two steps. First, train a hypernetoork, M, to Iearn a prior over a set of multiview consistent NeRFs of high-quality shape and texture. Second, we employ image-based denoising network that takes input already multiview consistent set of images, rendered from the predicted NeRF, and improves the shape and texture of NeRF to higher quality by finetuning explain each step in detail below.
set of denoised images. Our architecture is outlined in Figure 3 and
Step 1: Hypernetwork for Learning NeRF Prior. We want to design our hypernetwork, M, with trainable parameters, Ω that can predict NeRF parameters {Φn, hn) given conditioning code en = (Sn, C. J. where S, and Cn are the shape and color codes, respectively, for an object instance n belonging to a specific category. Here, S, and C, belong to codebooks, S and C that are trained along with Ω in an auto-dccoding fashion.
As shown in Figure 3 (top), ideally we want M to learn a prior {Pc. $ over S and C' such that given a random set of codes, (D/s ~ &s ~ }c), M should be able to generate valid NeRF with consistent shape and texture for the given category of objects. To achieve this, we train M by assuming the same constraints as are needed to train a NeRF a set of multiview consistent images I = {{ ) L for a set of poses, O In each training step, we start with a random object instance, n, and use the corresponding codes S,, and C, from the codebooks as an input for M. Our key insight is that estimating both the MRHEs and MI.P weights results in a higher quality than other alternatives. M then predicts the NeRF parameters {Φn, h which is then used to minimize the

---

Hypernelwork
S1
FC
FC
Weight Initalization
4n
(Op. Gp)
6 Ground Truth
Roceration Hoth Eiceding
FC
FC
Volumetric Rendering
Prodiction
Projection
Objeckn
Neural Radiance Field
Multi-view Projections
Neural Radiance Field
0 0m
Denoising
Finetune
Figure 3: Architecture Diagram: HyP-NeRF is trained and inferred in steps. In the first step (top). our hypernetwork, M, is trained to predict the parameters of a NeRF model, fn. corresponding to object instance At this stage, the NeRF model acts the volumetric rendering loss, using which is trained on a set of N objects, thereby learning a prior = ss, Φc) over the shape and color codes given by S and C, respectively. In the second step (bottom), the quality of the predicted multiview consistent NeRF, fa, is improved using denoising network trained directly in the image space. To do this, Sn rendered from multiple known poses to set of images that are improved to photorcalistic quality. fu. is then finetuned on these improved images. Importantly, since fu is only finetuned and not optimized from scratch, and thus fn retains the multiview consistency whilst improving in terms of texture and shape quality. following objective:
set of differentiable layers to compute
L(S, Sn, C,)
Σ (IV (on rER
(o Gn (af) (500))1) (r(r))l
(uf) (of for
(1] (1.....)) ( and [on, Mn Cn)
Cn
where V denotes the volumetric rendering function as given in [32 eqn. and 5, r is ray projected along the camera pose 0, xF E x and I. denote the number of points sampled along and V, denote the ground truth value for projection of the nhh object along
Note that, in this step, the only trainable parameters the meta-network weights, Ω, and the codebooks and C. In this setting, the NeRF functions f(), only act as differentiable layers that allow backpropogation through M enabling it to train with multiview consistency loss attained by the volumetric rendering loss as described in (32]. We use an instantiation of InstantNGP 35 as our function f(a,, consisting of MRHE and small ML.P
general limitation of hypernetworks arises from the fact that the intended output space (i.c. the space of valid MLP weight matrices) is subset of the actual output space, which unristricted and can be any 2D matrix. Thus, hypernetwork trained on loss functions in the weight space can result in unstable training, and might require lot of training examples to converge. To overcome this issue, we train our hypernetwork end-to-end to-end directly on images, so that learns the implicit NeRF space along with the category specific prior on it, which simplifies the setting for the hypernetwork and allows for more stable training. As a causal effect of this, HeP-NeRF, when trained on less number of examples, essentially acts compressing model.
Step 2: Denoise and Finetune. In the first step, is trained to produce consistent NeRF with high-fidelity texture and shape. However, we observed that there is room to improve the generated
5

---

Chairs
Sofa
PSNR1
SSIM
LPIPS.
FID..
PSNR
SSIM
LPIPS.
FIDJ
PixeINeRF 75
18.30
0.83
0.31
292.32
17.51
0.84
0.28
323.89
ABO-512
CodeNeRF 19
19.86
0.87
0,298
19.56
0.87
0.290
HyP-NeRF (Ours)
24.23
0.91
0.16
68.11
23.96
0.90
0.18
120,80
w/o Denoise
23.05
0.90
0.16
102.45
23.54
0.90
0.174
121.69
Table 1: Generalization. Comparison of single-posed-view NeRF generation. Metries are computed on renderings of resolution 512 x 512. HyP-NeRF significantly outperforms PixeINeRF and CodeN- cRF on all the metrics in both the datasets.
NeRFs to better capture fine details like uneven textures and edge definition. To tackle this challenge, we augment M using denoising process that takes f(.), and further finetunes it to achieve (c).
As shown in Figure 3 (bottom), we render novel views from the multiview consistent NeRF into m different predefined poses given by (f..0mm. to produce set of multiview consistent images {4) We then use a pre-trained image-level denoising autoencoder that takes {L. { as input and produces images of improved quality given as {1 ) These improved images are then used to finetune f(.), to achieve f().. Note that, we do not train the NeRFs from scratch on {f and only finetune the NeRF',, which ensures fast optimization and simplifies the task of the denoising module that only needs to improve the quality and does not necessarily need to maintain the multiview consistency. While our denoising is image-level, still obtain multiview consistent NeRFs since finetune on the NeRF itself (as we also demonstrate through experiments in the supplementary).
For our denoising autoencoder, we use VQVAE2 42 as the backbone. To train this network, we simply use images projected from the NeRF, predicted by the hypernetwork (lower quality relative to the ground truth) as the input to the VQVAE2 model. We then train VQVAE2 to decode the ground truth by minimizing the L2 loss objective between VQVAE2's output and the ground truth.
3.2 HyP-NeRF Inference and Applications
Training over many NeRF instances, M learns a prior & that can be used to generate novel consistent NeRFs. However, & is not a known distribution like Gaussian distributions that be naively queried by sampling a random point from the underlying distribution. We tackle this in two ways:
Test Time Optimization. In this method, given single-view multi-view posed image(s), we aim to estimate shape and color codes (S.. C. of the NeRF that renders the view(s). To achieve this, we freeze M's parameters and optimize the (S., C.} using the objective given in Equation (2).
Query Network. We create query network, A, that maps a point from known distribution to O. As CLIP's [41] pretrained semantic space, say C, is both text and image aware, we chose C, as our known distribution and learn a mapping function A(z ~ C) + 4. Here, A is an MLP that takes input and produces V- & as output. To train A, we randomly sample one pose from the ground truth multiview images 15 (Loee)n and compute the semantic embedding z = CLIP(IE) and map it to {S,, C..) E & given as C.) = A())). We then train our query network by minimizing the following objective:
Ls Σ IIMS., (I),,,Cn), (S.,, Cl
At the time of inference, given text or image modality such as text prompt, single-view unposed (in-the-wild) image, or segmented image, we compute the semantic embedding using CLIP encoder and map it to & using A, from which we obtain the shape and color codes input for the HyP-NeRF.
Note that for N query points in a scene, the forward pass through the hypernetwork (computa- tionally expensive) happens only once per scene. Only the NeRF predicted by the hypernetwork (computationally less expensive) run for each query point.
4 Experiments
We provide evaluations of the prior learned by HyP-NeRF specifically focusing on the quality of the generated NeRFs. We consider three dimensions: (1) Generalization (Section[4.]) we validate whether HyP-NeRF can generate novel NeRFs not seen during training by conditioning on only

---

Input
PixeINeRF
HyP-NeRF
w/o Denoising
Figure 4: Qualitative Comparison of Generalization on ABO. The NeRFs rendered
resolution of 512 x 512. HyP-NeRF is able to preserve fine details such as the legs, creases, and texture even for novel instances. PixeINeRF fails preserve details and to model the structure.
ABO Chairs
ABO Table
AofaS SSIMI
PSNR1
SSIM
LPIPS
CD
PSNR
SSIMf
LPIPS
CD
PSNR1
LPIPS
CD
35
35.43
0.96
0.07
34.07
0.95
0.07
33.87
0.95
0.08
Ours
31.37
0.94
0.1
0.0082
29.52
0.93
0.11
0.0033
30.32
0.94
0.11
0.0118
Table 2: Compression. We randomly sample 250 datapoints from our training dataset and compare the NeRFs learned using InstantNGP (35) on the individual instances against HyP-NeRF that learns the entire dataset. Note, we do not employ the denoising module (see Section 3.1 for this evaluation.
single-posed-view of novel NeRF instances. (2) Compression (Section 4.2 since HyP-NeRF is
trained in an auto-decoding fashion on specific NeRF instances (see Equation we can evaluate the quality of the NeRFs compressed in this process. (3) Retrieval (Section 4.3 shown in Figure 2 HyP-NeRF's prior enables various downstream applications. We show how to combine our prior with to retrieve novel NeRFs.
CLIP 41
Datasets and Comparisons. We primarily compare against two baselines, PixelNeRF 75 and InstantNGP 35 the Amazon-Berkeley Objects (ABO) (II) dataset. ABO contains diverse and detailed objects rendered at a resolution of 512 x 512 which is perfect to showcase the quality of the NeRF generated by HyP-NeRF. Rather than use computationally expensive model like VisionNeRF (on the SRN [56] dataset) on a resolution of 128 x 128, we show our results on 512 x 512 and compare with PixeINeRF. Additionally, resolution qualitatively in the main paper (Figure 5) and quantitatively in the supplementary. For compression, directly compare with InstantNGP [35 that proposed MRHE, trained to fit individual objects sestance-by-instanc
compare with the other baselines on SRN at 128 x 128
Architectural Details. We use InstantNGP as S().: with 16 levels, hashtable size of 211, feature dimension 2, and linear interpolation for computing the MRHE; the MI.P has a total of 5, 64- dimensional, layers. We observed that a hashtable size 2 produces NeRF of high-quality at par with

---

Input
PixelNoRF
VisionNeRF
Ours
Input
PixelNoRF
VisionNeRF
Ours
Figure 5: Qualitative Comparison of Generalization on SRN on the task of single-view inversion (posed in our case) and compare the quality of the views rendered at 128 x 128. HyP-NeRF renders NeRFs of similar quality to the PixeINeRF and VisionNeRF baselines.
the a size of 2 Hence, we use 2 to speed up our training. Our hypernetwork, M, consists of 6 MLPs, for predicting the MRHE, and the rest predicts the parameters for each of the MLP layers of f. Each of the MLPs are made of 3, 51 512-dimensional, layers. We perform all of our experiments on NVIDIA RTX 2080Tís.
Metries. To evaluate NeRF quality, we render them at 91 distinct views and compute metrics on the rendered im- ages. Following PixeINeRF, we use PSNR(T), SSIM(1), and LPIPS(() 78 Additionally, we compute Fréchet Inception Distance (FID)(J) [17 to further test the visual quality. AI- though these metries measure the quality of novel-view syn- thesis, they do not necessarily evaluate the geometry captured by the NeRFs. Therefore, we compute Chamfer's Distance (CD) whenever necessary by extracting mesh from NeRF densities 330. Please see the supplementary for additional details.
ABO Chairs
ABO Sofa
Top Top 3
Top 1
Top 3
98.72%
99.81%
91.6%
95.27%
Table 3: Retrieval. We design a sim- plc query network (see Section 3.2) to retrieve NeRF instances from HyP- NeRF's prior seen at the time of training and achieve almost 100% accuracy.
4.1 Generalization
One way to evaluate if HyP-NeRF can render novel NeRF instances of high quality is through unconditional sampling. However, our learned prior & is a non-standard prior (like a Gaussian distribution) and thus random sampling needs carefully designed mapping between such a known prior and Q. Therefore, we instead rely on a conditional task of single-view novel NeRF generation: given single arbitrarily-chosen view of novel object, we generate the corresponding NeRF, f(.), through test-time optimization (see Section 3.2). We compare quantitatively with PixelNeRF on ABO high resolution of 512 512 and qualitatively with the rest of the baselines SRN at 128 128.
As shown in Table we significantly outperform PixeINeRF on all of the metrics. Further, the qualitative results in Figure shows the difference between the rendering quality of HyP-NeRF against Pi Specifically, PixelNeRF fails to learn details, especially for the Sofa category. On the other hand, HyP-NeRF preserves intricate details like the texture, legs, and folds in the objects even at a high resolution. Further, we show our results on the widely used SRN dataset at the resolution of 128 x 128 in Figure 5 Here, our quality is comparable with the baselines.
4.2 Compression
Unlike InstantNGP, which trained on a single 3D instance, HyP-NeRF is trained on many NeRF instances which effectively results in the compression of these NeRFs into the latent space (or the codebook). We evaluate this compression capability by computing NeRF quality degradation compared to single-instance-only method, InstantNGP.
We randomly sample 250 instances from the training set and train InstantNGP separately on each of them. These samples are a subset of the training data used in HyP-NeRFF codebook. We show degradation metrics Table 2 Note that do not perform denoising on the generated NeRFs as we want to only evaluate the compression component of HyP-NeRF in this section. As can be seen Table 2 there is significant degradation terms PSNR (an average of 11%) 1), but overall geometry is preserved almost as well as InstantNGP. However, InstantNGP is trained on single instance, whereas we train on 1000s of NeRF instances (1038, 783, and 517 instances for ABO Chairs, Sofa, and Tables, respectively). This results in 60x compression gain: for ABO Chairs,

---

with 1038 training instances, HyP-NeRR needs 163MB to store the model, whereas a single instance of InstantNGP needs on average 8.9MB. Note that we use the same network architecture [I] for HyP-NeRF and InstantNGP making this a fair comparison. Moreover, the storage complexity for InstantNGP-baxs NeRFs linear with respect to the number of instances, whereas our degradation in visual quality is sublinear.
4.3 Retrieval
Input
Predicted NeRF (Single Forward Pass)
A generalizable learned prior has the ability to generate NeRFs based on different input modalities like text, images, segmented and occluded images, random noise, and multi- view images. We now demonstrate addi- tional querying and retrieval capabilities as described in Section 3.2
inchewis
Cluttered
This experiment's goal is to retrieve specific NeRF instances that HyP-NeRF has encoun- tered during training from single-view un- posed image of that instance. Section 4.3 presents the number of times we could cor- rectly retrieve from arbitrary view of seen NeRF instances. We achieve almost 100% accuracy for Chair and Sofa datasets. How- ever, we take this step further and try to retrieve the closest training instance code corresponding to unseen views of seen in- stances taken from in-the-wild internet im- ages. Figure
umposed
Text: A Green Cushion Chair
Figure 6: Qualitative Comparison of Querying (Sec- tion 3.2) on HyP-NeRF's prior. In the top, we use in-the-wild single- view unposed image retrieve the closest NeRF HyP-NeRE has seen during training. In middle, take cluttered scene, and mask out the object of interest using Segment Anything [22] and in the bottom to our query network, A. We then obtain the latent codes (S, C) from A, which are used as input for HyP-NeRF.
text prompt as input
6 (top) shows examples from this experiment in which we are able to re- trieve a NeRF closely matching the input query. This demonstrates the ease of design- ing a simple mapping network that can effectively interact with HyP-NeRF's prior.
Along with retrieving seen instance, we use the query network generate novel NeRFs of unseen instances as shown in Figure 66 (middle and bottom). In the middle row, we take an image of cluttered scene, segment with SAM (2),, and pass this input the query network, from which we obtain set of latent codes given as input to HyP-NeRF (see Figure 2). Finally, in the bottom row, we show text-to-NeRF capabilities enabled by HyP-NeRF.
4.4 Ablation
Two key designs of HyP-NeRF include incorporating the MRHE and the denoising network. We present the affect of removing these two components in Table and Figure and Figure 4
Chairs
PSNR1
SSIM
LPIPS
CD_
HyP-NeRF w/O MRHE
29.23
Figure for MRHE and Table [ for denoising. In the first ablation, we change the de- sign of our neural network by using a hypernetwork to predict the parameters of a standard nerf with positional encodings (32). Since we remove the MRHE, we also increase the number layers in the MLP match the layers mentioned in 32 Since there is a significant increase in the view rendering time, we randomly sam- ple 70 training examples for evaluation. As seen in Table the quality of the rendered views lags significantly all the metries including the CD (measured against NeRFs rendered on InstantNGP individually). This is showcased visually in Figure | and the supplementary. Similarly, we find significant differences between the quality of the NeRFs before and after denoising (Table Figure 2 and Figure particularly in the Chair category with more diverse shapes.
26.42
0.92
0.16
0.0100
Table 4: Ablation of removing MRHE on ABO dataset. Due to the significant rendering time of HyP-NeRF wo MRHE, we sample 70 object instances from the training dataset com- pute the metrics at 512 x 512 resolution.

---

Choini (from frain set)
Char-l-Geomety Chair-2-Coir
Choir-3 (from train set)
Chai--S Geommetyy Choin-d- Color
Chair-2 (from hrann set)
Char-2-Geomehy- Chai--l-olor
Chain-d (from troin set)
Chan-4-Geometry Chaln-S-oolrr
Figure 7: Qualitative results on color and geometry disentanglement: We take two instances from the training set and switch the geometry and color codes to generate novel instances. As can be seen, the geometry and the color are transferred while preserving fine shape details. Even the fine details, like stripes and color-contrast between the chair seats and edges from Chair-2, are accurately transferred Chair-1 (Chair-2- --oomeeryy + Chair-1-Color) I-Color). Zoom in for improved experience.
Chaler1
Color Clustoing
Shape Clustering
Figure 8: Latent visualization through TSNE plots on shape and color codes (from the codebooks and C). As can be seen, the underlying space forms meaningful clusters shown through examples randomly sampled from two different clusters.
4.5 Color and Shape Disentanglement
We start with two object instances from the train set and B and denote their corresponding shape and geometry codes as A,, B, and Ag. Bg. We switch the geometry and shape code and generate two novel NeRFs given by A,. Bg and B., Ag. Figure we can clearly see the disentanglement: geometry is perfectly preserved, and the color is transferred faithfully across the NeRFs. In Figure X we show clusters of color and shape codes using TSNE plots and visualize instances from the clusters.
5 Conclusion, Limitation, and Future Work
We propose HyP-NeRF, learned prior for Neural Radiance Fields (NeRFs). HyP-NeRF uses hypernetwork to predict instance-specific multi-resolution hash encodings (MRHEs) that significantly improve the visual quality of the predicted NeRFs. To further improve the visual quality, we propose a denoising and finetuning technique that result in an improved NeRF that preserves its original multiview and geometric consistency. Experimental results demonstrate HyP-NeRF's capability to generalize to unseen samples and its effectiveness in compression. With its ability to overcome limitations of existing approaches, such as rendering at high resolution and multiview consistency, HyP-NeRF holds promise for various applications we demonstrate for single- and multi-view NeRF reconstruction and text-to-NeRF.
Limitation and Future Work. One limitation of our work is the need for the pose to be known during test-time optimization (Section 3.2). Although we propose the query network to predict novel NeRFs conditioned on an unposed single view, the result may not exactly match the given view because of the loss of detail in the CLIP embedding, Future work should design mapping network that can preserve fine details. An iterative pose refinement approach that predicts the pose along with the shape and color codes could also be adopted. second limitation of our work is the non-standard prior O that was learned by HyP-NeRF which makes unconditional generation challenging. GAN-based generative approaches solve this problem by randomly sampling from a standard distribution (like Gaussian distribution) and adversarially training the network. However, those methods often focus more on image quality than 3D structure. Future work could address this by incorporating latent diffusion models that can map standard prior HyP-NeRF's prior.
10

---

References
[1] GitHub ashawkeytorcch-ng: A pytorch CUDA extension implementation of instant-ngp (sdf and nerf). with GUT. github.com. htttps://githbb.comm wtt..///aaaaaakke/rrrrraagg..... [Accessed 17-May-2023,.
[2] Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brually and Pratul P. Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. ICCV, 2021.
[3] Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. CVPR, 2022.
[4] Shengqu Cai, Anton Obukhov, Dengxin Dai, and Luc Van Gool. Pix2nerf: Unsupervised conditional p-gan for single image to neural radiance fields translation. In Proceedings the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3981-3990, June 2022.
[5] Ang Cao, Chris Rockwell, and Justin Johnson. Fwd: Real-time novel view synthesis with forward warping and depth. CVPR, 2022.
[6] Eric Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein. pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In Proc. CVPR, 2021.
[7] Eric R. Chan, Connor Z. Lin, Matthew A. Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, Tero Karras, and Gordon Wetzstein. Efficient geometry-aware 3D generative adversarial networks. In arXiv, 2021.
[8] Hansheng Chen, Jiatao Gu, Anpei Chen, Wei Tian, Zhuowen Tu, Lingjie Liu, and Hao Su. Single-stage diffusion nerf: A unified approach 3d generation and reconstruction. In ICCV, 2023.
[9] Shin-Fang Chng, Sameera Ramasinghe, Jamie Sherrah, and Simon Lucey. Garf: gaussian activated radiance fields for high fidelity reconstruction and pose estimation. arXiv e-prints, pages arXiv-2204, 2022.
[10] Jian Zhang Chong Mou, Qian Wang. Deep generalized unfolding networks for image restoration. In IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2022.
[11] Jasmine Collins, Shubham Goel, Kenan Deng, Achleshwar Luthra, Leon Xu, Erhan Gundogdu, Xi Zhang, Tomas F Yago Vicente, Thomas Dideriksen, Himanshu Arora, Matthieu Guillaumin, and Jitendra Malik. Abo: Dataset and benchmarks for real-world 3d object understanding. CVPR, 2022.
[12] Shangqian Gao, Feihu Huang, and Heng Huang. Model compression via hyper-structure network, 2021.
[13] Jiatao Gu, Alex Trevithick, Kai-En Lin, Josh Susskind, Christian Theobalt, Lingjie Liu, and Ravi Ramamoorthi. Nerfdiff: Single-image view synthesis with nerf-guided distillation from 3d-aware diffusion. In International Conference on Machine Learning. 2023.
[14] Pengsheng Guo, Miguel Angel Bautista, Alex Colburn, Liang Yang. Daniel Ulbricht, Joshua M Susskind, and Qi Shan. Fast and explicit neural view synthesis. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3791-3800 2022.
[15] David Ha, Andrew Dai, and Quoe V Le. Hypernetorkss. arXiv preprint arXiv: 1609.0910,, 2016.
[16] Ayaan Haque, Matthew Tancik, Alexei Efros, Aleksander Holynski, and Angjoo Kanazawa. Instruct-nerfZnefff: Editing 3d scenes with instructions. 2023.
[17] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.

---

[18] Ajay Jain, Ben Mildenhall, Jonathan T Barron, Pieter Abbeel, and Ben Poole. Zero-shot text-guided object generation with dream fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 867-876, 2022.
[19] Wonbong Jang and Lourdes Agapito. Codenerf: Disentangled neural radiance fields for object categories. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages -2958, 2021.
[20] Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. ArXiv, abs/2006.06676, 2020.
[21] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings the IEEE/CVF conference computer vision and pattern recognition, pages 4401-441, 2019.
[22] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan- Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. 2XXiv:2304.02443, 2023.
[23] A. Lamb, Evgeny S. Saveliev, Yingzhen Li, Sebastian Tschiatschek, Camilla Longden, Simon Woodhead, José Miguel Hernández-Lobato, Richard E. Turner, Pashmina Cameron, and Cheng Zhang. Contextual hy hypernetworks for novel feature adaptation. ArXiv, abs/2104.0558 2021.
[24] Fei Li, Lingfeng Shen, Yang Mi, and Zhenbo Li. Drenet: Dynamic image restoration contrastive network. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, Computer Vision - ECCV 2022, pages 514-532, Cham, 2022. Springer Nature Switzerland.
[25] Fu Li, Hao Yu, Ivan Shugurov, Benjamin Busam, Shaowu Yang, and Slobodan Ilie. Nerf-pose: approach for weakly- suukly--upervised 6d object pose estimation. arXiv preprint arXiv:2203.04800 2022.
first-reconstruct-hhrr
[26] Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. arXiv preprint arXiv:2108. 10257, 2021.
[27] Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, and Simon Lucey. Barf: Bundle-adjusting neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 5741-5751, October 2021.
[28] Kai-En Lin, Lin Yen-Chen, Wei-Sheng Lai, Tsung- Yi Lin, Yi-Chang Shih, and Ravi Ramamoor- thi. Vision transformer for nerf-based view synthesis from a single input image. In WACV, 2023.
[29] Xinyue Liu, Xiangnan Kong, Lei Liu, and Kuorong Chiang. Treegan: syntax-awar sequence generation with generative adversarial networks. In 2018 IEEE International Conference Data Mining (ICDM), pages 1140-1145 IEEE, 2018.
[30] William E Lorensen and Harvey E Cline. Marching cubes: A high resolution 3d surface construction algorithm. ACM siggraph computer graphics, 21(4):163-169, 1987.
[31] Quan Meng, Anpei Chen, Haimin Luo, Minye Wu, Hao Su, Lan Xu, Xuming He, and Jingyi Yu. Gnerf: Gan-based neural radiance field without posed camera. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 6351 6351-6330 1-361, October 2021.
[32] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoor- thi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
[33] Norman Müller, Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Buld, Peter Kontschieder, and Matthias Nießner. Diffrf: sendering-guider 3d radiance field diffusion. arXiv preprint arXiv: 2022.
12

---

[34] Norman Müller, Andrea Simonelli, Lorenzo Porzi. Samuel Rota Bulò, Matthias Nießner, and Peter Kontschieder. Autorf: Learning 3d object radiance fields from single view observations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2022.
[35] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans. Graph., 41(4):102:1-102:155, July 2022.
[36] Phuoc Nguyen, T. Tran, Ky Le, Sunil Gupta, Santu Rana, Dang Nguyen, Trong Nguyen, Shannon Ryan, and Svetha Venkatesh. Fast conditional network compression using bayesian hypernetworks. In ECML/PKDD, 2022.
[37] Michael Niemeyer, Jonathan T Barron, Ben Mildenhall, Mehdi SM Sajjad,, Andreas Geiger, and Noha Radwan. Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages $480-5490, 2022.
[38] Michael Oechsle, Songyou Peng, and Andreas Geiger. Unisurf: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction. In International Conference on Computer Vision (ICCV), 2021.
[39] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceed- ings of the IEEE/CVF conference computer vision and pattern recognition, pages 165-174, 2019.
[40] Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv, 2022.
[41] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 3768-8763. PMLR, 2021.
[42] Ali Razavi, Aäron den Oord, and Oriol Vinyals. Generating diverse high-fidelity images 2019.
with vq-vae-2. ArXiv,
[43] Daniel Rebain, Mark Matthews, Kwang Moo Yi, Dmitry Lagun, and Andrea Tagliasacchi. Lolnerf: Learn from one look. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1558-15671 2022.
[44] Barbara Roessle, Jonathan T. Barron, Ben Mildenhall, Pratul P. Srinivasan, and Matthias Nießner. Dense depth priors for neural radiance fields from sparse input views. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2022.
[45] Mehdi SM Sajjadi, Henning Meyer, Etienne Pot, Urs Bergmann, Klaus Greff, Noha Radwan, Suhani Vora, Mario Lučié, Daniel Duckworth, Alexey Dosovitskiy, et al. Scene representation transformer: Geometry- novel view synthesis through set-latent scene representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6229 6238, 2022.
[46] Rahul Sajnani, Adrien Poulenard, Jivitesh Jain, Radhika Dua, Leonidas J. Guibas, and Srinath Sridhar. Condor: Self-supervised canonicalization of 3d pose for partial shapes. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2022.
[47] Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. Graf: Generative radiance fields for 3d-aware image synthesis. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[48] Bipasha Sen, Aditya Agarwal, Vinay P Namboodiri, and C.V. Jawahar. INR-v: A continuous representation space for video-based generative tasks. Tronsactions on Machine Learning Research, 2022.
13

---

[49] Bipasha Sen, Aditya Agarwal, Gaurav Singh, Brojeshwar B., Srinath Sridhar. and Madhava Krishna. Scarp: 3d shape completion in arbitrary poses for improved grasping. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages
2023.
[50] Marcin Sendera, Marcin Przewiezlikowski, Konrad Karanowski, Macicj Zieba, Jacek Tabor, and Przemyslaw Spurek. Hypershot: Few-shot learning by kernel hypernetworks. 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 1468-2477) 2022.
[51] Anthony Simeonov, Yilun Du, Yen-Chen Lin, Alberto Rodriguez Garcia, Leslie Pack Kaelbling, Tomás Lozano-Pérez, and Pulkit Agrawal, SE(()-eqiivarian relational rearrangement with neural descriptor fields. In 6th Annual Conference on Robot Learning. 2022.
[52] Anthony Simeonov, Yilun Du, Andrea Tagliasacchi, Joshua B. Tenenbaum, Alberto Rodriguez, Pulkit Agrawal, and Vincent Sitzmann. Neural descriptor fields: Se(3)-cquivariant object representations for manipulation. 2022.
[53] Samarth Sinha, Jason Y Zhang, Andrea Tagliasacchi, Igor Gilitschenski, and David B Lin- dell. Sparsepose: Sparse-view camera pose regression and refinement. arXiv preprint arXiv:2211.16991 2022.
[54] Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein, Im- plicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 33:7462-7473, 2020.
[55] Vincent Sitzmann, Semon Rezchikov, Bill Freeman, Josh Tenenbaum, and Fredo Durand. Light field networks: Neural scene representations with single-evaluation rendering. Advances in Neural Information Processing Systems, 14:19313-19322 2021.
[56) Vincent Sitzmann, Michael Zollhoefer, and Gordon Wetzstein. Scene representation networks: Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché-BBBr Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Continuous 3d-structure aware neural scene representations.
[57] Ivan Skorokhodov, Savva Ignatyev, and Mohamed Elhoseiny. Adversarial generation of contin- uous images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10753-10764, 2021.
[58] Ivan Skorokhodov, Sergey Tulyakov, and Mohamed Elhoseiny. Stylegan-v: A continuous video generator with the price, image quality and perks of stylegan2. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 3626-3636, 2022.
[59] Cheng Sun, Min Sun, and Hwann- Jiwann-Tzong Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5459-5469, 2022.
[60] Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul Srinivasan, Jonathan T Barron, and Henrik Kretzschmar. Block-nerf: Scalable large scene neural view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8248-8258, 2022.
[61] Matthew Tancik, Ben Mildenhall, Terrance Wang, Divi Schmidt, Pratul P. Srinivasan, Jonathan Barron, and Ren Ng. Learned initializations for optimizing coordinate-based neural representa- tions. In CVPR, 2021.
[62] Ayush Tewari, Justus Thies, Ben Mildenhall, Pratul Srinivasan, Edgar Tretschk, Wang Yifan, Christoph Lassner, Vincent Sitzmann, Ricardo Martin-Brualla, Stephen Lombardi, et al. Ad- vances in neural rendering. In Computer Graphics Forum, volume 41, pages 703-735. Wiley Online Library, 2022.
[63] Johannes Von Oswald, Christian Henning, João Sacramento, and Benjamin F Grewe, Continual learning with hypernetworks. arXiv preprint
2019.
14

---

[64] Can Wang, Menglei Chai, Mingming He, Dongdong Chen, and Jing Liao. Clip-nerf: Text- and-image driven manipulation of neural radiance fields. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 3835-3844 2022.
[65] He Wang. Srinath Sridhar, Jingwei Huang, Julien Valentin, Shuran Song, and Leonidas J. Guibas. Normalized object coordinate space for category-level 6d object pose and size estimation. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2019.
[66] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. NeurIPS, 2021.
[67] Zirui Wang, Shangzhe Wu, Weidi Xie, Min Chen, and Victor Adrian Prisacariu. NeRF Neural radiance fields without known camera parameters. arXiv preprint 100::2202..0064, 2021.
[68] Francis Williams, Zan Gojeie, Sameh Khamis, Denis Zorin, Joan Bruna, Sanja Fidler, and Or Litany. Neural fields as learnable kernels for 3d reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18500-18510, 2022.
[69] Yuanbo Xiangli, Linning Xu, Xingang Pan, Nanxuan Zhao, Anyi Rao, Christian Theobalt, Bo Dai, and Dahua Lin. Citynerf: Building nerf at city scale. arXiv preprint arXiv:21/2.05500 2021.
[70] Yiheng Xie. Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan, Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, and Srinath Sridhar. Neural fields in visual computing and beyond. Computer Graphics Forum, 2022.
[71] Dejia Xu, Yifan Jiang, Peihao Wang, Zhiwen Fan, Humphrey Shi, and Zhangyang Wang. Sinnerf: Training neural radiance fields on complex scenes from single image. In Computer Vision- ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings. Part XXII, pages 736-753. Springer, 2022.
[72] Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. In Hiitt-Fiifth Conference on Neural Information Processing Systems, 2021.
[73] Alex Yu, Sara Fridovich-Keil, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. arXiv preprint arXiv:2112.05131, 2021.
[74] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. Plenoctrees for real-time rendering of neural radiance fields. Proceedings of the IEEE/CVF International Conference Computer Vision, pages 5752-5761, 2021.
[75] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixeINeRF: Neural radiance fields from one or few images. In CVPR, 2021.
[76] Sihyun Yu, Jihoon Tack, Sangwoo Mo, Hyunsu Kim, Junho Kim, Jung- Ha, and Jinwoo Shin. Generating videos with dynamics-aware implicit generative adversarial networks. ArXiv, abs/2202. 10571, 2022.
[77] Biao Zhang, Jiapeng Tang, Matthias Niessner, and Peter Wonka. 3dshape2vecset: A 3d shape representation for neural fields and generative diffusion models. arXiv preprint arXiv:2301.11445 2023.
[78] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreason- able effectiveness of deep features as perceptual metric. In CVPR, 2018.
[79] Peng Zhou, Lingxi Xie, Bingbing Ni, and Qi Tian. CIPS-3D: A 3D-Aware Generator of GANs Based on Conditionally- ConditionlllyInddeendeen Pixel Synthesis. 2021.
15


Reviews:
[{"review_id": "yNyRRzMwfu", "reviewer": "Reviewer_Muw9", "summary": "This paper proposes a way to encode a categorical prior for NeRFs. The method train an auto-decoder, which optimize together both the latent codes (one for every instance in the dataset) and the decoder parameters. The decoder will take the latent code and predict parameters for an instance-NGP backbone supported NeRF model. One key technical contribution is that the hyper-network predicts not only the parameters for the instant-NGP's MLP, but also the Multi-resolution hash encoding. This allows the predicted NeRF to get higher quality. The paper demonstrates this prior can be used for a variety of tasks.", "questions": "See weakness section.", "limitations": "The paper has a good discussion of potential limitation. Additional limitation I forsee is already listed in the weakness section.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This method is able to generate relatively high-quality NeRF thanks to the design of hyper-network that predict parameters for both the MRHE and the MLP. This design did circumvent the limitation of storing NeRF at a voxel. \n- The paper demonstrated that the learned prior is capable of various tasks. This demonstrate the versatility of this learned prior.\n- Novelty. To the best of my knowledge, the solution to build an auto-decoder solution for hyper-network that predicts both the MLP parameters and the MRHE is novel.", "weaknesses": "- Require separate pipeline for generation and desnoising. Since the denoising and fine-tuning pipeline is an optimization procedure, it's not clear to me whether the pipeline is able to enforce satisfication of the conditioning input. For example, if we use single image as an input, and would like to obtain a NeRF where certain pose is that image. After test-time optimization using the NeRF loss of the single image, this NeRF will be pipe into the denoise pipeline, which is another optimization. How does this second stage of the pipeline will predict something that's aware of the conditional input signal?\n- Hyper-network number of parameters. It occurs to me that the hyper-network can take a lot of parameters in order to predict the parameters for both the MRHE and the MLP. There are also other issue of the hyper-network, such as the output does not necessarily respect the permutation invariance of the MLP parameters space, which is especially the case when the hyper-network structure is only an MLP. Training such a large amount of parameters can require a lot of computing or data.\n- It's not clear how to achieve unconditional generation using this pipeline. Since the pipeline, after training, will only provide a codebook and a hyper-network, without a way to sample the distribution of the codebook. This suggests that in order to make this pipeline useful for generating a NeRF that satisfies the data distribution, it requires additional handling of the pipeline such as learning variational auto-decoder.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are delighted to receive such a positive assessment of our work and are grateful for it. Below, we provide the answers to the questions that were raised. \n\n1. **Require separate pipeline for generation ... this second stage of the pipeline will predict something that's aware of the conditional input signal?**\n\n    This is addressed in the global rebuttal.\n\n2. **There are also other issue of the hyper-network, such as the output does not necessarily … structure is only an MLP.**\n\t\n    Unfortunately, we do not understand this part clearly, but would to love to discuss this during the discussion phase. \n\n3. **Hyper-network number of parameters. It occurs to me that the hyper-network can take a lot of parameters in order to predict the parameters for both the MRHE and the MLP….Training such a large amount of parameters can require a lot of computing or data.** \n    \n    This is addressed in the global rebuttal\n\n4. **It's not clear how to achieve unconditional generation using this pipeline ... pipeline such as learning variational auto-decoder.**\n\n    - As mentioned on page 6, line 219, the hypernetwork is trained in an auto-decoding fashion and consequently learns a non-standard prior. This training strategy of auto-decoding has been well-adopted across an array of influential works that aim to predict an implicit function, such as DeepSDF [37], Scene Representation Networks (SRN) [54], Light Field Networks [53], INR-V [46], and many more. Therefore, as you rightly mentioned, it is not trivial to perform unconditional sampling through our learned prior, which is also listed in the limitation section of our work (page 9 line 346), and we would encourage future works to pursue this direction. \n\n    - One major aspect of unconditional sampling is to showcase generalization. In our case, we test HyP-NeRF's ability to generalize based on a well-adopted conditional task of NeRF generation from a single-view image. Further, we showcase many diverse downstream applications that can be enabled through HyP-NeRF through many examples in the paper and video, including compression, text-to-NeRF generation, generating NeRF from occluded and cluttered images **scraped directly from the internet without any preprocessing** (except the segmentation masks obtained from SAM [20]), and so on. "}, {"review_id": "drG5xk6NKa", "reviewer": "Reviewer_bkzC", "summary": "The paper proposes HyP-NeRF, a method using hypernetworks to learn generalizable category-level priors, addressing the limitations of existing work on generalization.  Specifically, the proposed hypernetwork-based method predicts the parameters of NeRF and multi-resolution hash encodings, and further incorporates a denoise and finetune strategy to improve quality while retaining multi-view consistency. Qualitative comparisons and evaluations on three tasks (generalization, compression, and retrieval) show that HyP-NeRF achieves state-of-the-art results. \n", "questions": "1.\tThe denoised images seem to be 3D inconsistent. I wonder why the fine-tuning process in the second step can retain the multi-view consistency and bypass the blurry results which are common when fine-tuning NeRF with inconsistent images. Lines 188-191 are not clear.  \n\n2.\tWhat about efficiency, i.e., Inference time and training time?\n", "limitations": "1.\tThe results are low-quality with blurry details and noises. Besides, the examples in the experiments are too simple.\n\n2.\tThe task is similar to NeRF-based generative models, like pi-GAN and EG3D. The paper lacks a discussion on the comparison between Hyp-NeRF and NeRF-based GAN.\n", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tIt is an interesting idea to take a hypernetwork to learn the category prior for NeRF, attaching the capacity of generalization to NeRF. \n\n2.\tThe fine-tuning procedure with a denoising network can improve the texture quality while retaining consistency.\n\n3.\tHyp-NeRF can extend to other downstream tasks, like the single image to 3D and text to 3D.\n\n4.\tThe paper is well-organized and easy to follow. \n\n5.\tThe experiments look convincing. It clearly supports the major contribution of the paper that a class of objects can be compressed in a unified network with the help of hypernetwork.\n", "weaknesses": "1.\tThere are not enough details on why fine-tuning procedures can retain consistency and avoid blurry results. Previous works, like instruct-NeRF2NeRF and StylizedNeRF, claim that fine-tuning with inconsistent images leads to blurry results.\n\n2.\tThe proposed method focuses on category-specific generalization similar to NeRF-based GAN. They both learn the prior of a class of objects and take NeRF as the 3D representation. Therefore, it is better to conduct an experiment comparing Hyp-NeRF and one of the 3D-aware GANs.\n\n3.\tThe examples in Figure 4 show low-quality results. Although the examples are simple, the result is not clear, with some noises in the appearance.\n\n4.\tIt lacks more quantitative comparisons with other methods like Table.4.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to appreciate the reviewer's efforts in studying our work and for their positive assessment of our contributions. Below, we elaborate on the questions raised, and provide additional comparisons to address all the concerns. \n\n1. **There are not enough details on why fine-tuning procedures can retain consistency and avoid blurry results. Previous works, like instruct-NeRF2NeRF and StylizedNeRF, claim that fine-tuning with inconsistent images leads to blurry results.**\n\n    This is addressed in the global rebuttal\n\n2. **The proposed method focuses on category-specific generalization similar to NeRF-based GAN. They both learn the prior of a class of objects and take NeRF as the 3D representation. Therefore, it is better to conduct an experiment comparing Hyp-NeRF and one of the 3D-aware GANs.**\n\n    - HyP-NeRF is primarily different in the manner that our main focus is on generating NeRFs, whereas the GANs aim to generate multi-view consistent images of high quality. For example, EG3D first generates an output in a lower resolution of $128 \\times 128$ due to a computationally expensive design. They then use a superresolution network to super-resolve the images to $512 \\times 512$. Further, these images are passed onto a discriminator, which guides multiview consistency for each image. In this case, the output is an image.  In contrast to EG3D, HyP-NeRF generates a NeRF trained through the volumetric rendering loss, and the resultant output (pre & post denoise and finetune) is a NeRF - that can be adopted in any downstream task. \n\n    - Moreover, compute requirements increase drastically for an architecture involving discriminators.  For example, an entire image should be rendered to be passed on to the discriminator in each forward pass. This would need one to sample number of rays equivalent to image resolution to render an entire image from the NeRF representation. On the other hand, we only need to sample a handful of rays that can fit in the memory in each training iteration and therefore are agnostic of the resolution enabling us to scale up to arbitrary resolutions. \n\n   - Lastly, we are primarily different in the task we aim to show. GAN-based methods heavily focus on unconditional sampling, whereas unconditional sampling is not trivial in our case. Instead, we evaluate the task of single-view NeRF generation with works that closely resemble our works, like CodeNeRF and PixelNeRF, that aim to learn a prior over many NeRF instances. \n\n3. **The examples in Figure 4 show low-quality results. Although the examples are simple, the result is not clear, with some noises in the appearance.**\n\n    - We would like to point out that Figure 4 has two of HyP-NeRF - (1) one with denoise and fine-tune (2) without denoise and finetune as a part of the ablation. Our final result is the 3rd column. The 4th column presents the ablation results, which are slightly inferior in quality compared to the 3rd. This contrast is also showcased in the video timestamp 2.45 to 3.27. \n    - While we understand quality is subjective, it is important to note that we showcase results on a resolution of 512 while much of the existing works in learning a prior over NeRFs remain in the resolution of 128. 3D GANs like EG3D showcase results in 512, but it is important to note that the method employs superresolution on the outputs of the NeRF renderings (which is originally at 128) that result in multi-view inconsistencies. On the other hand, we directly render at 512 at high quality. \n    - We would be happy to provide further clarification if R2 can point out the exact instances that do not feel up to par in the author-reviewer discussion phase.\n\n4. **It lacks more quantitative comparisons with other methods like Table.4.**\n\n    - Table 4 is an ablation table where we remove a part of our network design and compare it with the full network to demonstrate the removed part's importance. Therefore, in this table, we have not compared with the baselines. Table 3 is a retrieval experiment using the mapping network to retrieve results from our codebook. Since we are retrieving results from our own codebook, there is no baseline. For the other tables, we have compared with the respective baselines. We would be happy to provide further clarification on this point during the author-reviewer discussion phase.\n    - We have additionally added comparisons with CodeNeRF on ABO (128 and 512 resolutions) in the rebuttal PDF. \n    \n5. **What about efficiency, i.e., Inference time and training time?**\n    \n    This is addressed in the global rebuttal. "}, {"review_id": "c8P3HDora2", "reviewer": "Reviewer_vtAE", "summary": "In this paper, the authors present HyP-NeRF, a framework based on meta-learning principles, tailored for the learning of category-level NeRF priors. This is achieved through conditioning on latent codes with the assistance of hypernetworks. The NeRF model struggles with generalization across categories because of the high dimensionality of network space. To circumvent this issue, the authors propose a novel approach that estimates the multi-resolution-hash encoding and network’s weights via a hypernetwork conditioned on an instance specific latent vector. Furthermore, to denoise the predicted novel views during the fine-tuning phase, they utilize a VQ-VAE-2 model. The authors report significant performance gain in comparison with existing baselines. In addition, the authors showcase HyP-NeRF’s ability in downstream tasks such as single-view novel view generation, text-to-NeRF, and inpainting from cluttered scenes.", "questions": "Also listed in the weakness section for detailed reasons.  \n1.\tPlease run more experiments with different baselines, for example, Vision-NeRF, NeRFDiff, and additional dataset such as ShapeNet to validate HyP-NeRF’s performance.\n2.\tConduct ablation studies to explore if Sn and Cn can capture object shape and color, such as similar Sn (same category) or clustered Cn (if they are within the same category and similar color).\n3.\tAnother important aspect is computation budgets and rendering speed. Given that HyP-NeRF requires test time optimization, which Pixel-NeRF does not, a comparison between the two models’ computational requirements and rendering speed would be highly informative.\n", "limitations": "Yes, the authors have included limitation. Social impact does not apply. ", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1.\tThe authors propose a novel idea for conditioning the network weight and multi-resolution hash encoding via hypernetworks. This approach interestingly facilitates the sharing of prior knowledge within a category level among NeRF instances. Rather than optimizing a fixed set of parameters, the authors present an approach to learn a weight distribution and condition it directly on the instance-level latent code.\n2.\tBy providing a robust category-level prior, HyP-NeRF enables the training of a single model for objects within the same category. This approach effectively eliminates the tendency of overfitting to a single scene as vanilla NeRF and its variants. \n3.\tThrough the conditioning with the latent embedding, HyP-NeRF demonstrates the capacity to cooperate with other models for various downstream tasks. These applications range from text-to-NeRF to novel view synthesis in cluttered scenes. \n", "weaknesses": "1.\tMy main concern is hypernetwork’s ability to disentangle geometry and color using the latent code Sn and Cn. Taking the single-view reconstruction case as an example (Table 1), this problem is under-constrained as it requires 3D inductive biases learned from a large set of scenes like the target scene. This is apparent in methods such as Pixel-NeRF and its variants, which rely on 2D image features to generalize to unseen scenes after substantial training. However, HyP-NeRF’s approach to resolve the geometric ambiguity through the denoising the 2D outputs does not seem logically coherent. Additionally, it would have been beneficial for the authors to provide evidence showing category geometry for {Sn, Cn} in Section 3.1, and a demonstration of color clustering for scenes with similar colors. Moreover, cases where HyP-NeRF appears to overlook the shape or color of an object (as seen in Figure 6 row 1 and the supplementary video 3:52, 4:06, 4:31) raise further concerns about the ability of latent codes Sn and Cn to capture geometry and color.\n2.\tThe scope of the baseline and dataset appears insufficient to conclusively support the authors’ claim, which echoes my previous comment that HyP-NeRF only benchmarked against Pixel-NeRF under the ABO dataset. Comparative results with more baselines and datasets, such as VisionNeRF[1] and NeRFDiff[2], would have added credibility to their claims. This becomes particularly relevant given that in the supplementary material, HyP-NeRF does not outperform the baselines on the SRN dataset (e.g., PSNR of 21.02 versus 24.48 with Vision-NeRF). I do notice that authors claim that for SRN they did not apply denoising finetuning and SRN has different poses, but it would be hard to judge HyP-NeRF’s performance under different settings. Therefore, I would like to see HyP-NeRF’s performance on ShapeNet with other baselines under the same setting to validate its performance.\n3.\tAs the authors introduce hypernetwork, which adds extra computation on top of the original NeRF. I would like to see computation cost in terms of (# MLP parameters) and (# of FLOPS) in comparison with the baselines. This is particularly relevant considering that HyP-NeRF requires test time optimization. Additionally, how does the latent codebook size (Sn and Cn) impact the performance of the network? \n4.\tReadability issues, for example: \nln 170 (3) should end with comma “,”.\nln 171 uses “eqn.” to denote the equation, but later in ln 202 uses “Equation”.\n\n\n[1] Vision Transformer for NeRF-Based View synthesis from a Single Input Image https://arxiv.org/pdf/2207.05736.pdf\n\n[2] NeRFDiff: Single-image View synthesis with NeRF-guided distillation from 3D-aware diffusion\nhttps://arxiv.org/pdf/2302.10109.pdf\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's thorough analysis of our work and for raising their questions and concerns. In the global and local rebuttal, we have addressed the concerns and questions and provided additional comparisons and visualizations (rebuttal PDF).\n\n1. To demonstrate **shape and color disentanglement**, we add three qualitative results in the rebuttal PDF: \n    - Figure 1,\n        - We start with two object instances from the train set - $A$ and $B$ and denote their corresponding shape and geometry code as $A_s$, $B_s$, and $A_g$, $B_g$. \n        - Next, we switch the geometry and shape code and generate two novel NeRFs given by $\\\\{A_s, B_g\\\\}$ and $\\\\{B_s, A_g\\\\}$. \n        - Here, we can clearly see the disentanglement: geometry is perfectly preserved, and the color is transferred faithfully across the NeRFs. \n    - Figure 2: We fix a geometry code and interpolate the color codes. As shown, the geometry is perfectly preserved while the color smoothly transitions. \n    - In Figure 3, we cluster color and shape codes using TSNE plots and visualize instances from the clusters. As shown, each cluster represents a similar color or shape. \n \n    **HyP-NeRF appears to overlook the shape or color:** We propose two methods to query our learned prior - (1) test-time optimization (TTO) and (2) mapping network (MapN). \n\n      - TTO uses pixel-wise difference between the generated NeRF renderings and ground truth view. \n      - In the MapN, we encode the given single-view image (or text) using CLIP [39] encoder to obtain a feature vector. This feature vector is mapped to the hyper network's learned prior (separate mappings for shape and color codes), allowing the hypernetwork to generate the NeRF in a single forward pass. \n    - In MapN, we lose out on the **low-level details like fine shape and color details** through CLIP's encoding process. Therefore, the resultant generated NeRF does not exactly match the given input, as mentioned in the limitation section on Page 9, line 349. \n        - This indicates a limitation of the mapping technique **not** the prior learned by the hypernetwork as evident through TTO that operates within a generator's prior space. For this reason, we can obtain results on single (or multi)-view images, **even on views with severe occlusion** (video timestamps 5.39, 5.57, 3.51) that match the given input exactly.\n\n2. **The scope of the baseline and dataset appears ...**\n\n    - R3 suggests two baselines - NeRFDiff and VisionNeRF. \n        - NeRFDiff has been recently accepted to ICML 2023 (after our submission to NeurIPS), was submitted to arxiv very close to the  NeurIPS submission date, and hasn't released its codebase or dataset split on ABO yet. We will add NeRFDiff as a concurrent work in our related work section. \n        - VisionNeRF requires an exorbitant amount of compute, quoting from their paper - “16 NVIDIA A100 GPUs, where the training converges at 500K iterations” for training at a resolution of 128 x 128 on SRN. On the other hand, we train HyP-NeRF on a single NVIDIA 2080Ti RTX GPU. **HyP-NeRF specifically thrives and distinguishes itself at a higher resolution of 512 made of data with high-fidelity textures and shapes** - a resolution of 512 would need much more compute than 128  to train VisionNeRF.  In the supplementary, we have presented comparisons against CodeNeRF, FE-NVS, and VisionNeRF on SRN. \n    - To further add credibility to HyP-NeRF, **we compare it to another popular baseline - CodeNeRF - that employs a similar latent conditioning technique by modifying a NeRF to be a conditional NeRF, on ABO in the rebuttal PDF Table 4.** HyP-NeRF significantly outperforms CodeNeRF at both resolutions. \n\n3. **Baselines under the same setting**: We present our results on SRN with \"denoise and finetune\" in the rebuttal PDF, Table 1. We outperform the baselines on the Cars subset. Although we do not outperform on the Chair subset, the following points should be noted -\n    - We chose single-view NeRF as one of the several tasks we can perform to showcase generalization to novel NeRFs. The existing baselines are specifically trained for the task of single-view NeRF generation and modify the NeRF function by conditioning the NeRFs on additional features (like CNN features in PixelNeRF and VisionNeRF). Our NeRF is a standard NeRF that expects only the viewing direction and 3D point location as input. Thus, unlike the baselines, our NeRFs can be directly adopted in any downstream tasks that expect a standard NeRF as input. \n    - We incorporate test-time-optimization, which is known to fail at views that do not provide sufficient context. To further provide evidence of this, we compare with PixelNeRF (without the denoise and fine-tune step for a fair comparison) on **sparse-view NeRF generation ranging from a single view to 5 different views in the rebuttal PDF Table 3**. Our results significantly jump with two views and improve further as the views increase. This is also shown qualitatively in the supplementary paper, Figure 2. \n    - Finally, HyP-NeRF showcases many diverse downstream tasks (including many novel tasks not shown before for NeRFs) through many examples in the submitted papers and video ranging from compression, text-to-NeRF generation, generating NeRF from occluded and cluttered images **scraped directly from the internet without any preprocessing**, and so on.\n\n5. **how does the latent codebook size (Sn and Cn) ...**: The codebook size is equivalent to the number of training instances $\\times$ 512 where each code corresponds to one training instance. Therefore, reducing the codebook size will result in lower generalization on unseen datapoints as it might have seen less number of examples during training. We would be happy to discuss and clarify this further in the discussion phase.  \n\n6. We will replace eqn. with Equation. in the final version of the paper. \n\nRest of the concerns are addressed in the global rebuttal."}, {"review_id": "RDzDiyidcg", "reviewer": "Reviewer_bLah", "summary": "In the paper authors propose a novel method for NeRF based model which able to generalize. Authors use hypernetwork paradigm and Multi-Resolution Hash Encodings. \n\nThe paper is interesting but has two main problems: \n- two-stage training, where the second stage is applied to generate NeRF representation.  \n- authors do not mention a few important works in relation works and with comparison. ", "questions": "\n## Uncler parts of the paper\n\n1. Authors claim: “Our key insight is to use hypernetworks to generate both the network weights and instance-specific MRHEs.” and  “Each NeRF, f(·)n , is parameterized by the neural network weights, ϕn, and learnable MRHEs, hi ...” \n \nSo Hypernetwork produces hi or hi are trainable?  Such a sentence is misleading. \n\n2. Authors claim: “Given a set of NeRFs denoted by {f(ϕn,hn)}Nn, where N denotes the number of object instances in a given object category, we want to learn a prior Φ = {ΦS,ΦC},” \n\nIt suggests that we need the NERF representation of each object in the training dataset. Such a sentence is misleading. \n\n3. Hypernetwork is denoted by M. Usually, we use H for hypernetwork. \n\n4. “We want to design our hypernetwork, M,with trainable parameters, Ω that can predict NeRF parameters {ϕn, hn} given a conditioning code zn = {Sn,Cn}.  \n\nHypernetworks and conditioning mechanisms are not the same as the author notices in the introduction. I recommend to do not mixing such methods.  \n\n5. The sentence is unclear: “Here Sn Cn belong to codebooks, and that are trained within an auto-decoding fashion.” \n\nWhat does it mean? \n\n6. “However, Φ is not a known distribution like Gaussian distributions that can be naively queried by sampling a random point from the underlying distribution. “ \n \nWhy do we not add some term to loss to force latent to be Gaussian? ", "limitations": " The authors discuss the limitations of the model.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The generalization ability of NeRF based model is fundamental. \nThe paper shows an interesting application of the model. ", "weaknesses": "## Related works lack a few important works: \n\n1. The generalization ability of NeRF based model is significant. I believe the NeRF generalization models can be divided into two groups. Relatively small models which are similar to NeRF architecture like Voxel Base NeRF, tri-plane NeRF or MultiplaneNeRF, Pix2NeRF, and model which uses GAN, autoencoder, or diffusion model combined with NeRF.  \nThe model is in the second group since we use the network to generate another model like in Points2NeRF or  Hypernerfgan.  \n\n2. The related work does not mention the tri-plane NeRF mod model, which is now extremely important. The author mentions paper EG3D [7], but the relation between models should be highlighted. \n\n3. There are a few models which use hypernetworks in a similar fusion: \n\nPoints2NeRF: Generating Neural Radiance Fields from 3D point cloud  \nand  \nHypernerfgan: Hypernetwork approach to 3d nerf GAN \n\n4. Consequently, the sentence: “However, unlike us, they do not use a hypernetwork, and use the meta-learning algorithms only for initializing a NeRF, which is further fine-tuned on the multiview images” should be corrected. \n\n## Two-stage training \n\n5. In my opinion, the two-stage training is problematic since the second step can be applied in all models. In the second stage, we tune the model produced by the hypernetwork. \n\n6. In my opinion, all tables authors should give results for models with and without the second stage.  \n\n7. We need a pre-train autoencoder in the second stage.  \n\n## The model train generalizable NeRF priors \n\n8. It is unclear why we do not force laten to be generative by adding some distance to classical Gaussian prior, similar to VAE. \n\n9. Why do we not use an autoencoder instead of a trainable latent with a decoder? \n\n## Experiments \n\n10 . The experimental section is well organized, but in my opinion, the authors should compare models with GANS, autoencoders, and diffusion-based models in some sense.  \n\n11. The model should be compared with Points2NeRF, since the architecture is very similar.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their thoughtful comments. \n\n1. **Related works...**: We would like to thank the reviewer for the categorization. However, there could be several criteria for categorizing different techniques, like based on the downstream tasks the technique enables. Further, we will add the suggested related works in the final version. The sentence \"However, unlike us, they do not use a hypernetwork...\" is written in the context of Learned Intitialization [59], which is correct. We are happy to provide further clarification on this during the discussion phase. \n\n2. **Two-stage training pipeline**\n\n    - We don’t fully understand why the reviewer thinks it is problematic. \n    - While it is true that the second stage can be applied to the baselines -- **how to apply the second stage to the baselines is not trivial**.\n    - The expected input in our second stage is a standard NeRF whose parameters can be optimized through a standard volumetric loss on the set of denoised images. However, baselines like PixelNeRF and VisionNeRF work with a modified NeRF that is densely conditioned on pixel-level image features for a particular viewpoint (such as dense CNN-based features). In the fine-tuning step for such baselines, one would need to finetune both the NeRF and dense pixel-level feature parameters -- which is non-trivial.\n    - For a baseline like CodeNeRF, which could potentially be fine-tuned, we show the example of the denoising module in the added rebuttal PDF, Figure 4, right. As shown, the VQVAE2 fails to improve the results due to the amount of noise in the input. On the other hand, for HyPNeRF (Figure 4, left), VQVAE2 improves the images by refining the edges and improving the texture allowing us to perform fine-tuning on these denoised images. \n\n3. We have added results with and without denoise and finetune in the rebuttal PDF Table 1 and 3. \n\t\n4. **Pretrained autoencoder:** The autoencoder (VQVAE2) can be trained easily using images rendered from HyP-NeRF (as the input) and the ground truth NeRF renderings. We do not depend on any additional data. Moreover, the overall training regime is simple to follow. Given this, we are confident that we have proposed a novel technique that can be a valuable contribution to the NeRF community.\n\n3. **Design choice**: The technique of auto-decoding is inspired from an array of influential works like DeepSDF [37], Scene Representation Networks (SRN) [54], Light Field Networks [53], INR-V [46], CodeNeRF [17], and many more that learn high-quality prior over implicit neural representations (INRs) - in our case a NeRF. Following are additional reasons why auto-decoding is a more suitable technique to learn a prior in our case -  \n\n    - Building an encoder for our model without limiting the tasks is not trivial. For example, including an image-based encoder will add constraints of view and scale. pointcloud encoders would add a dependency on pointclouds adding a dependency on 3D supervision. Our current design only needs 2D supervision. \n    - Such a training technique would also make it hard to generalize to diverse inputs (such as text and single-or-multiview images). Further, an encoder would need to output two codes - shape and color - while ensuring the disentanglement between them. In our design, we learn a generic prior over NeRF representation (see rebuttal PDF Figures 1, 2, and 3). \n    - We also encourage the reviewer to refer to [54] and [17] for further clarification. \n    - **Adding a loss term to make it Gaussian**: The hypernetwork aims to learn a set of codes corresponding to a set of NeRFs. In this case, one could add a KL divergence loss to force these codes to follow a Gaussian distribution, but without \"random sampling\" as in Variational AutoEncoders, the space would be too sparse for us to force it to become a Gaussian distribution. Therefore we allow the hypernetwork to learn a prior by itself following the array of influential methods cited in the paper. \n\n3. **GANs and diffusion-based methods**: HyP-NeRF significantly differs from GAN-based works aiming to generate multi-view consistent images. We aim to produce a NeRF. Incorporating discriminators, GANs are often limited to lower resolutions, whereas HyP-NeRF thrives at a higher resolution of 512. Diffusion-based NeRFs (like DiffRF [31]) rely on an explicit version of NeRFs called radiance field and are thus also limited in the resolution they can output. We produce NeRFs in implicit space. Both of these works primarily show the task of unconditional sampling. On the contrary, our current baselines - PixelNeRF and CodeNeRF - also aim to learn a prior directly over NeRFs and do not provide comparisons with GAN or diffusion-based models. \n\n4. **Comparison with Points2NeRF**: Even though Points2NeRF follows a similar architecture as ours, this is also true for many other works like DeepSDF [37], LFNs [53], INR-V [46], and so on; However, like these works, Points2NeRF primarily aims to solve a very different task: converting a pointcloud to NeRFs. This would need us to either significantly modify our own method or the proposed Points2NeRF method to be able to compare both of them. \n\n5. **Unclear parts of the paper.**\n    1. $h_i$ is predicted by the hypernetwork. $f(.)_n$ consists of the MLP parameters $\\phi_n$ and MRHE $h_i$ and the hypernetwork predict both of them. \n    2. We will clarify the sentence: we only need multi-view image supervision. \n    3. Many different notations have been followed for denoting hypernetwork in the previous works, $\\Psi_\\psi$ in Light Field Networks[53], $d_\\omega$ in INR-V [46], and so on. \n    4. As the hypernetwork generates the output NeRF based on a given code, $z_i$ - we call it conditioning. This is the same style of notation used in [53] and [46]. \n    5. In the auto-decoding setup, $S_n$ and $C_n$ are trainable. \n    6. Addressed in point 3."}]

Author rebuttal:
We thank the reviewers for their feedback and positive assessment of our contributions. We address major factual errors and broad reviewer concerns in this global response and specific reviewer concerns/questions in the local response. We are confident that we will have addressed all their concerns with this rebuttal and additional visualizations and experiments. We would be happy to clarify any more questions during the author-reviewer discussion phase. 

**We refer Muw9 as R1, bkzC as R2, vtAE as R3, bLah as R4**

**Clearly, the reviewers appreciate several aspects of the paper**. R1, R2, and R3 appreciate that HyP-NeRF enables **several downstream tasks** (R1: “demonstrates the versatility of this learned prior”, R2: “can extend to other downstream tasks”, and R3: “cooperate with other models for various downstream tasks”). R1 & R3 find the **idea novel** (R1: “both the MLP parameters and the MRHE is novel” and R3: “authors propose a novel idea for…”). R2 also likes the idea of **denoising and fine-tuning** (“denoising networks can improve the texture quality while retaining consistency”). R3 & R4 appreciate our **experimental setup** (R3: “The experiments look convincing. It clearly supports the major contribution of the paper”, R4: “experimental section is well organized”). R1 finds the NeRFs to be **high quality** (“generate relatively high-quality NeRF thanks to…”).

1. **Factual Errors & Opinions:** We first want to address R4’s review, which, unfortunately, has several factual errors and opinions unsupported by facts. For example:

      - One of the major weaknesses raised by R4 is missing references to MultiPlaneNeRF, Pix2NeRF, HyperNeRFGAN, and Points2NeRF. We appreciate R4 for bringing this up. However, MultiPlaneNeRF was submitted to arxiv after the submission deadline, Pix2NeRF is already cited in the paper, and while we are happy to add references to HyperNeRFGAN and Points2NeRF, we want to note that they are not yet peer-reviewed. We hope the reviewer reconsiders this unfair criticism of our work.

      - The reviewer's comments on alternative design choices are arbitrary and unsupported by facts (review subsection titled "The model train generalizable NeRF priors"). In our paper, we justify our design choices and provide comprehensive ablations to evaluate them. Unfortunately, the reviewer has listed a number of arbitrary "what about x" design choices with no evidence to suggest that they would work better.

2. **R1: How is the condition retained?**

    - The denoise and finetune step operates on the output of the hypernetwork, $M$, that produces a NeRF, $f$, on a given condition. 
    - To retain the condition, we render $f$ to $m$ different poses (as mentioned in line 183). In our experiments, $m=91$.
    - The denoising module improves the texture of these $m$ rendered images. However, it is to be noted that this step only changes the input image marginally (this is also illustrated in the added rebuttal PDF, Figure 3), and thus the condition is not lost. 
    - Further, since $f$ is fine-tuned on the denoised images (which are already condition aware), the overall process does not lose the condition.

2. **R3: ... resolving geometric ambiguity through the denoising the 2D outputs does not seem logically coherent**

    - We do not resolve the geometric ambiguities through the denoising process. In fact, our assumption is the exact opposite - the input to the denoise and finetune step should already be multi-view consistent. 
    - 3D inductive bias is established through the standard volumetric loss proposed in [30] as mentioned in Equation 2 while training the hypernetwork, $M$. The NeRF predicted by $M$ is already multi-view consistent.

3. **R2 & R3: How are the results multi-view consistent and not blurry?**

    - While training a NeRF on multiview images that are highly inconsistent can result in inconsistency and blurry results, but we would like to point out that the multiview inconsistencies in the denoised images in the “Denoise and Finetune” step are negligible. 
        - To begin with, our images (to be denoised) are rendered from a fully-fledged NeRF, which is multiview consistent by design. This is evident from the experimental evaluations where it can be seen that HyP-NeRF significantly outperforms (on ABO) or at least is comparable (on SRN) to the baselines on the task of single (or sparse) view image to NeRF synthesis. This NeRF is rendered to 91 different views ($m=91$).
        - Secondly, even though VQVAE2 denoises the renderings, the denoised images are only marginally different from the input images, and therefore the multi-view inconsistencies are negligible (as clearly shown in the PDF Figure 2), unlike Instruct-Nerf2Nerf and StylizedNeRF that modify the images drastically resulting in inconsistencies and in blurry results. 
        - Our final trick is - instead of training a NeRF from scratch, only fine-tune the original predicted fully-fledged NeRF that is already very similar in geometry and texture to the denoised images, thus making sure that the final NeRF is not distorted and retains the initial geometric qualities while showcasing improved texture and fine geometry refinements (like smoothened edges).

**Computation Metrics (inference time, parameters, and FLOPS), rebuttal PDF, Table 2**. In the HyP-NeRF inference time, $m$ denotes the number of images used for the denoise and finetune process. Although we have the largest number of parameters (2$\times$ and 61$\times$ compared to PixelNeRF and CodeNeRF, respectively), our FLOPS are significantly lower than both the baselines (26$\times$ and 28$\times$ less compared to PixelNeRF and CodeNeRF, respectively). This is primarily because, given N query points in a scene, the forward pass through the hypernetwork (computationally expensive) happens only once for the scene. Only the NeRF predicted by the hypernetwork (the less computationally expensive part) is run for each query point.

Meta-review: All but one reviewer agree on accepting the paper, citing the good quality of the results, the novelty of predicting the multi-resolution hash encodings and the demonstrated generalizability of the approach. The remaining reviewer cites missing comparisons, but this AC finds that these references are either unpublished or not directly commensurate, and therefore not a reason for rejection. As such, this paper should be accepted.

Nevertheless, the authors are requested to add additional discussions to the related work to compare to any new references that came up that were published before the NeurIPS submission.
Decision: Accept (poster)