InsertNeRF: Instilling Generalizability into NeRF with HyperNet Modules
Authors: Yanqi Bao
Tianyu Ding
Jing Huo
Wenbin Li
Yuxin Li
Yang Gao
Venue: ICLR 2024 poster (2024)
Keywords: Neural Radiance Fields
Hypernetwork
Neural Rendering
Generalizability
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We present InsertNeRF, a novel paradigm that instills generalizability into NeRF and its derivative works.

Abstract: Generalizing Neural Radiance Fields (NeRF) to new scenes is a significant challenge that existing approaches struggle to address without extensive modifications to vanilla NeRF framework. We introduce **InsertNeRF**, a method for **INS**tilling g**E**ne**R**alizabili**T**y into **NeRF**. By utilizing multiple plug-and-play HyperNet modules, InsertNeRF dynamically tailors NeRF's weights to specific reference scenes, transforming multi-scale sampling-aware features into scene-specific representations. This novel design allows for more accurate and efficient representations of complex appearances and geometries. Experiments show that this method not only achieves superior generalization performance but also provides a flexible  pathway for integration with other NeRF-like systems, even in sparse input settings. 
Code will be available at: https://github.com/bbbbby-99/InsertNeRF.

Published as a conference paper at ICLR 2024
INSERTNERF: INSTILLING GENERALIZABILITY INTO NERF WITH HYPERNET MODULES
Yanqi Bao', Tianyu Ding", Jing Huo' Wenbin Li', Yuxin Li', Yang Gao
State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China yq bq_baol sma il n www.we.eoooooo edu. cn, (huojing, liwenbin, gaoy) @nju. edu. cn,
yiyuxin Lyuxinl6ésma 66 jjv. eau.
Applied Sciences Group, Microsoft Corporation, Redmond, USA
i innysding@niccroooff. com
ABSTRACT
Generalizing Neural Radiance Fields (NeRF) to scenes is significant chal- lenge that existing approaches struggle to address without extensive modifications vanilla NeRF framework. We introduce InsertNeRF, method for INStilling Ralizabiayy into NeRF. By utilizing multiple plug-and-play HyperNet mod- ules, InsertNeRF dynamically tailors NeRF's weights specific reference scenes, transforming multi-scale sampling-awee aware features into scene-specife representa- tions. This novel design allows for more accurate and efficient representations of complex appearances and geometries. Experiments show that this method not only achieves superior generalization performance but also provides a flexible pathway for integration with other NeRF-like systems, even in sparse input settings. Code will be available at: https: /tpp:://githble com/bbbbby 99/ InsertNeRF
INTRODUCTION
Novel view synthesis, fundamental discipline in computer vision and graphics, aspires to cre- ate photorealistic images from reference inputs. Early works (Debevec et al. 1996 Lin & Shum, 2004) primarily focused on developing explicit representations, facing challenges due to the ab- sence of 3D supervision. This issue has been alleviated by recent advancements in implicit neural representation research, which have led to improved performance. In particular, Neural Radiance Fields (NeRF) (Mildenhall al. 2021) has attracted significant interest. NeRF, and its derivative works, extract scene-specific implicit representations through overfitting training on posed scene images. Although NeRF uses neural scene representations effectively to yield realistic images, the scene-specific nature of these representations requires retraining when faced with novel scenarios.
An emerging topic known Generalizable NeRF (GNeRF) has recently garnered considerable at- tention for this challenge. GNeRF aims to learn scene- eecee-ineeeendentt inference approach that facili- tates the transition from references target view. Current methods enhance the NeRF architecture by adding structures that aggregate reference-iman image features, or reference features. Examples in- clude pixel-wise feature cost volumes (Johari et al., 2022), transformers (Wang et al.) 2022 Suhail et al. 2022), and 3D visibility predictors (Liu et al. [022].. However, fitting these additions into conventional NeRF-like frameworks such as mip-NeRF (Barron et al., (021), NeRI++ ZZhag et al. (2000), and others, often proves challenging and may fail to effectively harness the guiding potential of reference features. Furthermore, the extensive use of transformers or cost volumes can be time- consuming. Thus, intriguing question arises: Is it possible directly INStill GEneaaiizablliTy into NeRF (InsertNeRF) while staying faithful to the original framework?
A straightforward way accomplish this goal is to adaptively modify the NeRF network's weights, or implicit representations, for different reference scenes while preserving the original framework. The concept of hypeenetwork (Ha et al. (0166), which conditionally parameterizes target network, is an effective strategy in this scenario. The features extracted from the reference scene can be used as inputs to generate scene-specific network weights However, initial experiments in Tab. indi- cate that constructing a hypernetwork directly based on the NeRF framework can be inadequate, and often fails predict different attributes like emitted color and volume density. To address this, we
"Corresponding author.

---

Published as a conference paper at ICLR 2024
VeRF
mip- mip-NeRF NeRF
NoRPt+
Exiting GNoR P works (Colon (Celan-Depeh?
FOKS 7.29
FROT 13
PONK 10
Intiills Genuraaeeellaaaaaaa
TreerNNRRF
Insert NeR?
Insert NeRFes
Our InsetNoifF (Color Deych)
FANB 24.04
FAhIl 3082
Jahm
PORR 201e
In Jent
[s]
[5)
Figure 1: Overview of motivation. (a) We instill generalizability into NeRF-like systems, includ- ing vanilla NeRF, mip-NeRF, and NeRF++ frameworks, to achieve consistent performance across scenes without modifying the base framework requiring scene-specific retraining. (b) InsertNeRF significantly improves depth estimation compared to its original counterpart.
propose HyperNet modules, which are designed to serve as easily integrable additions to exist- ing NeRF-like frameworks. Owing to their flexibility, the resulting InsertNeRF excels at predicting the NeRF attributes by capitalizing on sampling-awan features and various module structures.
In InsertNeRF, we insert multiple HyperNet modules to instill generalizability throughout the frame- work's progression. This approach allows us to fully leverage the guiding role of scene features in determining the entire network's 's weights. Unlike existing works that solely utilize reference features inputs, InsertNeRR exhibits thorough grasp of reference scene knowledge. To further unlock the full potential of the HyperNet modules, it is crucial to aggregate scene features from set of nearby reference images. To achieve this, we introduce a multi-layy qulti-layer dynamic- static aggregation strategy. Compared to existing works, it not only harnesses the inherent completion capabilities of global features, but it also implicitly models occlusion through dynamic-static weights, as demonstrated on the depth renderings shown in Fig. Пb. By feeding the aggregated scene features into the HyperNet modules, we can generate scene-related weights based on the well-understood reference scene.
In summary, we make the following specific contributions:
We introduce InsertNeRF, novel paradigm that inserts multiple plug-and-play HyperNet mod- ules into the NeRF-like framework, endowing NeRF-like systems with instilled generalizability.
We design two types of HyperNet module structures tailored to different NeRF attributes, aiming for predicting acene-specific weights derived from sampling-awer awaare scene features. For these features, we further propose a multi-layer dynamic-static aggregation strategy, which models the views-occlusion and globally completes information based on the multi-view relationships.
We demonstrate that InsertNeRF achieves state-of-the-aa performance with extensive general- ization experiments by integrating the modules into the vanilla NeRF, Furthermore, we show the significant potential of our modules in various NeRF-like systems, such as mip-NeRF (Barron et 2021). NeRF++ (Zhang al. 2020), shown in Fig. [h. and in task with sparse inputs.
RELATED WORKS
2.1 GENERALIZABLE NEURAL RADIANCE FIELDS
Neural Radiance Fields (NeRF) by Mildenhall et al. 2021 and its subsequent derivatives (Barron al. 2022 Isaac-Medina et al. 2023 Bao et al. 2023) have gained momentum and capable of producing realistic images. However, a significant drawback is the need to retrain them for every new scene, which not efficient real-world applications. Recent works by (Wang et al. 2021 2022) introduce Generalizable Neural Radiance Fields that can represent multiple scenes, regard less of whether they are in the training set. To achieve this, many studies have focused on understanding the relationships between reference views and refining NeRF's
mechanism. For instance, NeuRay (Liu et al. 2022) and GeoNeRF (Johari et al. 2022) use pre-generated depth maps or cost volumes prior to alleviate occlusion issues. On the other hand, IBRNet (Wang al. 2021) and GNT (Wang et al.) 2022) implicitly capture these relationships through MLPs or transformers. Regarding the sampling rendering process, most works (Xu et al. 2023 Suhail et al. 2022 Wang ct al. 2021] Zhu et al. 2023) utilize the transformer- architectures to aggregate the sampling point features and replace traditional volume rendering with learnalle technique. However,
2

---

Published as a conference paper at ICLR 2024
common limitation is that most of these methods replace NeRF's network with transformers, making challenging to apply to NeRF derivatives and leading to increased computational complexity. Our research aims to address this by instilling generalizability into NeRF-like systems with scene-related weights while preserving its original framework and efficiency.
2.2 HYPERNETWORKS
The hypernetwork (Ha al. 2016 Chauhan al. 2023). often abbreviated hypernet, invented to generate weights for target neural network. Unlike traditional networks that require training from scratch, hypernets offer enhanced generalization and flexibility by adaptively parameterizing the target network (Alaluf et al. 2022 Yang et al.) 2022) Li et al. (2020). Leveraging these benefits, hypernets have found applications various domains including few-shot learning (Li et al., 2020), continual learning (Von Oswald et al.) 2019). computer vision (Alaluf et al. [2022), etc. In the realm
of NeRF, there have been efforts to incorporate hypernets to inform the training of the rendering process. For instance, (Chiang al. 20022) propose to train hypernet using style image features for
style transfer, while (Zimny et al. )2022) employ encoded point-cloud features for volume rendering. On related note, (Peng et al. (223)) utilize dynamic MLP mapping technique to create volumetric videos and (Kania et al.J (023) use a hypernet for 3D-aware NeRF GAN. In our work, instead of using the hypernet in NeRF framework directly, introduce plug-and-play HyperNet module, with a focus on providing reference scene knowledge to enable generalization to new scenarios.
METHOD
3.1 BACKGROUND
Neural Radiance Fields. Neural radiance fields (NeRF) (Mildenhall et al. 2021) is a neural repre- sentation of scenes. It employs MLPs map a 3D location E Rd and viewing direction d E g2 an emitted color E [0, 1] and a volume density a E [0, oo), which can be formalized as:
F(x,d;0) + (c,a),
where F is the MLPs, and @ is the set of learnable parameters of NeRF. Note that F can be further split into an appearance part Fopp and a geometry part Fgeo for the view-dependent attribute e and view-invariant attribute respectively (Zhang al. 2023).
Volume Rendering. Given ray in a NeRF, (t) = + td, where o is the camera center and is the ray's unit direction vector, sample K points, fr (t,) their color values c, and volume densities o;, The ray's 's color is then calculated by:
1, .... K), along the ray and predict
( Σ W,C,, where w/ exp ( Eo,o
(1 exp exp((o,)))).
where 6 is the distance between adjacent samples, and w, is considered to be the hitting probability or the weight of the i-th sampling point (Liu al. 2022).
views with known camera poses (In, P. P the goal of GNeRF is to synthesize a target novel view Iy based on these reference views, even for scenes not observed in the training set, thereby achieving generalizability. Current works (Wang) eet al. 2021 Liu et al. 2022: Wang et al., (2022) primarily focus on aggregating features along with the ray (t) from multiple reference views. The overall process can be outlined as:
Generalizable NeRF. Given N reference
F Feamll (fFricu (FR.
+ (c,o).
(3)
Here, II (x)) projects æ onto In, and (x(2) queries the corresponding feature vectors according to the projected points in n-th reference. Fview and Fample specifically denote the aggregation of multi-view features and the accumulation multiple sampling point features along the ray. These aggregations often carried out using common techniques such as MI.Ps and transformers.
3

---

Published as a conference paper at ICLR 2024
HyperNet Modules
HyperNet Modules
Novél Target View
Semplins
Photometnc
Volum
InzetbleRR
Apc
Node K
Statt Verght
Nedo- K
Ilypetett Xodtiee
Nodes K
Rafarence Images x
Acditiooal Decyier
4
Killi- Bysarrio Weights
+
Samppiing Pooé Project LI
Welghts 35as 0000
DFILM 000
Hyperhict Modales
Loss
U-Net
d)
(t) Iodiermee Posture Entraction
(c) Multt-logee Dynamic Static pggrgghoon Stritgy
Stpashit Miduus
Figure 2: Overview of InsertNeRF. (a) Within the NeRF framework, two types of HyperNet modules are inserted into Fee and Fopp- The HyperNet modules begin by (b) extracting features among multiple (N) reference images, and (c) using multi-layer dynamic-static aggregation strategy to aggregate the scene representations. Based on these scene representations and specially designed sampling-aware filters, (d) we develop dynamic MLPs and activation functions to guide the weights and instill generalizability into vanilla NeRF. Finally, (a)standard volume rendering is performed.
3.2 INSERTNERF
We introduce InsertNeRF, novel paradigm that instills generalizability into the NeRF frame- work, as illustrated in Fig. 2 While this method can be adapted to a variety of NeRF-based sys- tems (Sec. 4.4) we focus its application the vanilla NeRF in this section.
Overview. InsertNeRF achieves generalizability by inserting multiple HyperNet modules into NeRF These modules dynamically generate scene-specific weights for NeRF that are tailored to specific reference scene, denoted by Ω2. Based on it, by incorporating G2 into Eq. ( and combin- ing @ well f2g, NeRF's implicit representation (or weights) gains the generalizability across multi-scenes, which is explained in Appendix D.3] Specifically, it can be described as follows:
F(æ, O ST))
>c,w),
where 22g HyperNet ({Fremm (IF, (II,, (I,, (I.,((())))))>))))))))))
(4)
Comparing Eq. to Eq. 3 the key to InsertNeRF is the newly introduced architectures with dy- namic weights 12y, guided by the HyperNet modules based on specific reference inputs. The process begins with reference features extraction (Sec. 3.2.1 then multi-layer dynamic-static aggregation strategy is employed to fuse reference features from multi-views into scene features (Sec. 3.2.2). Subsequently, these aggregated scene features used to adaptively generate NeRF's sampling- aware weights via the HyperNet modules, which consist of sampling-aware filters, dynamic MLPs and dynamic activation functions (Sec. 3.2.3). These novel HyperNet modules are inserted before each MLP layer in the original NeRF, serving as an enhancement to the original MLP layers.
S.2..3.2.1
A notable aspect of InsertNeRF is its ability directly calculate the hitting probability w; in Eq. 2 for volume rendering, rather than simply outputting the volume density from Fgeo- This capability stems from the implicit modeling of the relationships between spatial points and the advantage of using multi-scale features. By combining F Fyco with F Fapp. the entire pipeline is trained end-to- end. Our unique design not only leads to superior rendering performance in GNeRF but also offers improved computational efficiency compared to cunsformer-basee structures (Wang et al. (022).
3.2.1 REFERENCE FEATURES EXTRACTION
In the exploration of reference images, generalizable methods often combine U-Net (Ronneberger et al. 2015) and ResNet (He et al. 2016) to extract local dense feature maps. These have proven effective in dealing with occlusion problems (Liu et al. (022).. Yet, there is a risk that an overem- phasis on local dense features might neglect global features, which are key occlusion completion
4

---

Published as a conference paper at ICLR 2024
and global inference (lizuka al. 2017). In work, take advantage of the spatial repre- sentation capabilities of multi-scale features to model complex geometry and detailed appearance. Specifically, we bring in global-local features successively update the Hypernet module's weights for Fee and dense feature for Fapp- Here, geometry requires multi-scale information to deduce occluded portions, while appearance concentrates dense fine-grained details. This process begins with multi-scale features Fi, from U-Net for each reference input I,, and can be expressed as:
Fin RETT X xC; 1 = 2, 1, n n= 1..1......
(5)
Here, W x H defines the image resolution, and C is the number of channels. During feature upsampling (as decreases), we output each layer's 's features, transitioning from global to local.
3.2.2 MULTI-LAYER DYNAMIC-STER AGGREGATION STRATEGY
Following the feature extraction, the next essential step is the aggregation of scene features. This is not only foundational for scene generalizability but also significantly impacts the effectiveness of the HyperNet modules. Most existing techniques focus primarily on preserving local geometry and appearance consistency, often employing visibility to model occlusions. A straightforward approach is to deduce the weight based on differences between reference and target views (Wang et al.] 2021]. We refer size, and it assigns higher weights to closer views in fixed manner. However, it may be unreliable as overlooks the correlation among the features. To remedy this, we introduce a dynamic prediction of multi-layer weights based on multi-scale features, involving blend of Maxpool-Mi -MLPs and Soft- Our approach hence adopts
static weight, denoted by MS B where B represents the batch
max layers, termed dynamic weights and denoted by MPY R BBxxxX a dynamic-static aggregation strategy for more nuanced multi-view scene feature aggregation.
Formally, given the corresponding features Fj E BBBxxNNx of B where di is the latent feature dimension, we calculate the weighted means and variances as 14 = En F @ MPY] E Bxxxx and en = Va [ @ MPY] E Bxxd, respectively. After concatenating Fj for each reference view with f4 and ey and halvely projecting its dimension, de-
points in the space,
noted as F2 E BxBxxxXxxx
it is applied to the static weight to obtain fái = E,, Fi o MST E
2xx and ūn = Vn F MST] E BB8xxxd/2 With Fi E ERBx BRBxxxx xd: representing the maximum features among all the reference views, and by concatenating i. and ö, and adding it to FITax, we accomplish the feature aggregation phase Friew in Eq. 4
The use of global-local dynamic weights leads to a significant enhancement in edge sharpness and the thorough completion of detail in the depth rendering images, as evidenced in Fig. [b. Note that unlike static weights, dynamic weights are guided by the relationships between multi-scale reference features and are learned with auxiliary supervision (Sec. 3.3)
3.2.3 HYPERNET MODULES
We now turn our attention to the HyperNet modules, the core element of InsertNeRF, integrated within both Fyea and Fapp- These modules are composed of three basic components: sampling- aware filters, dynamic MLPs (D-MLP), and dynamic activation functions.
Sampling-aware Filter. Unlike traditional hypemetworks, where reference features are gener- ally stable, those based pose-related epipolar geometric constraints in GNeRF are noisy. This noise complicates their direct use for weights generation. To address this challenge, we introduce sampling-awar filter that seeks implicitly find correlations between inter-samples and reduce noise within the reference features through graph reasoning. Specifically, following the aggrega- tion phase Friew, each aggregated point-feature regarded as node within graph structure. The relationships between these K points are then modeled using graph convolutions, formulated as:
H, = (I Ai) Frew Wi:
where Friew E R xx denotes the aggregated K point-features after Friews and A, and W represent the K x K node adjacency matrix and the learnable state update function, respectively. here denotes the identity matrix. This specific graph structure helps filter out noise by state-updating,
The advantages of this strategy in comparison with Wang et al. 2021 discussed in the Appendix D.I
5

---

Published as a conference paper at ICLR 2024
enabling the network concentrate on key features more effectively. Additionally, for intricate tiny structures, we adopt an approach inspired by Chen et al. (219). where linear layers across different dimensions are utilized instead of standard matrix multiplications within the graph convolutions.
Dynamic MLP. Using the filtered features H,, the HyperNet module is designed to generate cor- responding We,ghtH, and Bias H, within specific MLPs. This instills scene-awareness into vanilla NeRF, ensuring compatibility with Finpot- the output of the previous layer in the original NeRF framework. To enhance efficiency, these MLPs are integrated within the sampling-aware filter.
Dynamic Activation Function. Activation functions plays an essential role in the NeRF frame- work (Sizzmann et al. 2020). Traditional options, such the ReL.U function, may struggle with de- tail rendering and hinder the performance of D-MLPs due to their static nature. To address this, we introduce dynamic activation function. This function adaptively activates features in accordance with the unique characteristics of a given scene. Inspired by Perez et al. (2018), we propose the Dynamic Feature-wiss Linear Modulation (DFiLM), which the frequencies (FreqH, ) and phase- shifts (Shift are dynamically determined from Hi, allowing for more responsive activation.
The entire MLP-Block, including both the D-MLP and the activation function, can be expressed as: Foutput Sr,ft (Freq (Weight x Finput + Bias
To insert the HyperNet modules into the NeRF framework, Fourpur is subsequently fed into orig- inal NeRF's MLP layer for the final result. This yields superior performance, as validated through experimental results. We remark that the parameters not shared among the HyperNet modules. Moreover, their compact structures ensure that the impact on rendering efficiency is negligible.
HyperNet Modules in F geo and Fapp. In vanilla NeRF, goo and F Fagp serve distinct purposes but employ similar MLP structures, albeit with varying complexities. Fgeo focuses on geometric properties, whereas Fapp encodes view-dependent features using a smooth BRDF prior for surface reflectance. This smoothness can be facilitated by progressively exploiting guided scene features, along with reduction in both MLP parameters and activation functions for variable dl (Zhang al.) 2020 Recognizing this need, we propose a modified HyperNet module architecture specifically for F Fapp Our design employs a progressive guidance mechanism within Fapp. incorporating multiple parallel dynamic branches into the NeRF framework. The weights of the D)MLPP in each branch are progressively generated from the preceding branch, enabling the capture of reference features at dif- ferent levels for complex appearance modeling. Finally, the results of all branches are summed and used as input to the original MLP for predicting the RGB value. In accordance with our analysis, the DFiLM is not used in Fapp. setting it apart from other elements in the architecture. [Appendix C.I
3.3 Loss FUNCTIONS
The InsertNeRF pipeline is trained end-to-end utilizing three carefully designed loss functions.
Photometric loss. First, we employ the photometric loss in NeRF (Mildenhall et al. 202 i... the Mean Square Error (MSE) between the rendered and true pixel colors:
Σ -|(rm)
MSS
(8)
FER
where R is the set of rays in a batch, and C(r) is the ground -ruth RGB color for ray r E R.
Backbone loss. During end-to-end training, optimizing the feature extraction without addi- tional guidance poses considerable challenges. To address this, we draw inspiration from auto- encoding (Kingma& & Welling, 2013). By adding an additional upsampling layer and small decoder (used exclusively for loss computation), we seek to reconstruct reference images from encoded fea- tures. The original images serve as supervision, and we refer to this particular loss term as Lrackbone-
Dynamic weights loss. Initiating the learning of dynamic weights from scratch introduces dif- ficulties in understanding the connections among multi-scale features. To tackle this issue, we introduce an auxiliary supervision to encompass global-local information. Specifically, we let (r) E fx Xxxx3 represent the ground-trutt RGB values in corresponding reference images for K points in ray r within a batch R. We compute c/ = -En..reR (r) @ MPY, the weighted sum of these RGB values by dynamic weights. Utilizing c',, C" (r) is subsequently calculated ac- cording to Eq. (2), and supervised by the true color C(r). We designate this loss term LDY

---

Published as a conference paper at ICLR 2024
Table I: Comparisons of InsertNeRF against SOTA methods with Setting I.
NeRF Synthetic
LLFF
DIU
Methods
PSNR1
SSIM
LPIPS.
PSNR
SSIM
LPIPS]
PSNR
SSIMT
LPIPS)
PR (CVPR2021)
18.66
19.40
0.463
0.447
MYSNeRF (ICCV202))
25.15
0.853
0.159
21.18
0.691
0.301
23.83
0.723
0.286
(CVPR202L)
CoutraNeRF (CVPR2022)
25.44
0.842
0.178
27.69
0.904
0.129
GeoNeRFI (CVPR2022)
28.33
0.938
25.44
0.180
WaveNeRF' (ICCV202))
26.12
0.918
0.113
24.28
0.794
0.212
NeuRaa(CCVPR2022)
28.92
0.920
0.096
25.85
0.832
0.190
28.30
0.907
0.130
InsertNeRF (Ours)
26.44
29.75
Table Comparisons and ablations with Setting II.
Table 3: Results with sparse inputs.
PSNRT
NeRF SSIM
LPISS
PSNRT
SSIM1
LPIPS
Methods
PSNR1
3-view SSIM
LPIPS.
GNT (CLR22023
27.29
0.937
0.056
25.59
0.858
0.128
DieNNRR (ICC
Baseline (NcRF)
7.29
0.512
0.690
11:46
0.328
0.582
RegNeRF 2023)
0.587
0.336
HyperNetwork
0.081
GoCoNeRF (ICM) 2023)
18.77
0.596
0.338
Nese MIDDS
25.12
B.896
0:058
0.156
PreeNeRF (CVPR 2023)
19.63
0.612
0,308
InsenNeRRP (Ours)
27.57
0.936
0.156
25.68
0.861
0.126
InsertNeRF (wie
0,618
0.330
We formulate our final loss function as
LMSE + X1 bbuckboee + XzLDD
where A1 and X2 are hyperparameters controlling the relative importance of these terms.
4 EXPERIMENTS
We conduct comparative experiments with state-of-the-ard (SOTA) methods across different settings on mainstream datasets. Additionally, we validate the effectiveness of the proposed paradigm in the context of derivative NeRF-like systems generalizations and tasks involving sparse inputs.
4.1 EXPERIMENTAL PROTOCOL AND SETTINGS
Following IBRNet (Wang et al. 2021), GNeRF exploits the target-reference pairs sampling strategy during both the training and inference phases. Here, reference views are selected from a set of nearby views surrounding the target view. Specifically, N reference views are chosen from a pool of P x N (P Z 1) neighboring views of target, ensuring that the target view is excluded from the reference views. During the evaluation phase, we conduct evaluations using three metrics: PSNR, SSIM, and LPIPS, on well-established datasets such as NeRF Synthetic, LLFF, and DTU, More training and inference details are provided in the Appendix A and Algorithm in the Appendix
In our experiments, we follow two GNeRF settings existing methods:
Setting I. Following NeuRay (Liu et al.) 2022), we use three types of training datasets for training GNeRF, including three forward-facing datasets, the synthetic Google Scanned Object dataset and the DTU dataset. Note that we only select training scenes in the DTU dataset, excluding the four evaluation scenes. Following their setting in the experiments, we set 8.
Setting II. Following GNT (Wang al. 2022). train GNeRF using three and the Google Scanned Object dataset. Unlike Setting I, the DTU dataset is not used for either training evaluation. In addition, set N 10 in this setting.
datasets
4.2 COMPARATIVE EXPERIMENTS
We evaluate InsertNeRF for its generalization based on the vanilla NeRF framework, comparing its performance with SOTA methods under two GNeRF settings. Through extensive quantitative and qualitative experiments, we explore the advantages of our approach, even with fewer references.
Quantitative comparisons. We present quantitative comparisons with SOTA methods under Set- ting I and Setting II, as reported in Tab. and Tab. 2 For Setting I, the quantitative comparisons
GeoNeRF (ohari al. 2022) and WaveNeRF Xu al. and evaluated the distinct scenes with in the DTU dataset.
2023 trained original rectified images

---

Published as a conference paper at ICLR 2024
Coundiindh
Booollae
IBRNEE
NauRay
GNT
de
Figure 3: (a) Qualitative comparisons of InsertNeRF against SOTA methods. (b) A t-SNE plot of the scene-specific representations from our HyperNet modules. More analysis the Appendix D.2
Table 4: HyperNet modules ablations.
Table 5: MLDS aggregation strategy ablations.
Methods
PSNR1
LLFF SSIM
LPIPS
Weight
Static
Dynamic-
Weight
Supervision Layers Layer
Auxiliary-
Multi- Single-
PSNR1
SSIM
LLFF
LPIPS.
w/o D-MLP
23.33
0.774
0.198
24.88
0.827
0.154
Sampling Filter
24.67
0.815
0.158
/
25.55
0.851
0.128
w/o DFLLM
25.04
0.832
0.152
25.53
0.850
0.131
original MLP
25.44
0.131
0.838
0.139
InsertNeRF (Ours)
25.68
0.861
0.126
25.68
0.861
0.126
in Tab. T display our model's competitive results in evaluation datasets, with significant improve- ments in PSNR, SSIM and LPIPS in comparison existing SOTA methods. Specifically, PSNR and LPIPS exhibit substantial enhancements by -..16dB t and ~23.6 & respectively. For Setting II, InsertNeRF consistently outperforms the SOTA method (Wang al., 2022). as substantiated by the results in Tab. 2 We observe that these improvements become even more pronounced with fewer reference images, alongside higher efficiency, as demonstrated in subsequent sections.
Qualitative comparisons. Fig. T and Fig. (a) show the qualitative performances of our method against baseline and SOTA methods. InsertNeRF achieves improved geometric fidelity and clear edges, attributable to the completion capability of global features and the modeling of sample spatial relationships from graph structures. For more analysis and results, please refer to the Appendix E.5
4.3 ABLATION STUDIES
In Tab. 2. we analyze the core components of our method. The findings underscore the vital role of HyperNet modules in enhancing rendering performance, further evidenced in Fig. (b) that they instill scene-specific capabilities into NeRF's representation. Additionally, the multi-layer dynamic- static aggregation strategy proves to be essential. By integrating both modules, our novel paradigm instills generalizability into the NeRF framework, leading to a performance boost of approximately two to three times compared to the baseline model, i.e., vanilla NeRF, Additionally, we explore the underlying mechanisms driving the effectiveness of these components. More experiments, including single-scens scene setting, fine-tuning and ablation studies about Fapp in the Appendix
HyperNet modules. Tab, demonstrates that both the sampling-aware filters and dynamic activation functions are vital in the HyperNet modules, with the sampling-aware filters having more substan- tial impact. This could be due to the need to con- sider relationships between sampled points in the rendering process, which implicitly models occlu- sions, as noted in et al. (2022). Solely using dy- namic activation functions without D-MLP leads to marked decline in performance, highlighting the essential role of MLPs in neural representation. Furthermore, using only the HyperNet modules and omitting the original NeRF's MLP layers results in inferior performance, reducing training stability.
24
Eodia Tmoe
GNT
Figure 4: Performance and efficiency under different input-number on NeRF Synthetic.
Multi-layer dynamic-static aggregation strategy. In Tab. 5 ablation studies reveal the signif- icance of dynamic-static weights and multi-layer features. Using only dynamic weights appears more effective than static weight, likely because they are adaptively generated to suit different scene features. The auxiliary supervision for dynamic weights and multi-layer global-local features also play essential roles in aggregating multi-view features, underlining their importance in this strategy.
8

---

Published as a conference paper at ICLR 2024
Table 6: Quantitative results of InsertNeRF and Insert-mip-NeRF on multi-scale NeRF Synthetic.
Methods
Pull Res.
1/2 Res.
PSNK
1/4 Res.
1/8 Res.
Full Res.
1/2 Res.
SSIM
1/4 Res.
188 Bes.
Pull Res.
1/2 Res.
LPIPS]
144 Res.
1/8 Res.
mip- Nett"
12.94
13.05
13.18
13.33
0.700
0.636
0.563
0.609
0.424
0.450
0.670
0,530
InsortNeRE
27.60
28,58
29.45
29.83
0.926
11945
0.960
0,972
0.066
0,054
0,045
0,026
Insort mip NeRF'
28.15
29.17
30.23
30.02
0.935
0,951
0.966
0,977
0.056
0.045
0,037
0,029
Input number (N) and efficiency. Since feature extraction is consuminn, reducing the num-
ber of reference images substantially improves the training and inference efficiency of the network. Fig. illustrates the performance of InsertNeRF as the number of reference images (N) varies for training on NeRF Synthetic. In comparison GNT (Wang al., (022), InsertNeRR consis- tently demonstrates superior rendering performance and inference efficiency. This success can be attributed our novel generalization paradigm and the compact structures of the HyperNet modules.
4.4
FRAMEWORKS
Thanks to the plug-and blug-and-pop -ad--aaadddadaa-- advantage of the HyperNet modules, we extend the study of generaliza- tion to derived domains of NeRF, such as mip-NeRF (Barron et al. 2021] I) and NeRF++ (Zhang et al., 2020). areas that have rarely been discussed before. More details are provided in the Appendix
Insert-mip-NeRe Mip-NeRF's is a multi-scale NeRF-like model used address the inherent aliasing of NeRF, a significant challenge for GNeRF. Unlike Huang al. 2023), we explore how to instill generalizability into mip-NeRF. following its original setup. We report the qualitative and quanti- tative performance of mip-Nost mip-NeRF, InsertNeRF, and Insert-mip-Ner MiR--NRRF-a multi-scale NeRF Synthetic in cross-scene generalization setting (see Tab. 6 Fig. [and Fig. 5 One can observe that incorpo- rating the HyperNet modules not only enhances generalization for mip-NeRF but also addresses the inherent aliasing of InsertNeRF and improves the performance in the task of multi-scale rendering.
Insert-NeRF++. NeRF++, an unbounded NeRF. Fig. D and Fig. depicts qualitative and quanti- tative rendering results of Insert-NeRr++. It is evident that our approach has successfully instilled generalizability into the NeRF++ framework, doubling its PSNR compared to the original.
Sparse Inputs. Training NeRF with sparse inputs has become a notable focus re- cently (Niemeyer et al.] 2022: Yang et al.) 2023). Unlike our nearby reference views set- ting (Sec. [.1). this task often involves train- ing from limited number of fixed viewpoints to represent the entire scene. Under this set- ting. we relax constraints on selecting nearby viewpoints and uniformly select fixed sparse seen viewpoints to infer on arbitrary unseen viewpoints. Unlike existing works, our method trains on extensive auxiliary datasets, allowing us represent the entire evaluation scene from sparse inputs without retraining (see Tab. 33. To ensure fairness, all scenes in evaluation are excluded in the training phase. In conclusion, InsertNeRF offers a novel insight that employs pre-training on auxiliary datasets to enhance representation capabilities with sparse inputs. We believe that, through fine-tuning on the evaluation scene and incorporating existing technologies like geometry and color regularization, our paradigm will achieve even better performance under sparse inputs.
Holl 12kn
b
GoundTruth
InsertNeRy
Insert-alp-NeRF
Figure 5: Qualitative results of Insert-mip-NeRF
Please refer the Appendix E.5 for more results.
5 CONCLUSION
We present InsertNeRF, a novel paradigm that instills generalizability into NeRF systems. Unlike popular transformer- based structures, our HyperNet modules are efficiently incorporated into the original NeRF-like framework, leveraging reference scene features to generate scene- seene-specffic net- work weights. To achieve this, we design a multi-layer dynamic-static feature aggregation strategy for extracting scene features from reference images and employ sampling- lationships between sample points. Experiments on well-established datasets show that InsertNeRF and other Insert-NeRF-lik. frameworks can render high-quality images across different scenes with- out retraining. This offers insights for future works on: (i) generalization tasks for additional NeRF. like systems such as mip-NeRF 360; and (ii) sparse inputs tasks based on auxiliary datasets.
filters to explore re-

---

Published as a conference paper at ICLR 2024
ACKNOWLEFGEMENTS
This work was supported in part by the National Natural Science Foundation of China under Grant 62276128, Grant 62192783, and Grant 62106100; in part by the Jiangsu Natural Science Foundation under Grant BK20221441; in part by the Young Elite Scientists Sponsorship Program by CAST under Grant 2023QNRC001; part by the Collaborative Innovation Center of Novel Software Technology and Industrialization.
REFERENCES
Yuval Alaluf, Omer Tov, Ron Mokady, Rinon Gal, and Amit Bermano. Hyperstyle: Stylegan inver- sion with hypemeetorks for real image editing. In Proceedings of the IEEE/CVF conference on computer Vision and pattern recognition, pp. 18511-18521, 2022.
Yanqi Bao, Yuxin Li, Jing Huo, Tianyu Ding, Xinyue Liang, Wenbin Li, and Yang Gao. Where and how: Mitigating confusion in neural radiance fields from sparse inputs. arXiv preprint arXiv:2308.00 02908, 2023.
Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Partin-Bruaala, and Pratul P Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 5855-5864, 2021.
Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5470-5479, 2022.
SC Chan, Heung-Yeur Shum, and King-To Ng. Image-based renderrin and synthesis. IEEE Signal Processing Magazine, 24(6)222-33, 2007.
Vinod Kumar Chauhan, Jiandong Zhou, Ping Lu, Soheila Molaei, and David A Clifton. A brief review hypernetworks deep learning. arXiv preprint arXiv:2306.06955 2023.
Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su. Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 14124-1411 2021.
Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In European Conference Computer Vision, pp. 333 353-350.300 Springer, 2022.
Yunpeng Chen, Mareus Rohrbach, Zhicheng Yan, Yan Shuicheng, Jiashi Feng, and Yannis Kalan- tidis. Graph-based global reasoning networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 433 442, 2019.
Pei-Ze Chiang, Meng-Shiun Tsai, Hung- Yu Tseng, Wei-Sheng Lai, and Wei-Chen Chiu. Stylizing 3d scene via implicit representation and hypernetwork. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pp. 1475-1484, 2022.
Paul E Debevec, Camillo J Taylor, and Jitendra Malik. Modeling and rendering architecture from photographs: A hybrid geometry and image- based approach. In Proceedings of the 23rd anmual conference on Computer graphics and interactive techniques, pp. 11-20, 1996.
Emilien Dupont, Miguel Bautista Martin, Alex Colburn, Aditya Sankar, Josh Susskind, and Qi Shan. Equivariant neural rendering. In International Conference on Machine Learning, pp. 2761-2770. PMLR, 2020.
Kyle Genova, Forrester Cole, Avneesh Sud, Aaron Sarna, and Thomas Funkhouser. Local deep implicit functions for 3d shape. In Proceedingy of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 4857-4866, 2020.
David Ha, Andrew Dai, and Quoe V Le. Hypernetworks. arXiv preprint arXiv: 160.00960000900 2016.
10

---

Published as a conference paper at ICLR 2024
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Xin Huang, Qi Zhang, Ying Feng, Xiaoyu Li, Xuan Wang, and Qing Wang. Local implicit ray func- tion for generalizable radiance field representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 97-107, 2023.
Satoshi lizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. Globally and locally consistent image completion. ACM Transactions on Graphics (ToG), 36(4):1-14, 2017.
Brian KS Isaac-Medina, Chris G Willcocks, and Toby P Breckon. Exact-nerf: An exploration of precise volumetric parameterization for neural radiance fields. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 66-75, 2023.
Nishant Jain, Suryansh Kumar, and Luc Van Gool. Enhanced stable view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13208-13217, 2023.
Chiyu Jiang, Avneesh Sud, Ameesh Makadia, Jingwei Huang, Matthias Nießner, Thomas Funkhouser, et al. Local implicit grid representations for 3d scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6001-6010 2020.
Mohammad Mahdi Johari, Yann Lepoittevin, and François Fleuret. Geonerf: Generalizing nerf with geometry priors. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 18365-18375, 2022.
Adam Kania, Artur Kasymov, Maciej Zieba, and Przemyslaw Spurek. Hypernerfgan: Hypernetwork approach 3d nerf gan. arXiv preprint arXiv:2301.1163/ 2023.
Diederik Kingma and Max Welling. arXiv:1312.6114 14, 2013.
Auto-encoding variational bayes. arXiv preprint
Arno Knapitsch, Jaesik Park, Qian Yi Zhou, and Vladlen Koltun. Tanks and temples: Benchmarking large-scale scene reconstruction. ACM Transactions on Graphics (ToG), 1-13, 2017.
Johannes Kopf, Fabian Langguth, Daniel Scharstein, Richard Szeliski, and Michael Goesele. Image- based rendering in the gradient domain. ACM Transactions Graphics (TOG), 32(6):1-9, 2013.
Jonáš Kulhánek, Erik Demerr, Torsten Sattler, and Robert Babuška. Viewformer: Nerf-free neural rendering from few images using transformers. In European Conference on Computer Vision, pp- 198-216. Springer, 2022.
Yawei Li, Shuhang Gu, Kai Zhang, Lue Van Gool, and Radu Timofte. Dhp: Differentiable meta pruning via hypernetworks. In Computer gow, UK, August 23-28, 2020, Proceedings, Purt VIII 16, pp. 608 624. Springer, 2020.
2020: 16th European Conference, Glas-
Zhouchen Lin and Heung-Yeung Shum. A geometric analysis of light field rendering. International Journal of Computer Vision, 58:121-138 2004.
Shichen Liu, Shunsuke Saito, Weikai Chen, and Hao Li. Learning infer implicit surfaces without 3d supervision. Advances in Neural Information Processing Systems, 32, 2019.
Yuan Liu, Sida Peng, Lingjie Liu, Qianqian Wang, Peng Wang, Christian Theobalt, Xiaowei Zhou, and Wenping Wang. Neural rays for occlusion- aware image-based rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7824-7833, 2022.
Zhen Liu, Yao Feng, Michael J Black, Derek Nowrouzezahrai, Liam Paull, and Weiyang Liu. Meshdiffusion: Score-based generative 3d mesh modeling. arXiv preprint arXiv:2303.081333 2023.
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Oc- cupancy networks: Learning 3d reconstruction in function space. In Proceedings of IEEE/CVF conference on computer vision and pattern recognition, pp. 4460 4470, 2019.

---

Published as a conference paper at ICLR 2024
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106 106, 2021.
Michael Niemeyer, Jonathan T Barron, Ben Mildenhall, Mehdi SM Sajjadi, Andreas Geiger, and Noha Radwan. Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 5480-5490, 2022.
Sida Peng, Yunzhi Yan, Qing Shuai, Hujun Bao, and Xiaowei Zhou. Representing volumetric videos as dynamic mlp maps. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4252-4222 4262, 2023.
Songyou Peng, Michael Niemeyer, Lars Mescheder, Mare Pollefeys, and Andreas Geiger. Con- volutional occupancy networks. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16, pp- 523-540. Springer, 2020.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with general conditioning layer. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209. 14988, 2022.
Gernot Riegler and Vladlen Koltun.
Free view synthesis. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIX 16, pp. 623- 640. Springer, 2020.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomed- ical image segmentation. In Medical Image Computing and Computer-Assisted Intervention- MICCAÍ 2015: 18th International Conference, Munich, Germany, October 5.9, 2015, Proceed- ings, Purt III 18, pp. 234-241. Springer, 2015.
Sudipta Sinha, Drew Steedly, and Rick Szeliski. Piecewise planar stereo for image-based rendering. In 2009 International Conference on Computer Vision, pp. 1881-1888, 2009.
Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Con- tinuous 3d-structure-aware neural scene representations. Advances in Neural Information Pro- cessing Systems, 32, 2019.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in neural information processing systems, 13:7462-7473, 2020.
Przemyslaw Spurek, Sebastian Winezowski, Jacek Tabor, Maciej Zamorski, Maciej Zieba, and Tomasz Trzciński. Hypernetwork approach to generating point clouds. arXiv preprint arXiv:2003.00802. 2020a.
Przemystaw Spurek, Maciej Zieba, Jacek Tabor, and Tomasz Trzciński. Hyperflow: Representing 3d objects as surfaces. arXiv preprint arXiv:2006.08710. 2020b.
Mohammed Suhail, Carlos Esteves, Leonid Sigal, and Ameesh Makadia. Generalizable patch-based neural rendering. In European Conference on Computer Vision, pp. 156-174. Springer, 2022,
Johannes Von Oswald, Christian Henning, Benjamin F Grewe, and João Sacramento. Continual learning with hypernetworks. arXiv preprint arXiv:1906. wttp:/900000055,, 00695, 2019.
Chu Wang, Babak Samari, and Kaleem Siddiqi. Local spectral graph convolution for point set feature learning. In Proceedings of the European conference on computer vision (ECCV), pp. 52-66, 2018.
Peihao Wang, Xuxi Chen, Tianlong Chen, Subhashini Venugopalan, Zhangyang Wang, et al. Is attention all nerf needs? arXiv preprint arXiv:2207.132000 2022.
12

---

Published as a conference paper at ICLR 2024
Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul P Srinivasan, Howard Zhou, Jonathan T Barron, Ricardo Martin-Brualla, Noah Snavely, and Thomas Funkhouser. Ibrnet: Learning multi- view image-based rendering. In Proceedings of the IEEE/CVF Conference on Computer Viston and Puttern Recognition, pp. 4690-4699, 2021.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600- 612, 2004.
Muyu Xu, Fangneng Zhan, Jiahui Zhang, Yingchen Yu, Xiaoqin Zhang, Christian Theobalt, Ling Shao, and Shijian Lu. Wavenerf: Wavelet-based generalizable neural radiance fields. arXiv preprint arXiv:2308. 04826, 2023.
Jiawei Yang, Marco Pavone, and Yue Wang. Freenerf: free frequency regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8254-8200 2023.
Improving few-shot neural rendering with
Lingfeng Yang, Xiang Li, Renjie Song, Borui Zhao, Juntian Tao, Shihao Zhou, Jiajun Liang, and Jian Yang. Dynamic mlp for fine-grained image classification by leveraging geographical and temporal information. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10945-10954, 2022.
Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan. Mvsnet: tured multi-view stereo. In Proceedings of the European conference on computer vision (ECCV), pp. 767-783, 2018.
Depth inference for unstruc-
Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one or few images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4578 4587-4577222 2021.
Maciej Zamorski, Maciej Zieba, Piotr Klukowski, Rafal Nowak, Karol Kurach, Wojeiech Stokowiec, and Tomasz Trzciński. Adversarial autoencoders for compact representations of 3d point clouds. Computer Vision and Image Understanding, 193:102921, 2020.
Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun. Nerf++: Analyzing and improving neural radiance fields. arXiv preprint arXiv:2010.07492, 2020.
Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unreasonable efffcctiveness of deep features perceptual metric. In Proceedings the IEEE conference computer vision and pattern recognition, pp. 586-595, 2018.
Zicheng Zhang, Yinglu Liu, Congying Han, Yingwei Pan, Tiande Guo, and Ting Yao. Transforming radiance field with lipschitz network for photorealistic 3d scene stylization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2071 10712-207211 2023.
Linqi Zhou, Yilun Du, and Jiajun Wu. 3d shape generation and completion through point-voxel diffusion. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp- 5826-5833 2021.
Haidong Zhu, Tianyu Ding, Tianyi Chen, Ilya Zharkov, Ram Nevatia, and Luming Liang, Cae- samnerf: Calibrated semantic representation for few-shot generalizable neural rendering. arXiv preprint arXiv:2311.15510, 2023.
Dominik Zimny, T Trzciński, and Przemystaw Spurek. Points2nerf: Generating neural radiance fields from 3d point cloud. arXiv preprint arXiv:2206.01280 2022.
Dominik Zimny, Jacek Tabor, Maciej Zieba, and Przemyslaw Spurek. Multiplanenerf: Neural radi- ance field with non orainaiieeee representation. arXiv preprint 2rXiv:2305. 2023.
13

---

Published as a conference paper at ICLR 2024
A EXPERIMENTAL DETAILS
During the training process, the feature extraction and InsertNeRF-systems are trained with different learning rates in end-to-end manner, and we apply the Adam optimize the entire network with an exponentially decaying learning rate. In our experiments, λ is set as 0.1 and A2 is set as 1. During the cross-scene evaluation phase, we individually test the average PSNR, SSIM, and LPIPS metrics of all testing-view renderings for each scene and report the average values across all scenes, as shown in Tab. [ For the sake of fair experimental comparison and efficiency with (Wang al. 2022), we set the rendering stride size to 2 across all experiments. Empirical evidence demonstrates that this choice influences the evaluation of metrics, as shown in Tab. X Our model is implemented using PyTorch 1.11.0 and all experiments are conducted on Nvidia RTX 3090 GPUs with CUDA 11.4.
Metrics. We calculate the Peak Signal to-Noise Ratio (PSNR) and Structural SIMilarity (SSIM) (Wang et al. 2004) to evaluate rendering quality for target novel viewpoints. Additionally, the Learned Perceptual Image Patch Similarity (LPIPS) (Zhang et al. 2018) is adopted perceptual metric. For all our experiments, we report the average of the PSNR, SSIM and LPIPS under different testing views in multiple scenes to verify the generalization. It is noteworthy that, similar to the majority of GNeRF research (Liu et al.) 2022] Wang et al. 2022), our focus lies on foreground- centric metrics for both the NeRF Synthetic and DTU datasets during the evaluation process.
A.1 EVALUATION DATASETS.
NeRF Synthetic. The dataset consists of 8 synthetic objects with viewpoints uniformly sampled on the upper hemisphere. Each scene comprises 200 test images, wherein sampling strategy with an interval of 8 images is employed during the evaluation phase.
Local Light Field Fusion (LLFF). The dataset consists of 8 complex real-world scenes. Each scene includes real images, 1/8 of which are used as the evaluation dataset.
DTU Dataset. The dataset consists of 128 object-scenes, which is a classic dataset of MVS. In the experiments, select 4 scenes (birds, tools, bricks and snowman) for evaluation (Liu al. 2022
A.2 EXPERIMENTAL DETAILS FOR INSERTNERF
For all our experiments, we maintain the experimental protocols of NeuRay (Liu et al. 2022) and GNT (Wang et al. 2022). In Setting we also employ depth maps (Liu et 2022) as priors to assist the Hypernet modules to generate adaptive weights. Following (Liu et al. 2022), we randomly sample 2048 rays from Target-Reference pairs, and it trains for a total of 600,000 steps. In Setting II (Wang et al. 2022), we randomly sample 512 rays from Target-Refee Reference pairs, and it trains for a total of 400,000 steps without any priors. order to enhance training and inference efficiency, we sample K 64 points along each ray and simplify the volume rendering process in our paradigm.
A.3 EXPERIMENTAL DETAILS FOR
Insert-mip-NeRe NeR substitutes point-samplings with a sequence of conical-frustums and introduces the integrated positional encoding into the InsertNekF framework. In contrast to InsertNeRF, sample K = 65 positions for integrated positional encoding on conical-frustums. Throughout all experiments, we employ the multi-scale variants of NeRF Synthetic, such Full, 1/2, 1/4 and 1/8 Resolutions, to simulate multi-resolution scenes for training and evaluation as (Barron et al. 2021 The remaining experimental configurations follow those of mipNeRF and our Setting II.
A.4 EXPERIMENTAL DETAILS FOR INSERT-NERF++
Insert-NeRF++ divides the InsertNeRF's scene representations into foreground and background components and combines both for finally rendering (Zhang et al., 2020). In Insert-NeRE++, we avoid using inverted sphere parametrization and chose standard parameterization across foreground- background spatial ranges, which possesses the capability to offer precise spatial positions for projection process (x). Following (Zhang et al.) 22200). employ the Tanks and Temples dataset (Knapitsch et al. 2017). a real-world unbound dataset captured with hand-held cameras, for training and evaluation under cross-scenes setting. During the evaluation process, we separately
14

---

Published as a conference paper at ICLR 2024
report the renderings for the foreground and background (Zhang et al., 2020 The remaining experimental configurations also follow those of vanilla NeRF++ and our Setting II.
B
RELATED WORKS
B.1 MAGE-BASER RENDERING
Image-based rendering (IBR) (Chan et al. 2007) is a classic technique that aims to generate novel views within specific scene by warping and integrating pixel information from reference-view images. To ensure spatial consistency, most existing works simplify this challenge by resorting to estimated explicit geometry depth maps (Riegler & Koltun [2200] Jain al. (202).. In order to obtain proxy geometry, fiructureffrom-Motti
(SIM) (Sinha et al. 2009) and MultiViev Stereo (MVS) Yao et al. 2018) have recently attracted the attention of researchers in the field of IBR. How- ever, hints from explicit geometry without 3D supervision are unstable in most scenarios. Recently light field rendering (Lin & Shum 2004) has become one of the alternatives explicit representa- tions, which considers the lighting and reflection properties of the scene to ensure visually plausible rendering. Moreover, some works (Kopf al.) 20133 Chen et al.) 202T) focus on aggregating infor- mation from multiple reference views, which exploits the relationships between references and also implicitly solves the occlusion. As opposed to explicit geometry, these methods rely on constructing implicit representations to enable reasoning about novel views (Liu et al. 2019). Different from the above explicit or implicit scene customized representations for IBR, our method can be used in a large number of scenes simultaneously without retraining.
B.2 NEURAL SCENE REPRESENTATION
Representing the geometry and appearance of scenes with neural networks has been considered an alternative to 3D scene representations in recent years (Mescheder et al. 2019 Peng et al.] 2020).
Existing works demonstrate the potential of Multi-Layer Perceptrons (MI.P) (s) in implicit representa-
tions, which activate spatial features by continuous functions (Genova et al. 2020: Jiang et al. 2020).
Neural radiance fields (NcRF) (Mildenhall et al. (02I]) apply such functions for coordinate based representations, which use high-dimension target views. On basis, mip-NeRF (Barron et al. 2021) replaces rays with casting cones during volume rendering, changing the input of NeRF from points to cone frustums, and introduces an integrated positional encoding for multi-resolutions images. Subsequently, NeRF++ (Zhang et al.] 2020) and mip-NeRF 360 (Barron et al., 2022) further improve NeRF and mip-NeRF to adapt to distant targets under unbounded scenes. To further enhance representation efficiency, (Chen et al.) 2022) and (Zimny et al. (023)) proposed efficient representation methods to replace a large number of neural network parameters. Although derivative works has been a surge, similar to most IBR works, NeRF must also be trained for each novel scene, which is time-consuming in practice.
interpolation to produce photorealistic renderings of
B.3 GENERATIVE MODELS
With the development of generative models, 3D generative models have been widely discussed, abling the direct construction of 3D representations such as point clouds (Zamorski et al. 2020),
surfaces (Spurek et al.) 2020b), voxels (Zhou et al. 2021) and NeRF (Poole al.,) [2022). A signif- icant amount of works have leveraged techniques from image generative models and applied them 3D generation, including GAN (Kania al. 2023) and diffusion model (Liu al. 2023). In this work, we focus on some 3D generative models with HyperNetwork. (Spurek et al. (202aa) is an early work that builds variable size representations of point clouds with HyperNetwork. Then, Hy- perFlow (Spurek et al., (2020b) uses hypernetwork to model 3D objects as families of surfaces and Points2NeRF (Zimny et al. 2022 utilizes a HyperNetwork to generate NeRF from a 3D point cloud. Additionally, in recent years, there have been some NeRF works that focus this technique, they directly incorporate HyperNetwork into NeRF, as described in Section 2.2. However, the gen- eralizable NeRF task, such idea is suboptimal, overlooking the characteristic of different attributes, such as volume density and color. Furthermore, they struggle to capture the relationship between the inputs (reference images) in the target's sampling process. Therefore, this paper proposes two types of HyperNet module structures for F and F Fapp and Sampling-awar aware Filter separately to mitigate the aforementioned two issues.
15

---

Published as a conference paper at ICLR 2024
C IMPLEMENTATION DETAILS
C.I. HYPERNET MODULE ARCHITECTURE
HyperNet
HyperNet Modulle Appasirance
Irguut
Dynamic MLP
Dynamic A
Woggl
Bigs
Dynamic MLPT
Dynamic
Dynamic MLP N
MLP
tepst
Output
Imput
(a) HyperNet module in Fgeo
(h) HyperNet module in Fapp
Figure Detailed HyperNet module of Fgen and Fapp
Diverging from the conventional HyperNetworks (Ha et al. 2016). the direct prediction of distinct attributes, such as emitted color and volume density, within the NeRF framework often proves sub- optimal. Drawing analogy to the vanilla NeRF (Mildenhall et al. 202I). where different network depths were assigned for Fgeo and Fapp- it is essential for InsertNeRF to discuss distinct Hyper- Net module structures for them. Fgee plays pivotal role in the NeRF's geometric representations, which necessitates accurate inference and completion of the relationship between references based on global-local features. Based on this analysis, multi-scale features Hi are systematically intro- duced into the Fgeo's 's HyperNet modules, as shown in Fig. 6(a) After Eq. 77 Foutpu is subse- quently fed into an original NeRF's MLP layer. Note that we also incorporate an additional ReLU and BatchNorm after MLP to ensure training stability for HyperNet modules.
F Final Rel.U(BBtchNorm(MMM (Fonpu)))..
(10)
Note that in Fgeos the network's depth increases, we leverage denser features for guiding weights prediction, i.c., global to dense. Additionally, when the number of MLP layers surpasses that of the feature layers, Ho will be recurrently utilized in the remaining MLP layers.
Unlike Fap exhibits a heightened focus on dense features (Wang et al. 2022) and the smooth BRDF prior for surface reflectance (Zhang al. (020). As shown Fig. 6(b Fopp's 's HyperNet modules employ a parallel progressive generation paradigm and residual connection that respond to the desired smoothness. Specifically, given dense feature Ho.
F Final MLP Σ (Weight,,
Fmmuu
Fimput
[T, = Weighty, T- Ho
2
(11)
==
where Z represents the number of parallel branches. Note that DFiLM and dynamic bias not utilized in Fapp for the smooth BRDF prior.
C.2 PSEUDOCODE
In contrast the scene-customized vanilla NeRF and its derivative works, GNeRF primarily con- centrates on cross-scene rendering tasks without any retraining. As shown in Fig. T our HyperNet modules possess the capacity instill generalizability into NeRF-like systems. Here, further into the training processes of InsertNeRF-like systems. In Algorithm 1, due to [2Tr:Scene's adaptive response the stochastic sampling of scenes by Drescene (that represents data of training scenes), InsertNeRF-lie systems acquires inherent generalizability. where NeRF-like systems en- compass diverse frameworks, including but not limited to mipNeRF, NeRF++, NeRF- and others. In the evaluation phase, for any given Pr, we sample neighboring views (In, 1 rendering for I, with the pretrained @ like Systems and + IlpperNet shown in Algorithm 2.
delve
n=1 from Dresomees
C.3 GRAPH REASONING
Graph-basee based methods have been the focus of extensive research recently and shown to be an efficient way of relation reasoning (Wang al. 2018). Following the spatial properties, conceptualize
16

---

Published as a conference paper at ICLR 2024
Algorithm 1: Training for InsertNeRF-like Systems
Data: Training Datasets Drain
Result: @eRRF Systems. HyperNet
while I it do
Sample: r(44)
{ (ns Paln=1] Drescene
Drrain
HyperNet {Frico (LFa (Ln(((())))) i=1
STT
DittScenee
InsertNeRF-like Systems + NeRF-like Systems (r(()), (TSSeen
NeRF-like Systems HyperNot f
f <
1;
end
Algorithm 2: Testing for InsertNeRF-like Systems
Input : NeRF-like Systemss
② HypperNet. Drest and Random Pr in Testing Scenes.
Output: IT
Sample: r(t:) (I (uln + Drescene Drea } [Pr
((Freon (IF, (Ta(((())))))))) K
Ieeenn HyperNet
IT InsertNeRF-like Systems (r(4)), (eeescene)
all sampled points along ray in a fully connected graph to find correlations between sater-s and further update node features. Specifically, shown in Eq. (6). InsertNeRF initially predicts a learnable adjacency matrix A, to parameterize the edge weights between nodes, which models the relationships between sampled points. Subsequently, W is employed to update node states, mitigating the noise from epipolar geometric constrains. Furthermore, the identity matrix I is in- troduced to guide the learning process to pay more attention the intrinsic characteristics of node features. An naive approach is to calculate the adjacency matrix based on the similarity between node features or Euclidean distance, and update node states, similar existing graph convolution works. However, this is computationally expensive, especially for large number of MLP blocks in InsertNeRF. Inspired by Chen et al. (2019), A, and W are replaced by two separate linear layers operating in different dimensions, while the identity matrix is represented residual connection,
H Linear (Linear (Frsew) Fview-
(12)
D DISCUSSION
D. DIFFERENT FROM IBRNET ABOUT DYNAMIC-STATIC
For inputs, employ the global-dense features multi-layer inputs, compared to IBRNet's single-layer dense feature, not only retains the detailed information from dense features but also utilizes global features to predict occluded regions, demonstrated the depth renderings shown
Table Ablation studys for Multi-layer Dynamic-Statie Aggregation Strategy with IBRNet (Wang al. 2
NeRF Synthetic SSIM
LLFF SSIM
Methods
PSNR
LPIPS
PSNR
LPIPS
InsertNeRF w/o Max
27.44
0.930
0.061
25.52
0.854
0.131
FnRernssrNN
25.71
0.909
0.085
25.00
0.836
0.140
Multi-layer F
26.89
0.915
0.074
25.31
0.845
0.131
InsertNeRF (OUR)
27.57
0.936
0.056
25.68
0.861
0.126
17

---

Published as a conference paper at ICLR 2024
Reference foalures
Input
Volume Rendering
daptive Generation
HyperNet, Modules
Pose x
Max
Ingut
Pose N
Dynamic Weights
Static Weight
XN
Static Weight
Visual Map
A Auiilary Supervistio
Sum
Merge Multi-layers
(b) InsertNeRF
Figure The structures of multi-views feature aggregation parts IBRNet and InsertNeRF
Table
Ablation studies for rendering resolutions.
LLFF PSNR
DTU
Methods
SSIM
LPIPS
PSNR
SSIMI
LPIPS
InsertNeRF(Original Res)
26.22
0.838
0.184
29.88
0.929
0.096
I112 Res)
26.44
0.844
0.169
29.75
0.925
0.077
in Fig. Dp, where reports the rendering results of IBRNet (top) and InsertNeRF (bottom) in terms of color-depth. It is evident that InsertNeRF produces sharper edges and achieves more accurate depth predictions for background regions, even when occluded in the reference views.
For architecture, IBRNet generates the visual maps using features based on the MST, while our InsertNeRF directly predicts MPY using multi-layer features before MST, Intuitively, our strategy prevents excessive reliance on the MS and contributes to the adaptive inference of relationships be- tween reference-targel images. To provide additional validation, we conduct ablation experiments. Concretely, we replace the Multi-layer Dynamic-Static with the aggregation strategy from IBRNet and introduce identical multi-layer inputs into IBR-InserNNeRF fairness. As shown in Tab. 7 despite its simplicity, our approach yields significant improvements under GNeRF settings. In ad- dition, we also observe that the Multi-layer inputs still contributes to a notable enhancement in IBR-Inser,NeRFF which is consistent with the findings in Tab.
For supervision, in contrast to the direct generation of visual maps without any supervision, we introduce auxiliary supervision guide MPY in fully encoding global-dense features. As shown in Tab. 55] the significance of our auxiliary supervision cannot be disregarded. In summary, as shown in Fig- 7 compared to IBRNet, the Multi-layer Dynamic-Static Aggregation Strategy focuses predicting dynamic weights and combines them with static weight based on the multi-layer inputs and the auxiliary supervision to aggregate multiple reference features.
D.2 SCENE REPRESENTATION ANALYSIS
Ficus
Materials
Mic
The HyperNet modules instill generalizability into NeRF by generating scene-specific weights in the original framework. To verify this, we visualize the intermedi- representations of InsertNeRF through a t-SNE plot. As shown in Fig. ()). it is noteworthy that the reduced- dimensional features exhibit scene clustering characteris- tics in LLFF evaluation data, which may be attributed to the dynamic MLPs and activation functions in our Hy- perNet modules. NeRF Synthetic also exhibits similar trend. In Fig. although the scenes still possess cluster- ing characteristics, they appear relatively more dispersed, which may be attributed to significant disparities between evaluation viewpoints in the NeRF Synthetic.
Ship
Figure 8: A t-SNE plot of the scene- specific representations NeRF Syn- thetic's scenes.
18

---

Published as a conference paper at ICLR 2024
Table 10: Comparisons of HyperNet modules against SOTA methods on ShapeNet.
Chairs PSNRT
1-view SSIM
Chairs 2-views PSNR SSIM
Cars I-view SSIM
Cars 2-views
Methods
PSNRT
PSNR
SSIM
ENR Dupont et al. 2020
22.83
22.26
SRN
Sitzmann al., 20199)
22.89
0.89
24.48
0.92
22.25
0.89
24.84
0.92
Viewerrmer (ECCV 2022)
14.74
0.79
17.20
0.84
19.03
0.83
20.09
0.85
PixelNeRF (CVPR 2021)
23.72
0.91
26.20
0.94
23.17
0,90
25.66
0,94
DixelNeFF+HyperNee modules
24.51
0.92
26.71
0.94
24.18
0.91
26.05
0.95
D.3 WHY INSERTNERF DESIGNS COULD HELP NERF GENERALIZATION
Vanilla NeRF can be considered as an implicit representation used to depict a scene through the parameters of neural network, i.e. O described in Eq. (I)
natural idea is how to alter θ for different scenes 8, so that O, possesses the ability to represent S and generating different +., this be considered
this scene. By sampling different as endowing vanilla NeRF representation with generalizability in multi-scenes. However, unlike explicit 3D representations such voxels, meshes, and point clouds, constructing an implicit repre- sentation O, directly for a given s is challenging.
Therefore, in this paper. we introduce the HyperNet modules, which is invented generate weights for a target neural network, to address this issue. Through two types of the HyperNet modules we propose, scene-customization weights (parameters) 529 the NeRF framework are generated in given Here, we predict St by combining the feature extraction from reference images and the multi-Layer dynamic-static aggregation strategy. Finally, by combining and new weights S2 within the NeRF framework, we obtain O₂ that can adapt to different scenes s, described in Sec.
E ADDITIONAL RESULTS AND ANALYSIS
E.1 ADDITIONAL ABLATION STUDIES
During the evaluation phase, prior works employed different rendering resolutions, which has impact the metric. To investigate this issue, evaluate the render- ing performance at different resolutions without altering the training settings. In Tab, 3 reducing the rendering resolution not only improved rendering efficiency but also demonstrated performance enhancements in LLFF. However, in the DTUU dataset, a contrasting trend is evident, which may be attributed to its emphasis on foreground rendering.
Rendering Resolutions:
Parallel Branches Z: We further analyze the influence of the number of parallel branches Z on network render- ing performance, as shown in Tab, 9 When Z 1. i.c., Fapp is replaced by Fgeo with the same input, signif- icant performance drop occurs. This might be attributed to the implicit modeling of BRDF prior by Fapp. Further- more, with an increase in the number of branches, Fayp endows the NeRF framework with enhanced fine-detail generalizability.
Table Ablation studies for Fopp-
DTU SSIM
Z
PSNR1
LPIPS
Z = 1 (F (FFee)
29.03
0.918
0.086
7 7=
29.75
0.925
0.077
7=
29.83
0.925
0.075
E.2 RESULTS SHAPENET
In this section, we explore the performance of the InsertNeRF in ShapeNet under Chairs and Cars scenes. Due to the primary emphasis of InsertNeRF on multi-view settings and II, the vali- dation for multi-layer dynamic-static aggregation strategy under few-views settings is unnecessary. Therefore, we integrate the HyperNet modules into the original pixeINeRF Yu et al. (2021). altering its training inputs accordingly. As shown in the Tab. 10
modules exhibit significant improve-
19

---

Published as a conference paper at ICLR 2024
Table 11: The performance in different scenes and the results after Fine-Tuning in NeRF Synthetic.
PSNR
Method
FT
Lego
Chair
Drums
Ficus
Hotdog
Materials
Mic
Ship
Avg.
InR
30.00
31.97
25.59
26.08
35.04
28.91
35.13
30.08
30.35
MVSNeRF
27.21
IBRNet
30.05
GNT
31.38
33.70
26.98
29.55
36.95
29.11
33.35
30.54
31.45
NeuRay
32.35
InsertNeRF
31.93
34.41
27.76
29.08
37.25
31.46
36.43
31.98
32.54
(b) SSIM
Method
FT
Lego
Chair
Drums
Ficus
Hotdog
Materials
Mic
Ship
Avg.
InRF
0.939
0.969
0.910
0.915
0.969
0.922
0.972
0.888
0.936
MVSNeRF
0.888
IBRNet
0.935
NeuRay
0.960
InsertNeRF
0.962
0.981
0.937
0.959
0.988
0.957
0.978
0.931
0.962
(c) LPIPS
Method
FT
Lego
Chair
Drums
Ficus
Hotdog
Materials
Ship
Avg-
InsertNeRF
0.056
0.033
0.079
0.085
0.037
0.076
0.029
0.129
0.066
MVSNeRF
0.162
IBRNet
0.066
NeuRay
0.048
InsertNeRF
0.044
0.024
0.065
0.054
0.027
0.048
0.021
0.102
0.048
ments compared pixelNeRF Yu al. (2021). especially in the 1-view setting. It's also evident that compared to NeRF Synthetic, LLFF, and DTU, InsertNeRF shows less improvement on ShapeNet. This might due the relatively simplistic appearance and geometry of ShapeNet-Sceneee and our work primarily focuses multi-view settings as mentioned in (Kulhánek et al. 2022.
E.3 RESULTS FROM
We also explore the rendering performance of InsertNeRF after fine-tuning in various scenes. In contrast to the fine-tuning methodology adopted by (Liu et al. 2022). we fine-tune directly on the pre-trained model. Tab. [1 presents the performance of InsertNeRF across different scenes and the results after fine-tuning in NeRF Synthetic.
E.4 SINGLE SCENE RESULTS
Existing works (Wang al. 2022) also tend to focus on single-scene rendering within the framework of GNeRF. We conduct quantitative comparison with existing works in the single-scene setting and achieve satisfactory performance, as shown in Tab. I2
E.5 MORE QUALITATIVE RESULTS
We present additional qualitative results to further analyze the superiority of InsertNeRF: i). Fig and Fig. 10 report qualitative results in LLFF and NeRF Synthetic. ii). Fig. []] showcase more Color-Depth results LLFF and DTU datasets. iii). We also qualitatively analyze the generaliz- ability of the InsssteeRFssystemss including Insert- mipert- Fig. and Insert-NeRF++ Fig. 13 Note that the presence color distortion in the Insert-NeRF++'ss foreground rendering is observed, yet it does not impact the combined results, possibly attributable the replaced sampling process.
20

---

Published as a conference paper at ICLR 2024
Table 12: Comparison of InsertNeRF for single scene rendering on the NeRF Synthetic.
(a) PSNR
Method
Room
Fern
Leaves
Fortress
Orchids
Flower
T-Rex
Horns
LLFF
24.54
28.72
21.13
21.79
18.52
20.72
27.48
23.22
NeRF
32.70
25.17
20.92
31.16
20.36
27.40
26.80
27.45
GNT
32.96
24.31
22.57
32.28
20.67
27.32
28.15
29.62
InsertNeRF
32.55
24.88
22.59
31.82
21.18
28.39
27.49
29.37
(b) SSIM
Method
Room
Fern
Leaves
Fortress
Orchids
Flower
T-Rex
Horns
LLFF
0.932
0.753
0.697
0.872
0.588
0.844
0.857
0.840
NeRF
0.948
0.792
0,690
0,881
0.641
0.827
0.880
0.828
GNT
0.963
0.846
0.852
0,934
0.752
0.893
0.936
0.935
InsertNeRF
0.961
0.846
0.853
0.925
0.756
0.904
0.928
0.932
(c) SSIM
Method
Room
Fern
Leaves
Fortress
Orchids
Flower
T-Rex
Horns
LLFF
0.155
0.247
0.216
0.173
0.313
0.174
0.222
0.193
NeRF
0.178
0.280
0.316
0.171
0.321
0.219
0.249
0.268
GNT
0.060
0.116
0.109
0.061
0.153
0.092
0.080
0.076
InsertNeRF
0.063
0.121
0.109
0.062
0.152
0.070
0.085
0.074
F LIMITATIONS
In the majority of scenarios, higher number of sample points along the rays often leads to improved rendering performance. In essence, thanks to the transformer architecture, existing works (Wangl et al., 2022; 2021) can be trained on limited number of sample points (64 training samples) and evaluated on the more sample points (192 evaluation samples), resulting in elevated training effi- ciency and improved rendering performance. However, in the InsertNeRF-systen it is essential to maintain consistency the number of sample points between the training and evaluation processes. In order to ensure fairness in comparative experiments and strike the trade-off between training ef- ficiency and rendering performance, we set K = 64 both during training and evaluation, which imposes certain limitations on the rendering performance. Naturally, as we increase the number of sample points for training, the rendering performance will further improve.
G FUTURE WORK
We aspire to construct an all-encompassing InsertNeRF framework, endowing generalizability into various NeRF-derived works, such as TensoRF, NeRF-, NeuS, and forth. This facilitate existing or future NeRF research to transcend the constraints of scene-customization.
In addition, we have provided pre-trained model to address NeRF under sparse inputs in Sec.
4.4
While such models have demonstrated satisfactory performance without any retraining Tab. 3 we
still plan design fine-tuning approach for sparse inputs to further enhance rendering quality.
21

---

Published as a conference paper at ICLR 2024
Baseline NeRF
Scene
GoundTruth
IBRNet
NeuRay
GNT
InsertNeF
Figure Qualitative comparisons InsertNeRF against SOTA methods under LLFF scenes.
22

---

Published as a conference paper at ICLR 2024
GoundTruth
IBRNet
NeuRay
GNT
InsertNeRF
Figure 10: Qualitative comparisons of InsertNeRR against SOTA methods under NeRF Synthetic scenes.
IBRNet
InsertNeRF
NeuRay
InsertNeRF
Figure 11: Color-Depth results of InsertNeRF with existing works under LLFF and DTU's scenes.
23

---

Published as a conference paper at ICLR 2024
Baseline mip-NeRF
InsertNeRF
Insert-mip-NeRR
GoundTruth
Figure 12: Qualitative results of Insert-mip-NeRRinn in multi-scale NeRF Synthetic.
Foreground
Background
Combined
Baseline NeRF++
GoundTruth
Insert-NeRF+4
Figure 13: Qualitative results of Insert-NeRFt+ in Tanks and Temples.
24


Reviews:
[{"review_id": "scRmSBs3hr", "reviewer": "Reviewer_8PMb", "summary": "The paper targets the task of NeRF generalization, and proposes the HyperNet module before each MLP layer of vanilla NeRF, inspired by the concept of hypernetwork. The proposed method achieves comparable or even better performance compared to previous works.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Such a plug-and-play hypernetwork-based method can bring an extra performance boost for NeRF generalization, which is proved to be useful.\n\nComparison to previous works and several ablation studies were performed to verify the effectiveness of the aggregation strategy and each component proposed in the HyperNet module.", "weaknesses": "The paper may need more explanations or descriptions about the insight of the method, to demonstrate why such a strategy or technical designs could help NeRF generalization—the analysis of ``why'' is also important. \n\nThe experiments are very simple and short, in Figure 3, the methods are close in visualizations. As a prior and baseline work of generalizable work, IBRNet shows close performance to the proposed InsertNeRF.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ptpnxhQCfZ", "reviewer": "Reviewer_pJTL", "summary": "In the paper authors use multiple plug-and-play hypernetworks modules in NeRF based models to obtain generalization properties. The proposed method can be integrate with many different NeRF-based models.", "questions": "1. The model use hypenetwork so shoud be compared with hypenetworks based method like Points2NeRF or HyperNeRFGAN\n2. In Figure 1, the authors show that the introduced method works nicely with many NeRF framework but do not compare with other NeRF with generalization properties. It is misleading.\n3. Section 2.1 do not include Hypernetworks based NeRF. Authors do not mention TriPlaneNeRF or MultiPlaneNERF. Furthermore the generative models using NeRF should be mentioned. \n4. It is quite difficult to understand formula (3). F_view take as an input one point (x,d) or all points on the ray? F_sample take as an input one point (x,d) or all points on the ray?\n4. It is quite difficult to understand formula (4). it looks like  F_sample was change to NyperNet. What is an input to Hypernetwork and what is an input to F_sample? Such formula should be describe more carefully. \n5. Figure 2 is completely unclear.\n6. Authors write, \"While this method can be adapted to a variety of NeRF-based systems (Sec. 4.4), we focus on its application on the vanilla NeRF in this section.\" and then directly built the method on formula (3) dedicated to Generalizable NeRF.\n7. How difficult is to find parameters lambda_1 and lambda_2 in const function?\n8. In experimental section authors shud add experiments on ShapeNet data similar to pixelNeRF.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "1. The papers obtain good experimental results.\n2. The framework can be used for many different NeRF architectures", "weaknesses": "1. The paper is hard to read. It is unclear what is input to new meeting components. \n2. It is hard to understand the general idea of the model, and Fig. 2 is completely unclear.\n3. Genera formulas (3) and (4) are unceler.\n4. In the experimental section we do not have experiments with the ShapeNet-based dataset (see pixelNeRF).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hhUklEX3hp", "reviewer": "Reviewer_km3C", "summary": "This paper presents a method called InsertNeRF, which aims to instill generalizability into Neural Radiance Fields (NeRF) without extensive modifications to the vanilla NeRF framework. The method utilizes multiple plug-and-play HyperNet modules to dynamically tailor NeRF's weights to specific reference scenes, allowing for more accurate and efficient representations of complex appearances and geometries. The main contributions of InsertNeRF are: (1) introducing a novel paradigm that inserts HyperNet modules into NeRF-like systems to achieve generalizability, (2) designing two types of HyperNet module structures tailored to different NeRF attributes, and (3) demonstrating state-of-the-art performance and potential in various NeRF-like systems. The paper also discusses related works on generalizable NeRF and hyper-networks, and provides an overview of the method's background and implementation details.", "questions": "-How about the computational cost and efficiency of the proposed method compared with other methods?\n- It is suggest to public their Source code for the readers to better reproduce the proposed method.", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "-The idea of using HyperNet to solve the important problem of generalizable NeRF is a very good and promising pipeline. Thus, the novelty of the proposed method is very strong. \n-The proposed InsertNeRF can achieve the generalizability without extensive modifications to the vanilla NeRF framework. \n-HyperNet modules can dynamically tailor NeRF's weights to specific reference scenes, allowing for more accurate and efficient representations. \n- The proposed InsertNeRF achieves superior generalization performance and can be integrated with other NeRF-like systems. It also demonstrates state-of-the-art performance and potential in various NeRF-like systems, even in sparse input settings.\n- The writing and the presentation of the paper is good.", "weaknesses": "-All the contributions, points, and the details of methods and experiments have been clearly presented.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)