Mask-Based Modeling for Neural Radiance Fields
Authors: Ganlin Yang
Guoqiang Wei
Zhizheng Zhang
Yan Lu
Dong Liu
Venue: ICLR 2024 spotlight (2024)
Keywords: NeRF
Pretraining
Mask-Based Modeling
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Most Neural Radiance Fields (NeRFs) exhibit limited generalization capabilities,which restrict their applicability in representing multiple scenes using a single model. To address this problem, existing generalizable NeRF methods simply condition the model on image features. These methods still struggle to learn precise global representations over diverse scenes since they lack an effective mechanism for interacting among different points and views. In this work, we unveil that 3D implicit representation learning can be significantly improved by mask-based modeling. Specifically, we propose **m**asked **r**ay and **v**iew **m**odeling for generalizable **NeRF** (**MRVM-NeRF**), which is a self-supervised pretraining target to predict complete scene representations from partially masked features along each ray. With this pretraining target, MRVM-NeRF enables better use of correlations across different rays and views as the geometry priors, which thereby strengthens the capability of capturing intricate details within the scenes and boosts the generalization capability across different scenes. Extensive experiments demonstrate the effectiveness of our proposed MRVM-NeRF on both synthetic and real-world datasets, qualitatively and quantitatively. Besides, we also conduct experiments to show the compatibility of our proposed method with various backbones and its superiority under few-shot cases.

Published as a conference paper at ICLR 2024
MASK-BASED MODELING FOR NEURAL RADIANCE FIELDS
Ganlin Yang
Guoqiang Wei 2
Zhizheng Zhang 3+ Yan Lu
Dong Liu
University of Science and Technology of China 2 ByttDaacce Research Microsoft Research Asia yg166 1916660mmail uste.edee en weiguogiang. 9@byt edance. com {zhi (zhizzhang, yanlu aaalllummmccmmm .cm dongeliu@uste edw.:ussttceeee......ooo..oe..
ABSTRACT
Most Neural Radiance Fields (NeRFs) exhibit limited generalization capabilities, which restrict their applicability in representing multiple scenes using single model. To address this problem, existing generalizable NeRF methods simply condition the model image features. These methods still struggle to learn precise global representations over diverse scenes since they lack an effective mechanism for interacting among different points and views. In this work, we unveil that 3D implicit representation learning can be significantly improved by mask-based modeling. Specifically, we propose masked ray and view modeling for generalizable NeRF (MRVM-NeRF)) which is self-supervised pretraining target to predict complete scene representations from partially masked features along each ray. With this pretraining target, MRVM-NeRF enables better use of correlations across different points and views as the geometry priors, which thereby strengthens the capability of capturing intricate details within the scenes and boosts the generalization capability across different scenes. Extensive exper- iments demonstrate the effectiveness of our proposed MRVM-NeRF both syn- thetic and real-world datasets, qualitatively and quantitatively. Besides, we also conduct experiments to show the compatibility of our proposed method with var- ious backbones and its superiority under few-shot cases. Our codes are available at wttttps://githhi com Ganlin lin Yang/ Bang/MRVM MMRV NeRF
INTRODUCTION
Neural Radiance Field (NeRF)
Mildenhall et al.) 2021 has emerged as a powerful tool for 3D 2022| Yu et al. 202 Krdovich-Keil et al., 2022) and genera- Lin et al. 2023 Poole et al.) 2022). Though most NeRF-based still restricted to particular static scene, lim-
scene reconstruction (Sun et al.
tion (Niemeyer & Geiger 2021 methods render striking visual results, they iting their application in wide range. Recent works study Generalizable NeRF (Yu et al. 2021b Wang et al. 2021 2022b Reizenstein et al. 2021 Liu al. 2022) to model various scenes with single model, which can be directly applied to an unseen scene during inference.
Most of existing methods for generalizable NeRF sample image features from several visible refer- ence views as the conditions for learning scene representations. However, the correlations among the sampled features are not well exploited before. Previous masked modeling tasks, including masked language modeling (MLM) (Devlin et al. 20188) in natural language processing and masked image modeling (MIM) (Bao al.) 2021] Devlin al. 2018 Xie al.) 2022, He et al.. (022) in computer vision, exploiting such correlations among input signals by a maus--hhen-preiic task: masking out a proportion of inputs and trying to predict the missing information from the remaining ones. In this way, a high-level global representation could be learned, which is beneficial for downstream tasks. As for NeRFs, we find that the high-level global information learned through mask- pretraining, which we call the 3D scene prior knowledge, is also extremely useful for generalizable Neural Radiance Field. When applying for a novel scene, such a prior knowledge comes to use for reconstructing high-quality new scene from limited reference views.
"Corresponding authors: 7. Zhang and D. Liu

---

Published as a conference paper at ICLR 2024
To this end, propose innovative masked ray and view modeling (MRVM) tailored for NeRF, considering that there are correlations among the sampled points along rays and across the reference views naturally. Specially, we introduce pretraining objective predict the complete scene repre- sentations from the ones being partially masked along rays and across views, aiming to encourage the inner interactions the two levels. In view of the nature that NeRFs are implicit representa- tions, and motivated by Grill et al. (2020); Yu et al. (2022), we conduct our proposed predictive pretraining in the latent space and optimize it together with NeRF's original rendering task. After pretraining the generalizable NeRF model with our proposed MRVM, the model is further finetumed either across various scenes or on a specific scene. Such a simple yet efficient masked modeling design is actually model-agnostic method in the sense that it can be widely applicable to various generalizable NeRF models.
To demonstrate the effectiveness and wide applicability of our proposed MRVM, we conduct ex- tensive experiments both on commonly used large-scale synthetic datasets and more challenging real-world realistic datasets, based on both MLP-based and transformer-based network architec- tures. Quantitative and qualitative experimental results show that our proposed masked ray and view modeling significantly improves the generalizability of NeRF by rendering more precise geometric structures and richer texture details. Our contributions can be summarized as follows:
We find 3D implicit representation learning can be significantly improved by mask-based model- ing as MLM and MIM, when the inner correlations of 3D scene representations are harnessed in the right manner.
We present a simple yet efficient self-supervised pretraining objective for generalizable NeRF, termed as MRVM-NeRR. To our best knowledge, it the first attempt to incorporate mask-based pretraining into the NeRF field.
We conduct extensive experiments over various synthetic and real-world datasets based on dif- ferent backbones. The results demonstrate the effectiveness and the general applicability of our masked ray and view modeling.
RELATED WORK
2.1 NEURAL RADIANCE FIELDS
Generalizable NeRF Vanilla Neural Radiance Field (NeRF) introduced by Mildenhall et al. (2021) requires per-scene optimization which can be time-consuning and computationally expensive. To tackle with the generalization problem across multiple scenes, the network requires an additional
condition to differentiate them. Several works (Jang & Agapito 202I; Noguchi et al., 2021 Liu ctt al. 2021 use global latent code to represent the scene's ' identity, while more of the others Yu
et al. 20216 Wang et al. 2021| Liu et al. 2022 Zhang et al. 2022) extract a pixel-aligned feature map to be unprojected into 3D space. Generalizable NeRFs reconstruct the NeRF model the fly and can synthesize arbitrary views of a novel scene with a single forward pass.
Backbones Several earlier classical NeRF works (Mildenhall et al., 2021 Barron al. 1021 Yu
et al. 2021b Liu et al. 2022) adopt Multiple-Layer Perception (MLP) as the backbone for scene reconstruction. Recently inspired by great success of Transformer (Vaswani al.) (017) in com- puter vision area (Dosovitskiy et al. 2020), there have also been some attempts (Reizenstein et al.) 2021 Wang et al. 2022a to incorporate attention mechanisms into NeRF model. We evaluate the efficacy of our mask-based pretraining strategy on one representative work for each backbone.
2.2 MASKED MODELING FOR PRETRAINING
Mask-based modeling has been widely used for pretraining in various research domains. Natural Language Processing, Masked Language Modeling (MLM) is employed to pretrain BERT (Devlin ct al., 2018 and its autoregressive variants (Radford et al. 2018 2019 Brown et al. 2020). In
Computer Vision, Masked Image Modeling (MIM) (He et al.) 2022 Bao et al. 2021 Xie et al.)
2022 Baevski et al. 2022) has also gained significant popularity for sulf-super representation learning Different from the aforementioned works, we perform masking and predicting operations both in the latent feature space drawing inspirations from Grill et al. (2020); Yu et al. (2022), which
better coordinates 3D implicit representation learning for NeRF.
2

---

Published as a conference paper at ICLR 2024
Conme Branch
Volume resuering
stop grod.
mooting aver age
drnac
reference
Fine
Pront
veforesce teew
berd
LorgNE sieM
featoreal ruferurre 1
(voteer of reforenc nowe
Will makit
Figure 1: Overview of proposed MRVM-NeRF, To render image from target view, rays are cast into 3D space, and a series of points are sampled along each ray. These points are projected onto reference image planes to obtain pixel-aligned image features. We employ coarse-to- to-fine sampling strategy and mask portion of feature tokens input into the fine branch. The coarse and fine branches function the target and online networks, respectively. Our mask-based pretraining objective Cmrom aims to predict the corresponding latent representations of the target branch from the online ones within the latent space.
METHOD
We first briefly introduce the general framework for Generalizable Neural Radiance Field and ana- lyze the benefits incorporating mask-based pretraining strategy in Section [31] We then elaborate on the detailed procedure for mask-based pretraining, referred as masked ray and view modeling (MRVM), in Section 3.2 The pretraining objectives and implementation details are presented in Section 3.3 and Section 3.4 respectively.
3.1 GENERALIZABLE NEURAL RADIANCE FIELDS
Generalizable neural radiance field aims to share single neural network across multiple distinct scenes, which often involves a cross-scene pretraining stage followed by a per-scene finetuning stage. It often conditions the Neural Radiance Field image features aggregated from several reference views. Supposing S reference views {T2 12 Is) are available, pixel-aligned feature target
maps (F1, F2_ .... FS} can be extracted using 2D CNNs. To synthesize an image viewpoint, several rays are cast into the scene, N points {P1. P2. PN are then sampled along each ray. For each point p,, corresponding multi-view RGB components {c]. c² ...,, c } and feature components {f/, ft .... f can be simply obtained by projecting p. onto S reference image planes and sampling from p and pI~S For j E [1, f/ and cy are often merged and projected to a latent embedding h2 h. seen as the geometry and texture information acquired from reference view for point i, passes through several blocks of neural network modules for scene- specific information delivery and fusion. The network module can be either MLP or Transformer architecture. In this way. hy is mapped to the processed latent representation z/. {z/ 1 are then pooled among S reference views the global view-invariant latent feature z, which is finally decoded into volume density and color c; for ray-marching (Mildenhall et al. 2021).
Although the above-mentionel generalizable NeRF framework has made great success, uses re- construction loss only to supervise the learning of the mapping h² + z from end end, which is at the core of NeRF's reconstruction. We argue that such learning scheme lacks an explicit inductive bias to leverage information from other - 1 points on the ray and other 1 reference views. Prior works masked modeling have revealed that the mask-then-predict self-supervised task can encourage strong interactions between different input signals. Motivated by this, we pro- pose mask-based pretraining strategy tailored for NeRF, dubbed masked ray and view modeling, to better facilitate the 3D implicit representation learning. The learned 3D scene prior knowledge en- capsulates the correlations among point-to-point and across view-to-view, endowing the model with
3

---

Published as a conference paper at ICLR 2024
better capacity to effectively generalize to novel scenes with limited observations. We'll elucidate the mask-based pretraining strategy in detail in the following.
3.2 MASKED RAY AND VIEW MODELING
We adopt the hierarchical sampling procedure like most NeRF works (Mildenhall et al. [021] eu 202Tb; Liu 2022). At the coarse stage, we first use stratified sampling within depth range along the ray, forward the coarse- branch neural network to get the processed la- tent representation z and of, of as we de- scribed in Section 3.1 At the fine stage, ad- ditional points are sampled towards the rele- vant parts of the surface using importance sam- pling (Mildenhall et al. 2021). These points, together with those sampled at coarse stage, are processed by the fine-branch neural net- work, producing z and of, c/. We apply the masking operation to all the points processed at fine stage, and further supervise the mask- based pretraining task in projected feature space apart from the 2D pixel space.
coarse
ray
predict
ray
Figure 2: Illustration of masking operation. The striped rectangles denote the masked features which are randomly selected along the ray. The solid circles represent the points sampled at coarse stage and the hollow ones correspond to extra points sampled at fine stage. The rectangles with solid boxes are processed global view-invariant features by coarse and fine stage, and our MRVM task aims align them in the same feature space.
We denote the set of points on single ray at coarse stage as P and fine stage as Pl, while the former is subset of the latter:
P (pi- .i..... . P
p (pf pé................. ph,,
P (p PN. PKy
To facilitate the pretraining of generalizable NeRF, we propose to employ random masking opera- Specifically, we first perform random masking
tion at two levels, which is illustrated in Figure 2 at the ray-level to enhance the information interaction along each ray, where we randomly select a set of candidate points to be masked out from ps according to preset mask ratio 17. To promote the pesssage-passing across different reference views, we further employ masking at the view-level. For each selected masked point p!, we randomly mask out 1 ~ S feature tokens (hş ) acquired from S reference views.
Similar to Xie et al. (2022), we perform masking simply by replacing the corresponding masked feature token he with shared learnable mask token. In this way, along a specific ray, we randomly discard partial information at certain depths as well as from certain reference views, in accordance with our name masked ray and view modeling masking is executed along cast rays and across reference views, which aligns more closely with the fundamental nature of NeRF.
Advancing beyond previous generalizable NeRFs which solely rely on the pixel-level rendering loss, we aim to further regularize the pretraining process by incorporating constraints within the latent space. Motivated by BYOL (Grill et al. 2020) and several contrastive learning approaches, we designate the unmasked coarse branch as target branch and the masked fine branch as online branch. Our pretraining objective is to align the latent representations associated with the identically sampled points, yet processed through two branches individually. As illustrated in Figure z and z! are further projected to another latent space for feature alignment, which can be formulated as:
z (r 2(),
(3)
of Pred (sb, Proj (0, (0,z
(4)
where O a and are corresponding network parameters. The parameters of coarse-projector are updated by moving average from the fine-projector 0:
θ 70 + (1-7)0, +)
(5)
4

---

Published as a conference paper at ICLR 2024
where [0, is the moving average decay rate. The MRVM pretraining objective is defined as the feature prediction task in Z space:
Z
Z
mmrum
NG
7
2
|
(6)
Ne
2
2
22
No 1=1
112
|
Note that the constraint is only applied to the points appeared both at coarse and fine stages, i.e., the points in set p
Discussion While the mask-based pretraining strategy, as analyzed before, expected to assist generalizable NeRFs in learning useful 3D scene prior knowledge, there many mask-based pre- training options. Firstly, directly masking a certain percentage of pixels in reference images, as done in MIM, does not guarantee that each ray sampled during pretraining will be operated mask- ing. which hampers the pretraining efficiency. This is due the fact that the image features f/ are collected along the epipolar lines on reference image planes, not all of these epipolar lines will pass through the masked pixel regions. Secondly, masking applied feature tokens input into the fine branch, because the rendering results of this branch are used for evaluation. Our goal is to enhance the fine-branch's s generalization capacity when encountering a novel scene, which is endowed by our mask-learned prior knowledge. Since the coarse branch plays key role in guiding re-sampling near the surface manifold, is undesirable to downgrade its accuracy by masking out a portion of its in- puts. Finally, the latent representations output from unmasked coarse branch serve the prediction target, not only by the aspiration for a more streamlined architecture devoid of redundant modules, but also from the inspiration that each of the two branches is dedicated learning distinct scale knowledge of the scene, as claimed in MipNeRF (Barron et al., 202T). Consequently, this design choice enables the fine branch neural network to receive different-scale scene information distilled from the coarse branch. The ablation studies presented in Section 4.3 support our analysis.
3.3 TRAINING OBJECTIVES
To help the NeRF model learn better 3D implicit representations during pretraining stage, we also incorporate the conventional NeRF's volume rendering task, and the aforementioned mask-based prediction task in Section 3.2 acts as an auxiliary task to be optimized jointly.
During training, as long as we get the generated color and its corresponding density a as described in Section 3.1 we use the classical volume rendering equation (Kajiya & Von Herzen predict the rendering results:
1984) to
= =e=erp(-
σkōk),
C(r) = Σ (:(1- (o(1-exp(-o;)))ee
The rendering loss fuery is formulated as:
= -
(9)
where (r) and (r) are pixel values rendered by coarse and fine branch respectively, and C C*(r) the ground truth. The overall pretraining loss is:
Liotal
Σ (nnerf + A.mrrm),
(10)
PER
where À is set to balance different loss terms.
After pretraining, we perform finetuning most of the masked modeling works do. The projector and predictor are discarded and no masking operation is performed, only the rendering loss L used update the model until convergence.
5

---

Published as a conference paper at ICLR 2024
Table 1:
Quantitative results for category-agnosee ShapeNet-all and ShapeNet-unseen settings. Detailed breakdown results by categories could be found in Appendix. Best in bold.
ShapeNet-ail
ShapeNet-
Method
PSNR1
SSIM
LPIPS.
PSNR1
SSIM
I.PIS..
SRN
23.28
0.849
0.139
18.71
0.684
0.280
PixeINeRF
26.80
22.71
0.825
0.182
FE-NVS
27.08
0.920
0,082
21.90
0.825
0.150
VisionNeRF
28.76
0.933
0.065
NeRFormer
27.58
0.920
0.091
22.54
0.826
0.159
NeRFemecr+MRRM
29.25
0.942
0.060
24.08
0.849
0.117
Table 2: Quantitative results for seneeorr--pecifice ShapeNet-chair and ShapeNet-car settings, with
or 2 reference view(s). Best bold.
Chair L-view
Chair 2-views
Car L-view
Car 2-views
Method
PSNRT
SSIM LPIPS
PSNRT
SSIM
LPIPS
PSNRT
SSIM
LPIPS
PSNRT
SSIM
LPIPS
SRN
22.89
0.89
24.48
0.92
22.25
0.89
24.84
0.92
FE-NVS
23.21
0.92
25.25
0.94
22.83
0.91
24.64
0.93
PixelNeRF
23.72
0.94
0,90
CodeNeRF
23.66
0.90
25.63
0.91
23.80
0.91
25.71
0.93
VisionNeRF
24.48
0.93
0.077
22.88
0.91
0.084
NeRFormer
23.56
0.92
0.107
25.79
0.94
0.078
22.98
0.91
0,115
25.12
0.93
0.088
NeRForrer++MRM
24.65
0.93
0.076
26.87
0.95
0.058
24.10
0.92
0.084
26.20
0.94
0.067
3.4 IMPLEMENTATION DETAILS
To better demonstrate the wide applicability of our proposed mask-based pretraining strategy, we conduct experiments on both MLP- and transformer-based backbones. Specifically, we adopt NeuRay (Liu et al. 2022) as the MLP-based network, and utilize NeRFormer (Reizenstein et al.) 2021 the transformer-based model. The additional projector and predictor O, Φ and 0 are all simple two-layer MLPs. We sample 64 points along each ray at coarse stage, and extra 32 points at fine stage. The moving average decay rate T in Equation S is set to 0.99, the default mask ratio set to 50% and the coefficient À for loss term Lmrom is set to 0.1 during mask pretraining stage unless otherwise stated. Due the page limits, please refer to the Appendix for more details.
EXPERIMENTS
To validate the effectiveness of our proposed mask-based pretraining strategy, we conduct series of experiments under various circumstances. Specifically, we adopt transformer-basee backbone under synthetic NMR ShapeNet dataset (Kato et al., 2018), which introduced Section 4.1 We also employ MLP-based backbone under realistic complex scenes, with NeRF Synthetic (Niemeyer jet al., 2020), DTU (Jensen et al. 2014) and LLFF (Mildenhall et al., 2019) as the three evaluation datasets, presented in Section 4.2 We further conduct a detailed ablation study on 1) mask-based pretraining options, 2) mask ratios as well as 3) few-shot cases in Section 4.3
Baselines We take NeRFormer (Reizenstein al.) 2021 and NeuRay (Liu et al., 2022) as transformer-based and MLP-based baselines respectively. We denote the baselines without any mask-based pretraining NeRFormer and NeuRay. Accordingly, the model with MRVM pre- training followed by finetuning is referred as NeRFoorme+MMRM and NeuRay+MRVM. We use PSNR, SSIM (Wang et al. 2004) and LPIPS (Zhang et al. 2018 metrics for evaluation.
4.1 EFFECTIVENESS ON SYNTHETIC DATASETS
Settings NMR ShapeNet Kato et al. 2018 is large-scale synthetic 3D dataset, containing 13 categories of objects. Following the common practices introduced by PixeINeRF (Yu et al. 2021b), conduct experiments under three settings. 1) In canosory-agnostic ShapeNet-all setting, single model trained across all the 13 categories and evaluated over all the 13 categories as well. 2) In category-agnostic ShapeNet-unsece setting, the model is trained airplane, and chair classes

---

Published as a conference paper at ICLR 2024
while evaluated on the other 10 categories unseen during training. 3) In sategory-speccfice ShapeNet- chair and ShapeNet-car setting, two models are trained and evaluated particularly on 6591 chairs and 3514 cars respectively, which are subsets of the NMR ShapeNet dataset. For all these settings, we perform masked ray and view modeling simultaneously as we train the generalizable NeRF model across multiple scenes, and evaluate on testing scenes after finetuning without MRVM.
Results on the cafegory-agnostic setting Table I shows the quantitative results under category- agnostic ShapeNet-all and ShapeNet-unseen settings. Under the two settings, each object has 24 fixed viewpoints, with view randomly selected as the reference view while remaining 23 views used for evaluation. We compare our NeRFormer+MRVM with several dominant generalizable NeRF methods such as SRN (Sitzmann et al. 2019). PixeINeRF (Yu et al. 2021b). FE-NVS (Guo) ct al. 2022), SRT (Saijadi et al.] 2022) and VisionNeRE (Lin al. 2022). It can be seen that our baseline NeRFormer has already achieved comparable results with other baseline models. When incorporating mask-based pretraining scheme, its performance is further improved by large margin in PSNR, SSIM and LPIPS. It demonstrates that the 3D scene prior knowledge learned through our proposed masked ray and view modeling significantly improves the model's generalizability when applying on new scenes.
Results on the category-specific setting As for category-speciff ShapeNet-chair and ShapeNet-car settings, during training we ran- domly provide or reference view(s) for the network with 50 views around per object. Dur- ing testing, we fix 1 or view(s) as reference(s) and perform evaluation
NeRFormer MRVM
Reference
PixeINeRI
NeRFormer
G.T.
the rest of views. The experimental results are shown in Table 2] SRN (Sitzmann al., 2019), FE-NVS (Guo et al.] (2022), PixeINeRF Yu et al, 20216). Co- deNeRF (Jang & Agapito| 2021 and Vision- NeRF (Lin et al. 2022) are taken as baselines. The enhanced NeRFormer pretrained by our MRVM, i.e., NeRFormer+MRVM. achieves better results than previous methods both 1-
view and 2-view scenarios.
Figure 3: Visualizations of ShapeNet-all (row 1- 2), ShapeNet-unseen (row 3), ShapeNet-chair (row 4) and ShapeNet-car (row 5) settings. Our MRVM helps render novel views with more plau- sible structures, finer details and less artifacts.
Visual comparisons under the above- mentinmen three settings are shown in During pretraining, the MRVM- NeRF model encouraged to predict the masked information from the rest of available ones, which drives the model to capture the re- lationship between sampled points and across reference views. At inference, for novel scene, only partial information is accessible due to the limited reference views, so the mask-learned prior knowledge comes in handy for predicting the implicit representations of unseen parts. Therefore the rendering results have richer details and more precise structures compared to the baselines rendered with blurs and artifacts. More visual results could be found in the Appendix.
Visualizations
Figure 3
4.2 EFFECTIVENESS ON REALISTIC DATASETS
Settings To further demonstrate that our proposed MRVM is compatible with different NeRF ar- chitectures and is applicable beyond simple synthetic datasets, we adopt ML.--baedd NeuRay (Liu et al. 2022) as the baseline to evaluate on more challenging realistic scenes. Following its protocol, we first pretrain a generalizable NeRF across five datasets: Google Scanned Object dataset (Downs) et al. 2022), three forward-facing datasets (Mildenhall et al. 2019 Flynn et al. 2019 Zhou et al. 2018 as well as DTU dataset (Jensen et al. 2014) except for the testing scenes. The masked ray and view modeling is incorporated as auxiliary task when cross-scene pretraining. We use NeRF Synthetic (Niemeyer et al. 2020). DTU (Jensen et al. 2014) and LLFF (Mildenhall et al. (019) as evaluation sets following the train-test split manner of NeuRay (Liu al.) 2022). Afterwards,
7

---

Published as a conference paper at ICLR 2024
Table 3: Quantitative results on NeRF Synthetic, DTU and LLFF datasets. Our proposed MRVM proves to be beneficial for both cross-scene generalization and per-scene finetuning settings. Best bold.
Syntheti Object NeRF
Real Object DTU
Real Forward-facing LLFF
Method
PSNRT
SSIM
LPIPS
PSNRT
SSIMT
LPIPS
PSNRT
SSIMT
LPIPS
Cross: scne generalization
PixeINeRF
22.65
0.808
0.202
19:40
0.463
0.447
18.66
0.588
0.463
MVSNcRF
25.15
0.353
0.159
23.83
0.723
0.286
21.18
0.691
0.301
IBRNet
0.173
0.200
NeuRay
28.29
0.927
0,080
26.47
0.875
0.158
25.35
0.818
0.198
NeuRay+MRVN
MVSNeRF
27.21
0.858
0.162
25.41
0.767
0.275
23.54
0.733
0.317
Per-xcene fincuuning
NeRF
0.178
IBRNet
30.05
0.935
0.066
29.17
0,908
0.128
26.87
0.848
0.175
NenRay
0.172
NcuRay+MRVM
33.09
0.965
0.035
31.98
0.943
0.091
28.37
0.881
0.157
IBRNet
NeuRay
MRRM
G.T.
G.T. whole
Figure 4: Visualizations on NeRF Synthetic (first row), LLFF (middle row) and DTU (last row) datasets. Masked ray and view modeling aids in rendering images with enhanced texture details, reduced blurring and fewer artifacts.
finetune the generalizable NeRF model without masking operation either across five training sets, dubbed as cross-scene generalization setting, or target on specific scene in the three evaluation sets, denoted as per-scene finetuning setting.
Results The experimental results can be found in Table 3 We compare our NeuRay+MRVM with several well-known baselines including NeRF (Mildenhall al. 2021). PixeINeRF YYu et al. 2021b), MVSNeRF (Chen et al. 202T), IBRNet (Wang et al. 202T) and NeuRay (Liu et al. 2022 The prior knowledge acquired through mask-based pretraining substantially enhances the model's generalization ability when applied to new scenes in cross-scene generalization setting (the first large row Table Furthermore, the prior knowledge is still influential after executing per-scene finetuning (the last large row in Table 3). We show the visual comparisons under per-scene fine- tuning setting Figure More rendering results are placed in the Appendix. The model pretrained by our MRVM delivers better visual effects obviously. It is worth noted that the training and eval- uation sets encompass a wide variety of scenes, ranging from single object-centric scenes to more complex forward-facie indoor and outdoor scenes. This indicates that the proposed MRVM still works well under complex scenarios with complicated geometry. realistic non- Lambertian materials and various illuminations.
8

---

Published as a conference paper at ICLR 2024
Table 4: Ablation study on left: masking strategies and right: masking ratios.
Method
PSNR*
SSIMT
LPIPSL
#Params(M)
Mask ratio
PSNR1
SSIMf
LPIPS.]
NeRFormer
27.58
0,920
0.091
25.084
37.88
0.924
0.083
RGB musk
27.95
0.925
0.080
25.934
0.25
28.54
0.930
0.076
mask
28.58
29.25
0.942
0,060
Feat mask?
28.02
0,927
0,074
27.240
0.75
28.96
0.938
0.068
NRRF
29.25
0,942
0.060
25.151
28.02
0.927
0.080
4.3
ABLATION STUDY
We exccute ablation studies focusing three aspects described below.
Different masking strategies To validate the influence of differenn mask-based pretraining strate- gies, we evaluate with another three masking variants under calegory-agnostic ShapeNet-all setting.
RGB mask: Following MIM, we perform random block-wise masking reference images and incorporate an additional UNet-like decoder to reconstruct the masked region of pixels.
Feat mask We take the same masking strategy as descrihed Section 3.2 but introduce an additional decoder to recover the masked latent feature h? from the output representation z/.
Feat m Similar to our default MRVM but the target network is replaced by copy of the fine-branch instead of the coarse-branch, with parameters updated via moving average.
The comparisons are shown in Table 4 (left). Although all masking options yield some degree of improvements, our final proposal MRVM achieves the most significant improve- ment with the minimal additional parameters, demonstrating its superiority over other mask- ing strategies. Please refer to the Appendix for more details about the three variants.
Table 5: Ablation study for few-shot scenarios on NeRF Synthetic dataset.
#views
method
PSNR
SSIM1
LPIPS
NeuRay
29.78
0.940
0,078
NouRay-MRVM
30.88
0.948
0.060
NouRay
25.01
0.871
0.145
20-4
NeuRay-MRVM
26.61
0.891
0.114
NeuRay
10.3
NeuRay-MRVM
24.03
0.846
0.159
Different masking ratios We conduct an empirical study on the mask ratio 17 under canesory-agnostcc ShapeNet-all setting in Table 4 (right). We separately mask 10%, 25%, 50%, 75% and 90% points along each ray. relatively-large y) proves to be more beneficial, as it poses more challenging pretraining task. It compels the model to develop comprehensive understanding of the entire 3D scene on global scale, rather than merely interpolating information from adjacent points. While too large 1 may lead to too difficult task, it is inappropriate for pretraining to learn sufficient 3D scene prior knowledge.
Few-shot scenarios We validate that our MRV M-NeRF could help alleviate the limitation of NeRF's requirement on relatively dense inputs, referred as the few-shot scenarios in Table Specifically, adopt the per-scene finetuning setting using NeRF Synthetic dataset. The default configuration in Table 3 uses 100 views for finetuning and renders each image from 8 reference views. For few-shot scenarios, respectively. The results indicate that our MRVM achieves more significant improvements under few-shot scenarios, which implies that the prior knowledge learned through mask-based pretraining holds substantial potential to alleviate the relatively dense inputs required by NeRF.
decrease the training views to 50, 20. 10 and reference views to 5, 4,
CONCLUSION
In this paper, we propose masked ray and view modeling (MRVM), mask-based pretraining strat- egy specially designed for generalizable Neural Radiance Field. By enhancing inner correlations among rays and across views, MRVM shows great efficacy and wide compatibility under vari- ous experimental configurations. We hope our work could promote the development introducing mask-based pretraining into 3D vision research field.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was partially supported by the Natural Science Foundation of China under Grant 61931014.
REFERENCES
Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vec: A general framework for self-supervised learning speech, vision and language. International Conference Machine Learning, pp.
PMLR, 2022.
Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint 2tt::2106.082544 2021.
Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Partin-Bruuall,, and Pratul P Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 5855-5864, 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su. Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14124-14133, 2021.
Wenyan Cong, Hanxue Liang, Peihao Wang, Zhiwen Fan, Tianlong Chen, Mukund Varma, Yi Wang, and Zhangyang Wang- Enhancing nerf akin to enhancing Ilms: Generalizable nerf transformer with mixture-of-view-experr. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3193-3204, 2023.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 1929, 2020.
Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas B McHugh, and Vincent Vanhoucke. Google scanned objects: A high-quality dataset of 3d scanned household items. In 2022 International Conference on Robotics and Automation (ICRA), pp. 2553 22660250. IEEE, 2022.
John Flynn, Michael Broxton, Paul Debevec, Matthew DuVall, Graham Fyffe, Ryan Overbeck, Noah Snavely, and Richard Tucker. Deepview: View synthesis with leameed gradient descent. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 2367-2376, 2019.
Sara Fridovich-Kan Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pp. 5501-5510, 2022.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach self-supervised learning. Advances in neural information processing systems, 33:21271-21288 2020.
Pengsheng Guo, Miguel Angel Bautista, Alex Colburn, Liang Yang, Daniel Ulbricht, Joshua M Susskind, and Qi Shan. Fast and explicit neural view synthesis. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pp. 3791-3800, 2022.
10

---

Published as a conference paper at ICLR 2024
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
Wonbong Jang and Lourdes Agapito. Codenerf: Disentangled neural radiance fields for object categories. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp- 12949-12958, 2021.
Rasmus Jensen, Anders Dahl, George Vogiatzis, Engin Tola, and Henrik Aanæs. Large scale multi- view stereopsis evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 406 0666-413220 2014.
James T Kajiya and Brian P Von Herzen. Ray tracing volume densities. ACM SIGGRAPH computer graphics,
1984.
Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. Neural 3d mesh renderer. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3907-3916, 2018.
Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng. Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Yu Liu, and Tsung-Yi Lin. Magic3d: High-resolution text-to-3d con- tent creation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 300-309, 2023.
Kai-En Lin, Lin Yen-Chen, Wei-Sheng Lai, Tsung- Yi Lin, Yi-Chang Shih, and Ravi Ramamoor- thi. Vision transformer for nerf-based view synthesis from a single input image. arXiv preprint arXiv:2207.05736, 2022.
Steven Liu, Xiuming Zhang, Zhoutong Zhang, Richard Zhang, Jun-Yan Zhu, and Bryan Russell. Editing conditional radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5773-5783, 2021.
Yuan Liu, Sida Peng, Lingjie Liu, Qianqian Wang. Peng Wang, Christian Theobalt, Xiaowei Zhou, and Wenping Wang. Neural rays for acclusion-aware image-based rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7824-7833 2022.
Ben Mildenhall, Pratul P Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ra- mamoorthi, Ren Ng, and Abhishek Kar. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Transactions on Graphics (TOG), 38(4):1-14, 2019.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes as compositional genera- tive neural feature fields. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 11453-1 1464, 2021.
Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Differentiable volumet- ric rendering: Learning implicit 3d representations without 3d supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3504-35011 2020.
Atsuhiro Noguchi, Xiao Sun, Stephen Lin, and Tatsuya Harada. Neural articulated radiance field. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5762-5772 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 32, 2019.
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint 220i /4988, 2022.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language under- standing by generative pre-training. 2018.
11

---

Published as a conference paper at ICLR 2024
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Shordone, Patrick Labatut, and David Novotny. Common objects in 3d: Large-scale learning and evaluation of real-life 3d cat- egory reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10901-10911, 2021.
Mehdi SM Sajjadi, Henning Meyer, Etienne Pot, Uis Bergmann, Klaus Greff, Noha Radwan, Suhani Vora, Mario Lučić, Daniel Duckworth, Alexey Dosovitskiy, et al. Scene representation trans- former: Geometry-free novel view synthesis through set-latent scene representations. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6229-62233 2022.
Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein, Scene representation networks: Con- tinuous 3d-structure-awart neural scene representations. Advances in Neural Information Pro- cessing Systems, 32, 2019.
Cheng Sun, Min Sun, and Hwann- Hwann-Tzong Tzong Chen. Direct voxel grid optimization: Super-fast con- vergence for radiance fields reconstruction. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5459-5469, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- fion processing systems, 30, 2017.
Dan Wang, Xinrui Cui, Septimiu Salcudean, and Z Jane Wang. Generalizable neural radiance fields for novel view synthesis with transformer. arXiv preprint 2XXi:22206005375 2022a.
Peihao Wang, Xuxi Chen, Tianlong Chen, Subhashini Venugopalan, Zhangyang Wang, et al. Is attention all that nerf needs? arXiv preprint arXiv:2207.1322 2022b.
Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul P Srinivasan, Howard Zhou, Jonathan T Barron, Ricardo Mauain-Brualla, Noah Snavely, and Thomas Funkhouser. Ibrnet: Learning multi- view image-based rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4690-4699, 2021.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. JEEE transactions on image processing, 13():600 612, 2004.
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9653-9663, 2022.
Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. Plenoctrees for real-time rendering of neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3762-5761, 2021a. Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one or few images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2021b.
Tao Yu, Zhizheng Zhang, Cuiling Lan, Zhibo Chen, and Yan Lu. Mask-based latent reconstruction for reinforcement learning. arXiv preprint 12X00 2022.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 586-595, 2018
Xiaoshuai Zhang, Sai Bi, Kalyan Sunkavalli, Hao Su, and Zexiang Xu. Nerfusion: Fusing radi- ance fields for large-scale scene reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5449-5458. 2022.
Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. arXiv preprint 1805 09817, 2018.
12

---

Published as a conference paper at ICLR 2024
A MORE EXPERIMENTAL RESULTS
A.1 RESULTS ON SYNTHETIC DATASETS
Category-agnostic ShapeNet-all and ShapeNet-unseen settings The overall numerical results have already been presented in the main paper. The detailed results with breakdown by cate- gories are provided in Table Table We provide additional visual results in Figure Figure 11 for ShapeNet-all setting and Figure [2] Figure [3 for ShapeNet-unseen setting, respectively. We randomly sample object instances for each of the testing categories in ShapeNet dataset and show visual comparisons to PixelNeRF (Yu et al. 2021b) and our baseline NeRrermer.
Category-specific ShapeNet-car and ShapeNet-chair settings The quantitative comparisons on PSNR. SSIM and LPIPS are available in the main paper. SRN (Sitzmann al. 2019), FE-NVS (Guo al. 2022) and CodeNeRF (Jang Agapito (202)) do not provide LPIPS result in their paper. We calculate LPIPS result for PixeINeRF (Yu al., 2021b) using author-provided checkpoints. More visualizations are shown in Figure 14 and Figure [5| We use view-64 and view-64, 104 as input view(s) for one-shot and two-shot cases, For each scenario we randomly sample object instances, and show visual comparisons to PixelNeRF (Yu et al. 2021b) and our baseline NeRFormer.
IBRNet
NeuRay
NcuRay-MRVM
G.T.
G.T. whole
Figure 5: Visualizations for cross-scene generalization on NeRF Synthetic (first row), LLFF (mid- dle row) and DTU (last row) datasets.
A.2 RESULTS ON REALISTIC DATASETS
For real-world cross-scene generalization and per-scene finetuning settings, as we illustrated in the main paper, adopt NeuRay (Liu et al. 2022 baseline and evaluate three datasets: NeRF Synthetic (Niemeyer et al. 2020). DTUU (Jensen et al. (004)) and LLFF (Mildenhall et al.] 2019). The quantitative results are presented in T the main paper, and more visualizations for cross- scene generalization setting and per-scene finetuning setting are shown in Figure 5 and Figure respectively.
A.3 RESULTS ON OTHER BASELINES
We also provide the additional experimental results of adding our proposed masked ray and view modeling (MRVM) another advanced generalizable NeRF baseline GNT (Wang et al. 2022b), on NeRF Synthetic (Niemeyer et al.) 2020) and LLFF (Mildenhall et al. 2019) datasets respectively, and compare with another of the-art art method GNT-MOVE (Cong al. 2023). The default setting for novel-view synthesis is put in Table [ and the few-shot setting is located in Table 9 We conclude that the proposed masked ray and view modeling consistently benefits under all the cases.
13

---

Published as a conference paper at ICLR 2024
IBRNet
NeyRay
MRRR
G.E.
G.T. whole
Figure 6: Visualizations for per-scene finetuning on NeRF Synthetic (first row), LLFF (middle row) and DTU (last row) datasets.
Table 6: Detailed results of category-agnoste ShapeNet-all setting, with a breakdown by cate- gories. This table is an expansion of Table in the main paper.
Ti
Metric
Method
plane
hench
cbot.
chair
disp-
lamp
spkr.
rifle
sofa
table
phone
boat
avg.
SRN
26.62
22.20
23.42
24.40
21.85
19.07
17
21.04
24.95
23.65
22.45
20.87
25.86
23.28
PixcINeRF
29.76
26.35
27.72
27.58
23.84
24.22
28.58
24.44
30.60
26.94
25.59
27.13
29.18
26.80
PE-NVS
30.15
27.01
28.77
27.74
24.13
24.13
24.85
30.23
27.37
26.18
27.25
28.91
27.08
PSNR1
VisionNeRF
SRT
31.47
32.34
28.45
30.40
28.21
24.69
25.77
24.58
28.56
25.61
26.09
30.09
28.89
28.11
27.42
29.21
28.28
29.18
27.87
NeRFormer
30.50
27.19
28.88
28.12
24.49
25.21
29.34
25.22
31.13
27.65
26.67
27.93
30.12
27.58
NoRForoo+MRVM
32.10
28.91
30.94
29.16
26.20
27.27
31.54 54
27.24
32.18
29.25
28.88 82
29.70
31,13
29.25
SRN
0.901
0.837
0.831
0.897
0.814
0,744
0.801
0.779
0.913
0.851
0.828
0.811
0.898
0.849
PiseINeRF
0.947
0,911
0.910
0.942
0.858
0.867
0.913
0.855
0.968
0.908
0.898
0.922
0.939
0.910
FE-NVS
0.957
SSIMI
SRT
0.954
0.925
0.920
0.937
0.861
0.855
0.904
0.854
0.962
0.911
0.909
0.918
0.930
0.912
VisionNeRF
0.965
0,944
0.977
0.958
0.892
0.891
0.925
0.877
0.974
0.930
0.929
0.936
0,950
0,933
0.916
0.913
0.928
NeRForeerMMRV
0.966
0.945
0.94]
0.958
0.906
0.912
0.948
0.900
0.978
0.937
0.942
0.944
0.959
0.942
SRN
0.111
0,150
0.147
0.115
0.152
0.197
0,210
0.178
0.111
0.129
0.135
0.165
0.134
0.139
PixeINeRI
0.084
0.116
0.105
0,095
0.146
0.129
0.114
0.141
0.066
0.116
0,098
0.097
0.111
0.108
FE-NVS
0.061
0.080
0.076
0.085
0.103
0,108
0.091
0.116
0.048
0.081
0.071
0.080
0.094
0.082
LPIPS
0.066
VisionNeRF
0.042
0.067
0.065
0.059
0.084
0.086
0.073
0.103
0.046
0.068
0.055
0.068
0.072
0.065
NeRFormer
0.063
0.096
0.088
0.081
0.128
0,116
0.093
0.126
0.055
0.099
0.079
0.083
0.090
0.091
NeRFormes4MRVM
0.045
0.067
0.064
0.059
0.887
0.083
0.068
0.098
0.042
0.070
0.051
0.063
0.070
B MORE IMPLEMENTATION DETAILS
We first provide general configurations that are applicable across all settings, followed by configu- rations specific to each unique setting.
General configurations For mask-based pretraining, we incorporate Lyrrom as an auxiliary loss. It is optimized together with NeRF's rendering loss not from the beginning, but starting from 10% of the total training iterations until finishing. We also use a warm-up schedule for about 10k iterations which linearly increases the coefficient À from O to the final value 0.1. Both of these technical strategies contribute to stabilize the pretraining process. At inference time, we use the VGG network for calculating LPIPS (Zhang et al. 2018) after normalizing pixel values to [-1.1]. We perform ray casting, sampling and volume rendering all in the world coordinate, All the models are implemented using Pytorch (Paszke et al. 2019) framework.
14

---

Published as a conference paper at ICLR 2024
Table 7: Detailed results of calegory-agnostic ShapeNet-unseen setting, with breakdown by
categories. This table is an expansion of Table I in the main paper.
Metric
Method
bench
cbnt.
disp.
lamp
spkr.
riflc
sofa
table
phone
boat
avg.
SRN
18.71
17.04
15.06
19.26
17.06
23.12
18.76
17.35
15.66
24.97
18.71
PixeINeRF
23.79
22.85
18.09
22.76
21.22
23.68
24.62
21.65
21.05
26.55
22.71
PSNR
FE-NVS
23.10
22.27
17.01
22.15
20.76
23.22
24.20
20.54
19.59
25.77
21.90
NeRFormer
23.64
22.21
17.77
23.20
20.60
24.11
24.58
21.05
21.24
27.32
22.54
NcRFormer+MMRAM
25,46
23.28
18.72
24.79
21.93
25.19
26.63
22.61
21.78
28.54
24.08
SRN
0,702
0.626
0.577
0,685
0.633
0.875
0,702
0.617
0.635
0.875
0,684
PeR
0.863
0,814
0.687
0.818
0.778
0.899
0.866
0.798
0.801
0.896
0.825
SSIM
FE-NVS
0.865
0.819
0.686
0.822
0.785
0.902
0.872
0.792
0.796
0.898
0.825
NeRFormer
0.863
0,808
0.689
0.837
0.774
0,908
0.875
0.786
0.817
0.914
0.826
McRForocr-MRVM
0.892
0.815
0.693
0.857
0.786
0.921
0.899
0.822
0.827
0.927
0.849
SRN
0,282
0.314
0.333
0.321
0.289
0.175
0.248
0.315
0.324
0.163
0.280
PixcINeRF
0.164
0.186
0.271
0.208
0.203
0.141
0.157
0.188
0.207
0.148
0.182
LPIPS
FE-NVS
0.135
0.156
0.237
0.175
0.173
0.117
0.123
0.152
0.176
0.128
0.150
NeRFormer
0.141
0.175
0.243
0.181
0.185
0.109
0.127
0.177
0.182
0.101
0.159
NeRFormer+MRVM
0.096
0.135
0.220
0.135
0.148
0.082
0.088
0.115
0.146
0.089
0.117
Table 8: Experimental results of adding our proposed masked ray and view modeling on the baseline of GNT (Wang al. 2022b and compare with GNT-MOVE (Cong et al. 2023) NeRF Synthetic and LLFF datasets.
Synthetic Ohject NeRF
Real Forward-facing LLFF
Method
PSNR1
SSIM
LPIPS
PSNR1
SSIM
LPIPS
GNT
27.29
0.937
0.056
25.86
0.867
0.116
GNT-MOVE
27.47
0,940
0.056
26.02
0.869
0.108
GNT+MRVM
27.78
0.942
0.052
26.25
0,873
0.110
B. IMPLEMENTATION DETAILS FOR SYNTHETIC DATASETS
Considering the images of synthetic datasets have a blank background, we adopt two techniques following previous works (Yu et al.) 2021b; Lin et al. 2022) for better performance. 1) We use bounding box sampling strategy as Yu et al. (2021b) during pretraining, where rays are only sampled within the bounding box of the foreground object. In this way, it avoids the model learn much empty information at initial training stage. 2) We assign a white background color for those pixels sampled from the background to match the rendering ground truths in ShapeNet dataset.
Settings For category ShapeNet-all setting, we use batch size of 16, and sample 256 rays per object. We pretrain the model for 400k iterations on 4 GPUs, with a tight bounding box for the first 300k iterations, then we finetune the model without bounding box for 800k iterations. The two-stage training takes about 10 days on GTX-1080Ti.
For category-agnostic ShapeNet-unseer setting, we also use batch size of 16, and sample 256 rays object. We pretrain for 300k iterations with bounding box on 4 GPUs, and finetune the model for 600k iterations without bounding box, which takes about 8 days on GTX-1080Ti.
For aategory-specifie ShapeNet-car and ShapeNet-chair settings, we use batch size of 8, and sample 512 rays per object. We pretrain for 400k iterations on GPUs. For the first 300k iterations, use input views for the network to encode with tight bounding box. For the rest of 100k iterations, the bounding box is removed and we randomly choose or 2 view(s) as the input to make the model compatible with both one-shot and two-shot scenarios. We finetune the model for or view(s) respectively on GPUs for 400k iterations. The two-stage training takes about days on GTX-108015
15

---

Published as a conference paper at ICLR 2024
Table 9: The few-shot experimental results of adding our proposed masked ray and view model-
ing on the baseline of GNT Wang et al. 2022b) and compare with GNT-MOVE (Cong et al. 2023) on NeRF Synthetic and LLFF datasets.
Synthetic Object NeRF
Real Forward-facing LLFF
Method
6-shot
12-shot
3-shot
6-shot
PSNRT
SSIMT
LPIPS
PSNRT
SSIMT
LPIPS]
PSNKT
SSIMT
LPIPS
PSNRT
SSIMT
LPIPS].
GNT
0.856
0.139
25.25
19.58
0.653
0.279
22.36
0.766
0.189
UNT-MOVE
22.53
0.871
0.116
25.85
0.915
0.074
19.71
0.666
0.270
22,53
0.774
0.184
GNT+MRYM
23.52
0.869
0.120
26.10
0.911
0.079
20.88
0.672
0.257
23.54
0.777
0.175
neadering
Brarch
Ared
Decoder
Fine Frarch
reference SIeR
target view
Figure 7: Illustration for mask-based pretraining variant and try recover them pretraining.
RGB mask. We mask blocks of pixels
B.2 IMPLEMENTATION DETAILS FOR REALISTIC DATASETS
Following the training protocol in NeuRay (iu al. 2022, first perform cross-scene pretraining across five distinct datasets (Downs et al. 2022] Mildenhall et al. 2019; Flynn et al.) 2019 Zhou al. 2018 Jensen al., 2014) for 400k iterations. Afterwards, for cross-scene generalization setting, we finetune the model on the same five training sets for additional 200k iterations. For per-scene finetuning setting, the model is finetuned on each scene respectively the three testing datasets (Niemeyer et al.) 2020 Jensen et al. 2014 Mildenhall et al. 2019) for additional 100k iterations, except for the few-shot scenarios Table 5f the main paper where we find only 10k iterations is sufficient for finetuning. When training the generalizable model across multiple datasets, we randomly sample scene from the training sets per iteration. We sample 512 rays for each scene during training. All the training processes are conducted one V100 GPU, which takes about days for total pretraining and finetuning.
Branch
Lead
6
Falums roudening
rCf
reference vtew
Branch
beod
torget view
(eatar of Krneeernce Nen
Factury referoos
2000
Figure 8: Illustration for mask-based pretraining variant - Feat m We use the intermediate representation output (boxes in blue) by Fine-Branch to reconstruct the masked feature tokens.
16

---

Published as a conference paper at ICLR 2024
trod
Coane
rendering
Branch
Fyert
roference slow
turget
moorage
of inpermaeeeee
Fine Brarch
xmp7
Figure 9: Illustration for mask-based pretraining variant 3 - Feat m We make copy of Fine-Branch as the target branch, in place of Coarse-Branch in the main paper.
B.3 VARIANTS OF MASK-BASED PRETRAINING OBJECTIVES
As stated in the main paper, we conduct an elaborated ablation study on different mask-based pre- training strategies, which are illustrated in Figure Figure S and Figure
As shown Figure 7 we mask blocks of pixels on input images from reference views. After extracting pyramid features with 2D CNN, we additionally introduce an UNet-like decoder to recover the masked image pixels based on these features. Lmrom is the C2 distance between reconstructed pixels and the ground truth, the constraint is only added to masked regions. We set mask ratio to 50% and patch size to 4 at pretraining.
RGB mask:
Feat mask'; As illustrated in Figure g we perform masking operation on sampled points same as MRVM. Differently, after obtaining intermediate representation z from the fine branch, we use it to recover the masked latent feature h? by shallow 2-layer MI.P 4 is the L2 distance between the reconstructed latent feature vector and the unmasked ground truth. We normalize the vector to unit-length before calculating the distance.
Feat m The pipeline for this variant is presented in Figure 9 Different from the architecture in the main paper, don't utilize coarse branch the target. On the contrary, make copy of the fine branch as the target network. With the gradient stopped manually, this branch is updated by moving average of the parameters from the online fine branch. We experimentally find that this option may cause instability at based pretraining stage, making it inappropriate as our final proposal.
17

---

Published as a conference paper at ICLR 2024
PhotNatE
Relerence
NelFoormr
NolTarmer MRVM
G..
Plane
Bench
Cabinet
Car
Chair
Display
Figure 10: More visualizations for Category-agnoste ShapeNet-all setting, Part 1.
18

---

Published as a conference paper at ICLR 2024
Reference
FixctNeky
NeRFoornsrr
NoltFommr MRYM
G.
Reference
PixeNNoKF
NeKFarmer
NotEFaree -MYM
G.F.
Lamp
Speaker
Rifle
Sofa
Table
Phone
Boat
Figure 11: More visualizations for Category-agnosse ShapeNet-all setting, Part 2.
19

---

Published as a conference paper at ICLR 2024
PhotNoFF
NeltFormer
Bench
Cabinet
Display
Lamp
Speaker
Figure 12: More visualizations for Category-agnosti ShapeNet-unseen setting, Part 1.
20

---

Published as a conference paper at ICLR 2024
Reference
PioeNeRE
NeRUFormer
NetFarmmrr MIYYM
G.
Refevanee
PIDENeR
NoRFormer
NoREsmmrr MRYY
GT.
Rifle
Sofa
Table
Phone
Boat
Figure 13: More visualizations for Category-agnostic ShapeNet-unseen setting, Part 2.
21

---

Published as a conference paper at ICLR 2024
NeRFormer MRRV
Reference(s)
PixeINeRF
NeRFormer
G.T.
Figure 14: More visualizations for Category-specific ShapeNet-car setting.
22

---

Published as a conference paper at ICLR 2024
NeRFormer MRVM
Reference(s)
PixeINeRF
NeRFormer
G.T.
Figure 15: More visualizations for Category-speciffe ShapeNet-chair setting.
23


Reviews:
[{"review_id": "F31GuddPRk", "reviewer": "Reviewer_s7zj", "summary": "The paper under review introduces a self-supervised pretraining strategy termed Masked Ray and View Modeling for Generalizable Neural Radiance Fields (MRVM-NeRF). The proposed method uses a dual-branch network: an unmasked coarse branch serving as the target and a masked fine branch acting as the online network. By training the masked fine branch to predict the unmasked coarse branch, the authors show that the model can learn better representations and generalize better to unseen scenes. The author claims that this work is the first to adapt the concept of masked self-supervised learning to the domain of generalizable NeRF.", "questions": "1. Can the authors provide more details on the computational efficiency of MRVM-NeRF?\n\n2. To predict coarse latent features, we take the masked latent feature as input to the prediction network (Equation 4). Can the $Pred^f$ function take additional inputs besides the latent feature? E.g., geometrical", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This method is able to achieve better performance than the baseline methods on various datasets and evaluation settings (category agnostic/specific, generalization/fine-tuning, few-shot, etc.).\n\n2. This method does not require additional supervision data or priors.\n\n3. This method can be applied to various types of generalizable NeRFs (e.g., NeRFormer and NeuRay) as a plug-in module.", "weaknesses": "1. As this method adds an additional branch to the baseline network, it is not as computationally efficient as the baseline methods. The computational overhead is not well discussed in the paper.\n\n2. Baseline generalizable NeRFs are already doing \"pretraining\".  I think this paper's contribution is to **improve** the pretraining by adding a self-supervised masked prediction task to the baseline methods. The author may need to clarify this point in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ap4PMGnE3t", "reviewer": "Reviewer_P3FF", "summary": "This paper proposes a novel masked ray and view modeling for generalizable NeRF. With mask-based pretraining, the model can learn 3D scene prior knowledge which is useful for reconstructing a high-quality new scene from limited reference views. Experiments show that the MRVM-NeRF achieves state-of-the-art novel view synthesis with limited views and cross-scene generalization.", "questions": "Can this method only handle object-centric and facing-forward scenes? In addition, can the authors provide results on the tank-and-temple datasets, where the scenes have a wider range of views and complex backgrounds.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea is novel and interesting. The authors introduce mask-based pretraining into NeRF, which provides 3D scene prior knowledge for  NeRF generalization.\n2. The experimental results are impressive. The MRVM-NeRF achieves superior quantitative and qualitative results compared with other methods.\n3. This paper is well-written and easy to understand.", "weaknesses": "1. Why do the authors use masked-based modeling to learn high-level global information? What are the advantages and necessity of using masked-based modeling?\n2. The authors should add the masked modeling design to more generalizable NeRF models to demonstrate its wide applicability. In addition, the authors should provide qualitative and quantitative comparisons with SOTA generalizable NeRF models, such as FreeNeRF[1] and SparseNeRF[2].\n\n[1] Yang et al. FreeNeRF: Improving Few-shot Neural Rendering with Free Frequency Regularization. In CVPR, 2023.\n[2] Wang et al. SparseNeRF: Distilling Depth Ranking for Few-shot Novel View Synthesis. In ICCV, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4YbBwa36Xy", "reviewer": "Reviewer_Jc3W", "summary": "The paper presents several significant improvements to the standard generalizable-NeRF framework, in which a NeRF is trained on a set of scenes and is used at inference on a novel scene without training. It successfully incorporates some recent advances from self-supervised representation learning, which include the use of masked prediction, exponential moving average learning, using transformer-based backbones. The main contribution is in the way these techniques are introduced into the NeRF framework, with the particular choice of the student network acting in the 'fine' sampling branch with random masking and the teacher network acting on the 'coarse' branch. Results verify the method, with significant and consistent improvements in the different metrics across the board.", "questions": "0] Please relate to above 'weaknesses'\nIn addition:\n1] Have you tried comparing to per-scene NeRFs? \n2] Could you provide an ablation showing the contribution of each of the components? In particular - applying only one of EMA / Masking?\n5] How sensitive is the choice of \\lambda, the weight of the mrvm-loss?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1] The proposed method shows very good performance compared to the baselines, both significant and consistent, across all experimentation. This is apparent visually and quantitatively, though an extensive set of experiments.\n2] The idea of introducing masking into the training scheme is well motivated, as a means for improving generalization to new scenes. It extends the standard pipeline quite naturally and does not impact inference time complexity. The decision to mask at two levels, within the ray and across views is interesting.\n3] Incorporating student-teacher learning within NeRF is also quite a natural thing to do. It gives further regularization that is likely responsible for the improved generalization.\n4] Paper is well written and organized, especially the method and experiments that explain and demonstrate the advantages very clearly.", "weaknesses": "1] The two main additions - of EMA and masking are tested together. There is a lack in understanding how dependent they are. Firstly, if one could be used without the other. And if so, secondly, what are the individual contributions.\n2] Sampling is not well specified. The extra 'fine' samples - whether they are they taken, as in the classical NeRF, according to an initial estimation of density from the coarse sample. And in this respect - shouldn't the masking prediction be focused at the more important (close-to-surface) high density locations, rather than at the coarse sampling?\n3] Impact on training time is not clear. Even though training time and procedures are very clearly presented in the appendix, I am missing what is the addition in comparison to the baseline, and which components are responsible for it?\n4] Clarity [minor]: (i) I would suggest improving Figure 1 to include some of the notations (g, h, etc') ; (ii) What are the blue dashed arrows in Figure 2 supposed to represent?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IuBburItqm", "reviewer": "Reviewer_chDn", "summary": "This paper investigates a Mask-based pertaining strategy called masked ray and view modeling (MRVM) for a generalizable Neural Radiance Field. It is the first attempt to incorporate mask-based pretraining into the NeRF field. To fit the NeRF field, a simple yet efficient self-supervised pretraining objective is proposed. Abundant experiments demonstrate its benefits for different architectures and data categories. I wonder if the training code and models be released.", "questions": "The parameter updating strategy and discussion on the cost of the additional fine branch are necessary.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Mask-based self-supervised pretraining has been demonstrated to benefit wide NLP and CV tasks. This work firstly attempts to introduce mask-based pretraining into NeRF field.\n\n- The designed masking strategy and objectiveness are suitable for the NeRF field. Obvious and consistent improvements can be obtained in various settings. \n\n- The discussion and analysis of \"prior NeRFs lack an explicit inductive bias from other views and points\" and \"distinct scale learning of two branches” are good, which could benefit the architecture design.\n\n\n- Abundant experiments have been constructed in different settings to demonstrate the effectiveness of the proposed method.\nExperiments on different network architectures (i.e. MLP and transformer) demonstrate its generalization to different models. \nExperiments on cross-scene and per-scene fine-tuning settings indicate the benefits of the mask-based pertaining to a wide range of scenes, including complicated geometry, and realistic non-Lambertian materials.\nFurthermore, the setting of the few-shot scenario reveals its significant improvements on the few-views setting (10-3 views).", "weaknesses": "- If I understand correctly when applying to the proposed method, an additional fine branch would be added. I wonder if this will cause double parameters and inference costs. \n\n- As to the method part, the terminology is not easy to follow. What is the projector, predictor?\nBesides, the parameter updating procedure is not very clear.  For example, it is claimed that “the parameters of coarse-projector are updated by moving average from the fine-projector”, what is the updating procedure for other parts?\n\n- The explanation of different masking strategies in the ablation study of the main paper is not easy to understand. I see the details are well illustrated in the supplementary material. I recommend claiming these illustrations in the main paper as well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dSxocN38wl", "reviewer": "Reviewer_fmpu", "summary": "The authors propose MRVM-NeRF, a framework that leverages an additional masked training objective to improve rendering and generalization ability. The method uses multi-view input of a scene to perform novel view synthesis. The masking is executed along cast rays and across reference views, along with an online fine branch supervised by the coarse stage. Results indicate that the method achieves SOTA generalization performance on simple (ShapeNet) and complex (NeRF Synthetic, LLFF, and DTU) datasets. The masked-based pretraining also improves finetuning performance. The authors also conduct experiments on few-shot finetuning on the NeRF synthetic dataset and achieve better performance.", "questions": "- In the case of online training, where supervision comes from the coarse branch, I wanted to clarify, do you use the same randomly sampled input points to both coarse and fine branches, in addition to the (masked) importance sampled inputs used by the fine branch?\n- I wonder, do the results depend on the choice of masking? Have any other strategies than random masking been tried out (for example removing geometrically consistent regions across all the views)?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is easy to read and well-presented. \n- All the design choices are well-motivated. \n- Further quantitative results indicate performance improvements, across multiple datasets and settings (generalization, scene-specific tuning, and few-shot finetuning).", "weaknesses": "There are no major weaknesses, in my opinion, design choices are quite well motivated, and quantitative results do indicate improvements. I think it might be interesting to investigate the few-shot results in more detail (i.e. show that the pretraining objective now enables tasks that were previously not looked at by generalizable nerfs). \n- For example, since the model is trained on masked inputs - it must additionally be able to perform quite well on cross-scene generalization in a few-shot setting (w/o any finetuning). Even in the case of finetuning, [1] presents some results on few-shot finetuning with as little as 3-6 images on LLFF, and 6-12 on NeRF Synthetic. It might be worth comparing against the same.\n\n[1] Enhancing NeRF akin to Enhancing LLMs: Generalizable NeRF Transformer with Mixture-of-View-Experts", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FIIrUmM7ZU", "reviewer": "Reviewer_y3Pz", "summary": "In this paper, the authors introduce masked-based learning and propose a self-supervised pretraining for generalizable NeRF. Specifically, they randomly mask feature token $h_i^j$ along cast rays and across refrence views, then introduce a BYOL-like module to reconstruct missing information in $\\bar z$ space. In this way, the proposed method can utilize the correlations among point-to-point and across view-to-view and learn a 3D scene prior knowledge. In addition, extensive experiments are conducted to validate the effectiveness of the proposed method.", "questions": "- As shown in Weaknesses, there are some missing details about the architecture of the used backbones. It is better to provide these details in the paper.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ The proposed method is effective. It boosts the performance of generalizable NeRF on different datasets largely. Meanwhile, the proposed method is easy to implement due to its simplicity. \n+ This paper is written well and easy to understand, although there are some missing details in the main paper.", "weaknesses": "- In the experiments, NeuRay is adopted as the MLP-based network. But, to my knowledge, the MLP-based NeRF processes each sampled 3D point  independetly. It means that it is not possible to utilize the correlations among point-to-point and across view-to-view and learn a 3D scene prior knowledge. Are there some architecture modifications for NeuRay?\n- To validate the influence of different masking strategies, the authors conduct experiments with three masking variants. There is missing one variant: taking the same masking strategy as described in Section 3.2 but minimize the $\\mathcal L_2$ distance between $z_i^f$ and $z_i^c$. Note that normalizing the vector to unit-length before calculating the distance. This additional experiment can validate the effectiveness of the BYOL-like module.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hCtMahecAy", "reviewer": "Reviewer_GXVJ", "summary": "This paper aims to tackle the Generalizable NeRF task, which represents multiple scenes using a single NeRF model. The authors introduce masked ray and view modeling (MRVM) to demonstrate that 3D implicit representation learning can be improved by mask-based modeling. In practice, the masking mechanism in MRVM carries out a ray level to enhance the information interaction along each ray and also a view level to promote the message-passing across different reference views. The numerical experiments and visualizations show the efficiency of the proposed method on several NeRF image reconstruction datasets.", "questions": "In short, this paper gives a well-written method description for readers. Some minor concerns are that the experiments for deeper discussion and comparison, as shown in [weakness], could be conducted.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "•\tThe paper is easy to follow, with well-written paragraphs and good section/figure/table organization.\n\n•\tThe proposed masked ray and view modeling is sound. Experiments demonstrate the effectiveness and superiority of the proposed method.", "weaknesses": "* Concerning the different masking strategies, i.e., RGB mask and two feature masks, it is curious whether ray-level or view-level has the more considerable performance gain or whether each masking strategy prefers a different level of masking.\n\n* The MRVM merely carried out upon one MLP-based NeRF model, i.e., NeuRay (Liu et al., 2022), and one Transformer-based NeRF model, i.e., NeRFormer (Reizenstein et al., 2021). It would be great if more SOTA MLP/Transformer-based models could be integrated with MRVM for comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)