Is Attention All That NeRF Needs?
Authors: Mukund Varma T
Peihao Wang
Xuxi Chen
Tianlong Chen
Subhashini Venugopalan
Zhangyang Wang
Venue: ICLR 2023 poster (2023)
Keywords: Neural Radiance Field
Transformer
Neural Rendering

Abstract: We present Generalizable NeRF Transformer (GNT), a transformer-based architecture that reconstructs Neural Radiance Fields (NeRFs) and learns to render novel views on the fly from source views. While prior works on NeRFs optimize a scene representation by inverting a handcrafted rendering equation, GNT achieves neural representation and rendering that generalizes across scenes using transformers at two stages. (1) The view transformer leverages multi-view geometry as an inductive bias for attention-based scene representation, and predicts coordinate-aligned features by aggregating information from epipolar lines on the neighboring views. (2) The ray transformer renders novel views using attention to decode the features from the view transformer along the sampled points during ray marching. Our experiments demonstrate that when optimized on a single scene, GNT can successfully reconstruct NeRF without an explicit rendering formula due to the learned ray renderer. When trained on multiple scenes, GNT consistently achieves state-of-the-art performance when transferring to unseen scenes and outperform all other methods by ~10% on average. Our analysis of the learned attention maps to infer depth and occlusion indicate that attention enables learning a physically-grounded rendering. Our results show the promise of transformers as a universal modeling tool for graphics. Please refer to our project page for video results: https://vita-group.github.io/GNT/

Published as a conference paper at ICLR 2023
IS ATTENTION ALL THAT NERF NEEDS?
Mukund Varma T Peihao Wang"*, Xuxi Chen', Tianlong Chen', Subhashini Venugopalan' Zhangyang Wang
Indian Institute of Technology Madras, University of Texas at Austin, GGoogee Research mu mukundvarmat ggma com, vsubhashini@googlle com (pe peihhawang,, xxchen,,, tianlong chen, at laswwang Baapaaanngggttteeeee @utexas. edu
ABSTRACT
We present Generalizable NeRF Transformer (GNT), a transformer-based architec- ture that reconstructs Neural Radiance Fields (NeRFs) and learns to render novel views on the fly from souree views. While prior works on NeRFs optimize scene representation by inverting handcrafted rendering equation, GNT achieves neural representation and rendering that generalizes across scenes using transformers at two stages. (I) The view transformer leverages multi-view geometry inductive bias for attention- scene representation, and predicts coordinate- aligned features by aggregating information from epipolar lines the neighboring views. (2) The ray transformer renders novel views using attention to decode the features from the view transformer along the sampled points during ray marching. Our experiments demonstrate that when optimized on a single scene, GNT can successfully reconstruct NeRF without explicit rendering formula due the learned ray renderer. When trained on multiple scenes, GNT consistently achieves state-of-the- performance when transferring to unseen scenes and outperform all other methods by -10% on average. Our analysis of the learned attention maps to infer depth and occlusion indicate that attention enables learning physically- universal
grounded rendering. Our results show the promise of transformers modeling tool for graphics. Please refer to our project page for video results:
gropirrrouu..... github....tb........... 1tithuu..o//NT/ 1o/GNT/
INTRODUCTION
Neural Radiance Field (NeRF) (Mildenhall et al. 2020) and its follow-up works (Barron et al. 2021 Zhang et al. 2020 Chen et al. 2022) have achieved remarkable success on novel view synthesis, generating photo-realistic, high-resolution, and view-consistent scenes. Two key ingredients in NeRF are, (1) a coordinate- and density, and (2) differentiable volumetric rendering pipeline that composes the color and density of points along each ray cast from the image plane to generate the target pixel color. Optimizing a NeRF can be regarded an inverse imaging problem that fits neural network satisfy the observed views. Such training leads to a major limitation of NeRF, making it a time-consuming optimization
neural network that maps each spatial position to its corresponding color
process for each scene (Chen et al. 2021a Wang al. 2021b Yu al. 2021)
Recent works Neuray (Liu et al, 2022). IBRNet (Wang et al. 20216), and PixelNerf YYu al. 2021 go beyond the coordinate- basedd network and rethink novel view synthesis as cross-view image- based interpolation problem. Unlike the vanilla NeRF that tediously fits each scene, these methods synthesize a generalizable 3D representation by aggregating image features extracted from seen views according to the camera and geometry priors. However, despite showing large performance gains, they unexceptionally decode the feature volume to a radiance field, and rely on classical volume rendering (Max 1195;; Levoy 1988) to generate images. Note that the volume rendering equation adopted NeRF over-simplifies the optical modeling of solid surface (Yariv et al.] 202T]] Wang et al.. 2021a), reflectance (Chen et al. 2021cc Verbin et al.) 2021; Chen et al.) (022), inter-surface scattering and other effects. This implies that radiance fields along with volume rendering in NeRF are not a universal imaging model, which may have limited the generalization ability of NeRFs Fs as well.
"Equal contribution.

---

Published as a conference paper at ICLR 2023
In this paper, first consider the problem of transferable novel view synthesis two-stage information aggregation process: the multi-view image feature fusion, followed by the sampling- based rendering integration. Our key contributions come from using transformers (Vaswani et al. 2017) for both these stages. Transformers have had resounding success in language modeling (Devlin al. 2018) and computer vision (Dosovitskiy et al. 2020) and their "self-attention" mechanism can be thought of as a universal trainable aggregation function. In our case, for volumetric scene representation, we train a view transformer, aggregate pixel-aligned image features (Saito et al.) 2019) from corresponding epipolar lines to predict coordinate-wif features. For rendering a novel view, we develop a ray transformer that composes the coordinate-win wise point features along a traced ray via the attention mechanism. These two form the Generalizable NeRF Transformer (GNT).
GNT simultaneously learns to represent scenes from source view images and to perform scene- adaptive ray-based rendering using the learned attention mechanism. Remarkably, GNT predicts novel views using the captured images without fitting per scene. Our promising results endorse that transformers are strong, scalable, and versatile learning backbones for graphical rendering (Tewari ct al. 2020). Our key contributions are:
1. A view transformer to aggregate multi view image features complying with epipolar geome- try and to infer coordinate- aliigned features.
2. A ray transformer for a learned ray-based rendering to predict target color.
3. Experiments to demonstrate that GNT's fully transformer- based architecture achieves state- of -hethe-art results complex scenes and cross-scene generalization.
4. Analysis of the attention module showing that GNT learns to be depth and occlusion aware.
Overall, our combined Generalizable NeRF Transformer (GNT) demonstrates that many of the inductive biases that were thought necessary for view synthesis (e.g. persistent 3D model, hard-coded rendering equation) can be replaced with attention/traasformee mechanisms.
2 RELATED WORK
Transformers (Vaswani et al. 2017) have emerged as a ubiquitous learning backbone that captures
long-range correlation for sequential data. It has shown remarkable success in language understanding
(Devlin et al. 2018: Dai al. 2019 Brown et al. 2020), computer vision (Dosovitskiy et al.) [2020) Liu et al. 2021). speech (Gulati al. 2020) and even protein structure (Jumper et al. [222]) amongst others. In computer vision, (Dosovitskiy et al., 2020) were successful demonstrating Vision Transformers (VIT) for image classification. Subsequent works extended VIT to other vision tasks, including object detection (Carion et al. 2020), segmentation (Chen et al., 2021b] Wang et al. 2021c). video processing (Zhou al.] [201 Arnab et al. (22II) and 3D instance processing (Guo) et al. 2021 Lin et al. 2021). In this work, we apply transformers for view synthesis by learning to reconstruct neural radiance fields and render novel views.
Neural Radiance Fields (NeRF) introduced by Mildenhall et al. 2020) synthesizes consistent and photorealistic novel views by fitting each scene as a continuous 5D radiance field parameterized by an MLP. Since then, several works have improved NeRFs further. For example, Mip-NeRF Barron et al. (2021 2022) efficiently addresses scale of objects unbounded scenes, Nex wongga et al. 2021 models large view dependent effects, others (Oechsle et al.) [021 Yariy 2002 Wang
et al. 202 improve the surface representation, extend to dynamic scenes (Park el al. [20 la b Pumarola al. 202T). introduce lighting and reflection modeling (Chen al. [222 Ic] Verbin et al. 2021), or leverage depth to regress from few views (Xu et al., 2022; et al. 2022). Our work aims avoid per-scene training, similar to PixeINeRF (Yu et al. 2021). IBRNet (Wang al. 2021b).
MVSNeRF (Chen et al., 202 and NeuRay (Liu et (0022) which train cross-scene multi-view aggregator and reconstruct the radiance field with a one-shot forward pass.
Transformer Meets Radiance Fields. Most similar to our work are NeRF methods that apply transformers for novel view synthesis and generalize across scenes. IBRNet (Wang et al.) 202 lb) processes sampled points on the ray using an MLP to predict color values and density features which then input to transformer to predict density. Recently, NeRFormer (Reizenstein et al., 2021 and Wang et al. (2022) use attention module to aggregate source views to construct feature volume with epipolar geometry constraints. However, key difference with work is that, all of them decode
2

---

Published as a conference paper at ICLR 2023
Scene Representation
Ncural Rendering
Photometric Loss
Epipolar Line
Cast Ray
Ray Transformer
View Tranformer
Prod.
GT
Feature Encoder
Imtage
Views
Encoder
Image
Pos.
Target Views
Feature
Embeed
Figure I: Overview of Generalizable NeRF Transformer (GNT): 1) Identify source views for a given target view, 2) Extract features for epipolar points using trainable U-Net-like model, 3) For each ray in the target view, sample points and directly predict target pixel's color by aggregating view-wise features (View Transformer) and across points along a ray (Ray Transformer).
the latent feature representation to point-wise color and density, and rely classic volume rendering form the image, while our ray transformer learns render the target pixel directly.
Other works, that use transformers but differ significantly in methodology or application, include (Lin et af., 2022) which generates novel views from just a single image via vision transformer and SRT (Sajadii et al., 20226) which treats images and camera parameters in latent space and trains a transformer that directly maps camera pose embedding to the corresponding image without any physical constraints. An alternative route formulates view synthesis rendering a sparsely observed 4D light field, rather than following NeRF's 5D scene representation and volumetric rendering. The recently proposed NLF (Suhail et al., 2021) uses iitention-basse framework to display light fields with view consistency, where the first transformer summarizes information on epipolar lines independently and then fuses epipolar features using second transformer. This differs from GNT, where we aggregate across views, and are hence able to generalize across scenes which NLF fails do. Lately, GPNR (Suhail et al. 2022), which was developed concurrently with our work, generalizes NLF (Suhail et al. 2022) by also enabling cross-view communication through the attention mechanism.
METHOD: MAKE ATTENTION ALL THAT NERF NEEDS
Overview. Given a set N input views with known camera parameters {(I, E #HWW'3 P. R3x4)) goal is synthesize novel views from arbitrary angles and also generalize to scenes. Our method can be divided into two stages: (1) construct the 3D representation from source views the fly in the feature space, (2) re-render the feature field the specified angle synthesize novel views. Unlike PixeINeRF, IBRNet, MVSNeRF and Neuray that borrow classic volume rendering for view synthesis after the first multi-view aggregation stage, we propose transformers to model both stages. Our pipeline is depicted in Fig. First, the view transformer aggregates coordinate-aligned features from source views. To enforce multi-view geometry, we inject the inductive bias of epipolar constraints into the attention mechanism. After obtaining the feature representation of each point on the ray, the ray transformer composes point-wise features along ray to form the ray color. This pipeline constitutes GNT and it is trained end-to-end.
3.1 EPIPOLAR GEOMETRY CONSTRAINED SCENE REPRESENTATION
NeRF represents 3D scene radiance field J. (x, 0) + (c, o), where each spatial coordinate æ R together with the viewing direction [-T, x is mapped to a color c E R3 plus density R4 tuple. Vanilla NeRF parameterizes the radiance field using MLP, and recovers the scene in a backward optimization fashion, inherently limiting NeRF from generalizing to new scenes.
Generalizable NeRFs (Yu al. 2021
Wang et al. 2021b Chen et al. 2021a) construct the radiance

---

Published as a conference paper at ICLR 2023
NORM
X
Multi- Layer Percepirons (MLP)
-000
Addliio
-000
Sabrrcciion
Meailicatio
x
Feed-Forwar
Network (FFN)
View Transformer
(b) Ray Transformer
Figure 2: Detailed network architectures of view transformer and ray transformer in GNT, where X represents the epipolar features, Xo represents aggregated ray features, {z, d, Ad) indicates point coordinates, viewing direction, and relative directions of source views with respect to the target view.
field in feed-forward scheme, directly encoding multi-view images into 3D feature space, and decoding it to a color- color-density field.
In our work, we adopt the similar feed-forward fashion to convert multi-view images into 3D representation, but instead of using physical variables (e.g.. color and density), we model a 3D scene as a coordinate-alignen feature field F (ar, 0) - f E IRd where d is the dimension of the latent space. We formulate the feed-forward scene representation as follows:
F(x, 0) V(x, 0; (11, (NN),,
(1)
where V(-) is function invariant the permutation of input images to aggregate different views {I,.... .IN} .III into a voordinate-aligned feature field, and extracts features at a specific location. We transformers as aggregation function (Lee al. 2019). However, plugging in attention globally attend to every pixel in the source images (Sajjadi et al. 2022h a) is memory prohibitive and lacks multi-view geometric priors. Hence,
use epipolar geometry as inductive bias that restricts each pixel to only attend to pixels that lic on the corresponding epipolar lines of the neighboring views. Specifically, we first encode each view be feature map F, ImageEncoderr (I,) E I2 x
We expect the image encoder to extract not only shading information, but also material, semantics, and local/global complex light transport via its multi-scale architecture (Ronneberger et al., 2015). To obtain the feature representation at position æ, we first project æ to every source image, and interpolate the feature vector on the image plane. We then adopt a transformer (dubbed view transformer) to combine all the feature vectors. Formally, this process can be written as below:
F(x, 0)
(Il ()) FFN (IIx()),)
where (ran-Taanfformrr()) (-) is transformer encoder (see Appendix A (x)) projects x E R onto the i-th image plane by applying extrinsic matrix, and Fz 0) E Rd computes the feature vector position z E R via bilinear interpolation the feature grids. We use the transformer's positional encoding 7(-) to concatenate the extracted feature vector with point coordinate, viewing direction, and relative directions of source views with respect the target view (similar Wang et al. (2021b)). The detailed implementation of view transformer is depicted in Fig. 2 We defer our elaboration on its memory-efficien design to Appendix We argue that the view transformer can detect occlusion through the pixel values like a stereo-matching algorithm and selectively aggregate visible views (see details Appendix B
3.2 ATTENTION DRIVEN VOLUMETRIC RENDERING
Volume rendering (App. Eq. 7]. which simulates outgoing radiance from a volumetric field has been regarded as key knob of NeRF's success. NeRF renders the color of pixel by integrating the color and density along the ray cast from that pixel. Existing works, including NeRF (Sec. all use handcrafted and simplified versions of this integration. However, one can regard volume rendering as a weighted aggregation of all the piint-wise output, in which the weights are globally dependent on the other points for occlusion modeling. This aggregation can be learned by transformer such that point-wiss wise colors can be mapped to token features, and attention scores correspond to transmittance (the blending weights). This is how we model the ray transformer which is illustrated in Fig. 2b
4

---

Published as a conference paper at ICLR 2023
Table I:
Comparison of GNT against SOTA methods for single scene rendering. LLFF Dataset reports average scores on Orchids, Horns, Trex, Room, Leaves, Fern, Fortress.
Models
PSNR1
SSIM
LPIPS.
AvgL
Models
PSNRT
SSIM
LPIPS
Avgl
LLFF
24.88
0,911
0.114
0.051
LLFF
23.93
0.798
0.212
0.896
NeRF
31.01
0.947
0.081
0.025
NeRF
26.36
0.811
0.250
0.964
MipNeRF
33.09
0.961
0.043
0.016
NeX
27.03
0.890
0.182
0.049
NLF
33.85
0.981
0.024
0.011
NLF
28:03
0.917
0.129
0.038
GNT
33.71
0.975
0.025
0.011
GNT
27.97
0.902
0.078
0.034
(a) NeRF Synthetic Dataset
(h) Local Light Field Fusion (I.FFF) Dataset
To render the color of a ray r = (o, d). we can compute a feature representation f; = 0) E Rd for each point z; sampled on In addition to this, we also add position encoding of spatial location and view direction into fi. We obtain the rendered color by feeding the sequence of ..,,... ff into the ray transformer, perform mean pooling over all the predicted tokens, and map the pooled feature vector to RGB via an MLP:
C(r) MLP Mcan o Ray- Transformor(F(( t1d, 0)............... .F Plo+tadd 0)).
(3)
where .,.-.., I are uniformly sampled between near and far planes. Saay-Trasformer is stan- dard transformer encoder, and its pseudocode implementation is provided in Appendix [B] Rendering on feature space utilizes rich geometric, optical, and semantic information, which are intractable to be modeled explicitly. We argue that our ray transformer can automatically adjust the attention distribution control the sharpness of the reconstructed surface, and bake desirable lighting effects from the illumination and material features. Moreover, by exerting the expressiveness of the image encoder, the ray transformer can also overcome the limitation of ray casting and epipolar geometry to simulate complex light transport (e.g., refraction, reflection, etc.). Interestingly, despite all in latent space, we can also infer some explicit physical properties (such as depth) from ray transformer. See Appendix for depth cueing. We also involve discussion on the extension to auto-regressive rendering and attention-baa based coarse-to-fine sampling in Appendix C
EXPERIMENTS
We conduct experiments to compare GNT against state-of-the-art methods for novel view synthesis. Our experiment settings include both per-scene optimization and cross-scene generalization.
4.1 IMPLEMENTATION DETAILS
Source and Target View Sampling. Following IBRNet, we construct pairs of source and target views for training by first selecting a target view, and then identifying pool of k x N nearby views, from which N views are randomly sampled as source views. This sampling strategy simulates various view densities during training and therefore helps the network generalize better. During training, the values for is and N are uniformly sampled at random from (1,3) and (8, 12) respectively.
Training/ / Inference Details. We train both the feature extraction network and GNT end-to-end on datasets of multi-view posed images using the Adam optimizer to minimize the mean-squared error between predicted and ground truth RGB pixel values. The base learning rates for the feature extraction network and GNT are 10-3 and 5 x 10-4 respectively, which decay exponentially over training steps. For all our experiments, we train for 250,000 steps with 4096 rays sampled in each iteration. Unlike most NeRF methods, do not use separate coarse, fine networks and therefore to bring GNT to a comparable experimental setup, we sample 192 coarse points per ray across all experiments (unless otherwise specified).
Metrics. We use three widely adopted metrics: Peak Signal-to-Noit Ratio (PSNR), Structural Sim- ilarity Index Measure (SSIM) (Wang et al. 20044), and the Learned Perceptual Image Patch Similarity (LPIPS) (Zhang et al. 2018). We report the averages of each metric over different views in each scene (for single-scene experiments) and across multiple scenes in each dataset (for generalization experiments). We additionally report the geometric mean of 10 PSNR/10 /I SSIM and LPIPS,
which provides summary of the three metrics for easier comparison (Barron et al. 2021
5

---

Published as a conference paper at ICLR 2023
(a) NeRF
(b) NLF
(c) GNT
(d) Ground Truth
(a) NeRF
(b) MipNeRF
(c) GNT
(d) Ground Truth
Figure 3:
Qualitative results for single-scene rendering. In the Orchids scene from LLFF (first row), GNT recovers the shape of the leaves more accurately, In the Drums scene from Blender (second row), GNT's learnable renderer is able to model physical phenomena like specular reflections.
4.2 SINGLE SCENE RESULTS
Datasets. To evaluate the single scene view generation capacity of GNT, we perform experiments on datasets containing synthetic rendering of objects and real images of complex scenes. In these experiments,
the same resolution and train/test splits as NeRF (Mildenhall et al. 2020). Local Light Field Fusion (LLFF) dataset: Introduced by Mildenhall et al. (2019), it consists of 8 forward facing captures of real-world scenes using a smartphone. We report average scores across OOchidds, Horns, Trex, Room, Leaves, Fern, Fortress) and the metrics are summarized in Tab. [b] NeRF Synthetic Dataset: The synthetic dataset introduced by (Mildenhall al. 2020) consists of 8, scenes of objects with complicated geometry and realistic material. Each scene consists of images rendered from viewpoints randomly sampled on a hemisphere around the object. Similar to the experiments on the LLFF dataset, report the average metrics across all eight scenes in Tab. la
Discussion. We compare our GNT with LLFF, NeRF, MipNeRF, NeX, and NLF, Compared to other methods, we utilize a smaller batch size (specifically GNT samples 4096 rays per batch while NLF samples much 16384 rays per batch) and only sample coarse points fed into the network in one single forward pass, unlike most methods that use a two-stage coarse- fanne--ine sampling strategy. These hyperparameters have strong correlation with the rendered image quality, leaving our method at disadvantage. Despite these differences, GNT still manages to outperform most methods and performs on par when compared SOTA NLF method on both LLFF and Synthetic datasets. We provide a scene-wise breakdown of results on both these datasets in Appendix D (Tab. 7 In complex scenes like Drums, Ship, and Leaves. GNT manages to outperform other methods more substantially by 2.49 dB, 0.82 dB and 0.10 dB respectively. This indicates the effectiveness of our attention-driven volumetric rendering to model complex conditions. Interestingly, even in the worse performing scenes by PSNR (e.g. T-Rex), GNT achieves best perceptual metric scores across all scenes in the LLFF dataset (i.e LPIPS -27% J). This could be because PSNR fails to measure structural distortions, blurring, has high sensitivity towards brightness, and hence does not effectively measure visual quality. Similar inferences are discussed in Lin et al. (2022) regarding discrepancies in PSNR scores and their correlation to rendered image quality. Fig. 3 provides qualitative comparisons on the Orchids, Drums scene respectively and we can clearly see that GNT recovers the edge details of objects (in the of Orchids) and models complex lighting effect like specular reflection (in the case of Drums) more accurately.
4.3 GENERALIZATION TO UNSEEN SCENES
Datasets. GNT leverages multi-vie view features complying with epipolar geometry, enabling general- ization to unseen scenes. We follow the experimental protocol in IBRNetto evaluate the cross-scene generalization of GNT and use the following datasets for training and evaluation, respectively.

---

Published as a conference paper at ICLR 2023
Table 2: Comparison of GNT against SOTA methods for cross-scene generalization.
Models
PSNRT
Locul Light Field Fusio (LLFF)
SSEM
LPIPS]
Avg.
PSNRT
NeRF Synthetic
SSIM?
LPIPS,
Avei
Setting
Mindcl
PSNER
Shiny-
SSIM1
Thrace
1MPS
Argi
PisNNERE
18.66
0.585
0.463
0.159
22.65
0.508
0.302
0.075
Per-Scanz
Sck
26.65
0.990
0.165
0.049
MVSN:RF
21.18
0.691
0.301
0.108
25.15
0.853
0.199
0.057
Tramy
26.30
0.853
0.122
06447
25.17
0.813
0.280
0.064
26.75
0.908
0.101
0,900
NEF
27.30
0.937
0.045
0/029
Nenkay
0.818
18.29
0.927
01089
0:001
I8RNct
23.00
0.180
GPNK
25.72
0.175
26.45
0.94
0.006
Canerdization
GPNK
24.12
0.950
0.170
0883
25:86
27.29
CNT
0108
0010
(a) Blender and LLFF Datasets
(b) Shiny Dataset
(a) Training Datasets consists of both real, and synthetic data. For synthetic data, we use object renderings of 1023 models from Google Scanned Object (Downs et al., 2022). For real data, we make use of RealEstate 10K (Zhou et al.) 2018b), 100 scenes from the Spaces dataset (Flynn et al. 2019),
and 102 real scenes from handheld cellphone captures (Mildenhall et al.) 2019 Wang et al. 2021b)
(b) Evaluation Datasets include the previously discussed Synthetic (Mildenhall et al. 2020), LLFF datasets (Mildenhall et al. 2019) and Shiny-9 dataset (Wizadwongsa et al.] (021) with complex optics. Please note that the LLFF scenes present in the validation set not included in the handheld cellphone captures in the training set.
Discussion. We compare our method with PixeINeRF YYu ct al. 202I), MVSNeRF (Chen et al. 202Ia), IBRNet (Wang et al. (021b), and NeuRay (Liu et al. (022)) As seen from Tab. 2a our method outperforms SOTA by 17% J. -9% 4 average scores in the LLFF, Synthetic datasets respectively. This indicates the effectiveness of our proposed view transformer to extract generalizable scene representations. Similar to the single scene experiments, we observe significantly better perceptual metric scores 3% 1 SSIM, 27% LPIPS in both datasets. We show qualitative results Fig. where GNT renders novel views with clearly better visual quality when compared to other methods. Specifically, as seen from the second row in Fig. 5 GNT is able to handle regions that are sparsely visible in the source views and generates images of comparable visual quality NeuRay even with no explicit supervision for occlusion. We also provide an additional comparison against SKT (Sajjadi et al. 2022b) in Appendix D (Tab. 5). where our GNT significantly generalizes better.
GNT can learn to adapt to refraction and reflection in scenes. Encouraged by the promise shown by GNT in modeling reflections in the Drums scene via pre-scene training, we further directly evaluate pre-trained GNT on the Shiny dataset (Wizadwongsa et al.) 2021). which contains several challenging view-dependent effects, such as the rainbow reflections on a CD, and the refraction through liquid bottles. Technically, the full formulation of volume rendering (radiative transfer equation, as used in modern volume path tracers) is capable of handling all these effects. However, standard NeRFs use simplified formulation which does not simulate all physical effects, and hence easily fail to capture these effects.
Tab. 2b presents the numerical results of GNT when generalizing to Shiny dataset. Notably, our GNT outperforms state-of-the-art GPNR (Suhail et al. 2022) by 3dB in PSNR and -40% in average metric. Compared with per-scene optimized NeRFs, GNT outperforms many of them and even approaches to the best performer NLF (Shail et al. 2021 without any extra training. This further supports our argument that cross-scene training can help learn a better renderer. Fig. 4 exxhibits rendering results on the two example scenes of Lab and CD. Compared to the baseline (NeX), GNT is able to reconstruct complex refractions through test tube, and the interference patterns on the disk with higher quality, indicating the strong flexibility and adaptivity of our learnable renderer. That "serendipity" is intriguing to us, since the presence of refraction and scattering means that any technique that only uses samples along a single ray will not be able to properly simulate the full light transport. We conjecture GNT's success in modeling those challenging physical scenes to be the full transformer
ALOFF
ALOOF
(a) NeX
(b) NLF
(c) GNT
(d) GT
(a) NeX
(b) NLF
(a) GNT
(b) GT
Figure 4: Qualitative results of GNT for generalizable rendering on the the complex Shiny dataset, that contains more refractions and reflection. A pre-trained GNT can naturally adapt to complex refractions through test tube, and the interference patterns on the disk with higher quality.
7

---

Published as a conference paper at ICLR 2023
architectures for not only rays but also views: the view encoder aggregates multi-ray information and can in turn tune the ray encoding itself.
(a) IBRNet
(b) NeuRay
(c) GNT
(d) Ground Truth
Figure 5: Qualitative results for the cross-scene rendering. the unseen Flowers (first row) and Fern (second row) scenes, GNT recovers the edges of petals and pillars more accurately than IBRNet and visually comparable to NeuRay.
4.4 ABLATION STUDIES
We conduct the following ablation studies on the Drums scene validate our architectural designs.
One-Stage Transformer: We convert the point-wise epipolar features into one single sequence and pass it through "one-stage transformer" network with standard dot-product self-attention layers, without considering our two-stage pipeline: view and ray aggregation.
Epipolar Agg. + View Agg.: Mov- ing to two-stage transformer, we train network that first aggregates features from the points along the epipolar lines followed by feature ag- gregation across epipolar lines on dif- ferent reference views (in contrast to GNT's first view aggregation then ray aggregation). This two-stage aggrega- tion resembles the strategies adopted in NLF (Suhail et al. (02T).
Table 3; Ablation study of several components in GNT on the Drums scene from the Blender dataset. The indent indicates the studied setting is added upon the upper-level ones.
Model
PSNRT
SSIMI
I.PISS]
Avg.
One Stage Transformet
21.80
0.862
0.152
0,072
Two Stage Transformr
Epipolar Agg. View
21,57
0.863
0.153
0,073
Agg. Ray Asg-
Produet-Aiterion View Transformer
0.089
SuboractionnAtteen Atteacion View Traasformer w/ Volumetric Rendering:
24.24
0.92
0.076
0.043
Learaed Volumetric Kendering (Churs)
29.41
0.931
0.085
0.029
Dot Product-Attente View Transformer: Next train network that uses the standard dot- product attention in the view transformer blocks, in contrast to our proposed memory-efficient subtraction-based attention (see Appendix B
w/ Volumetric Rendering: Last but not least, we train a network to predict per-point RGB and density values from the point feature output by the view aggregator, and compose them using the volumetric rendering equation, instead of our learned iitention-driven volumetric renderer.
We report the performance of the above investigation in Tab. 3 We verify that our design of two-stage transformer is superior to one-stage aggregation or an alternative two-stage pipeline (Suhail et al.) 202T) since our renderer strictly complies with the multi-view geometry. Compared with dot-product attention, dubtraction-based attention achieves slightly higher overall scores. This also indicates performance of GNT does not heavily rely on the choice of attention operation. What matters is bringing in the attention mechanism for cross-point interaction. For practical usage, also consider the memory efficiency in our view transformer. Our ray transformer outperforms the classic volumetric rendering, implying the advantage of adopting data-driven renderer.
8

---

Published as a conference paper at ICLR 2023
(a) Trex
(b) Horns
Figure 7: Visualization of ray attention where each color indicates the distance of each pixel relative to the viewing direction. GNT's ray transformer computes point-wise aggregation weights from which the depth can be inferred. Red indicates far while blue indicates near.
4.5 INTERPRETATION ON ATTENTION MAPS
The use of transformers core element in GNT enables interpretation by analyzing the attention weights. As discussed earlier, view transformer finds correspondence between the queried points, and neighboring views which enables it to pay attention to more "visible" views or be occlusion-aware. Similarly, the ray transformer captures point-to-poing interactions which enable model the relative importance of each point be depth-aware. We validate our hypothesis by visualization.
50
10
440
40
View Attention. To visualize the view-wise
30
attention maps learned by our model, we use the attention matrix from Eq. and collapse the channel dimension by performing mean pooling. We then identify the view number which is as- signed maximum attention with respect to each point and then compute the most repeating view number across points along a ray (by computing mode). These "view importance" values denote the view which has maximum correspondence
Figure 6: Visualization of view attention where each color indicates the view number that has max- imum correspondence with a target pixel. GNT's view transformer learns map each object region in the target view to its corresponding regions in the source views which are least occluded.
with the target pixel's color. Fig. 6 visualizes the source view correspondence with every pixel in the target view. Given a region in the target view, GNT attempts to pay maximum attention to source view that least occluded in the same region. For example: In Fig. 6 the truck's bucket is most visible from view number 8, hence the regions corresponding to the same are orange colored, while regions towards the front of the lego are most visible from view number (yellow).
Ray Attention. To visualize the attention maps across points in ray, we use the attention matrix from Eq. and collapse the head dimension by performing mean pooling. From the derived matrix, we select point and extract its relative importance with every other point. We then compute a depth map from these learned point-wise correspondence values by multiplying with the marching distance of each point and sum- num-pooling along ray. Fig. 7 plots the depth maps computed from the learned attention values in the ray transformer block. We can clearly see that the obtained depth maps have a physical meaning i.e pixels closer to the view directions are blue while the ones farther away are red. Therefore, with no explicit supervision, GNT learns to physically ground its attention maps.
CONCLUSION
We present Generalizable NeRF Transformer (GNT), pure transformer-based architecture that efficiently reconstructs NeRFs on the fly. The view transformer of GNT leverages epipolar geometry as an inductive bias for scene representation. The ray transformer renders novel views by ray marching and decoding the sequences of sampled point features using the attention mechanism. Extensive experiments demonstrate that GNT improves both single-scene and cross-scene training results, and demonstrates "out of the box" promise for refraction and reflection scenes. We also show by visualization that depth and occlusion can be inferred from attention maps. This implies that pure attention can be "universal modeling tool" for the peesically-groundeee rendering process. Future directions include relaxing the epipolar constraints simulate more complicated light transport.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
We thank Pratul Srinivasan for his comments on a draft of this work.
REFERENCES
Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučic, and Cordelia Schmid. Vivit: A video vision transformer. In IEEE International Conference on Computer Vision (ICCV), 2021.
Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Partin-Bruaalla, and Pratul P Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In IEEE International Conference on Computer Vision (ICCV), 2021.
Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5470-5479, 2022.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. Advances in Neural Information Processing Systems (NeurIPS), 2020.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision (ECCV), 2020.
Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang. Fanbo Xiang, Jingyi Yu, and Hao Su. Mysnerf: Fast generalizable radiance field reconstruction from multi-view stereo. In IEEE International Conference Computer Vision (ICCV), 2021a.
Hanting Chen, Yunhe Wang. Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, and Wen Gao. Pre-trained image processing transformer. In IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2021
Hao Chen, Bo He, Hanyu Wang, Yixuan Ren, Ser Nam Lim, and Abhinav Shrivastava, Nerv: Neural representations for videos. Advances in Neural Information Processing Systems, 34, 2021c
Tianlong Chen, Peihao Wang, Zhiwen Fan, and Zhangyang Wang. Aug-nerf: Training stronger neural radiance fields with triple-level physically goysically-grounded augmentations. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Zihang Dai, Zhilin Yang, Yiming Yang. Jaime Carbonell. Quoc Le. and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond fixed-length context. In Annual Meeting of the Association for Computational Linguistics (ACL), 2019.
Kangle Deng, Andrew Liu, Yan Zhu, and Deva Ramanan. Depth-supervised nerf: Fewer views and faster training for free. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12882-12891, 2022.
Jacob Devlin, Ming-Wes Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805, 2018
Yikang Ding, Wentao Yuan, Qingtian Zhu, Haotian Zhang, Xiangyue Liu, Yuanjiang Wang, and
Xiao Liu. Transmysnet: Global context-aware multi-view stereo network with transformers. arXiv preprint arXiv:2111. 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference Learming Representations (ICLR), 2020.
Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas B McHugh, and Vincent Vanhoucke. Google scanned objects: A high-quality dataset of 3d scanned household items. arXiv e-prints, pp. arXiy-2204, 2022.
10

---

Published as a conference paper at ICLR 2023
Zhiwen Fan, Tianlong Chen, Peihao Wang, and Zhangyang Wang. Cadtransformer: Panoptic symbol spotting transformer for cad drawings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10986-10996, 2022.
John Flymn. Michael Broxton, Paul Debevec, Matthew DuVall, Graham Fyffe, Ryan Overbeck, Noah Snavely, and Richard Tucker. Deepview: View synthesis with learned gradient descent. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2367-2376, 2019.
Anmol Gulati, James Qin, Chung- Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al. Conformer: Convolution-augmentte transformer for speech recognition. arXiv preprint arXiv:2005.08/00 2020.
Meng-Hao Guo, Jun-Xiong Cai, Zheng-Ning Liu, Tai-Jiang Mu, Ralph R Martin, and Shi-Min Hu. Pet: Point cloud transformer. Computational Visual Media, 2021.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Židek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 2021.
Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. Neural 3d mesh renderer. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3907-3916, 2018.
Juho Lee, Yoonho Lee, Jungtack Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set trans- former: A framework for attention-ban based permutation-invarian neural networks. In International conference on machine learning, pp.
PMLR, 2019.
Mare Levoy. Display of surfaces from volume data. IEEE Computer graphics and Applications, 8(3): 29-37, 1988.
Kai-En Lin, Lin Yen-Chen, Wei-Sheng Lai, Tsung- Vision transformer for nerf-based view synthesis from a single input image. arXiv preprint 2rXiv:2207.05736. 2022.
Yi Lin, Yi-Chang Shih, and Ravi Ramamoorthi.
Kevin Lin, Lijuan Wang, and Zicheng Liu. End-to-end human pose and mesh reconstruction with transformers. In JEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Yuan Liu, Sida Peng, Lingjie Liu, Qianqian Wang, Peng Wang, Christian Theobalt, Xiaowei Zhou, and Wenping Wang. Neural rays for occlusion-awer aware maage-based rendering. In CVPR, 2022.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10012-10022, 2021.
Nelson Max. Optical models for direct volume rendering. IEEE Transactions Visualization and Computer Graphics (TVCG), 1995.
Ben Mildenhall, Pratul P Srinivasan, Rodrigo Ortiz- Pttiz,Cayon, Nima Khademi Kalantari, Ravi Ra- mamoorthi, Ren Ng, and Abhishek Kar. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Transactions on Graphics (TOG), 38(4):1-14, 2019.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European conference on computer vision, pp. 405-421. Springer, 2020.
Michael Oechsle, Songyou Peng, and Andreas Geiger. Unisurf: Unifying neural implicit surfaces and radiance fields multi-view reconstruction. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5589-5599, 2021.
Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan Goldman, Steven M. Seitz, and Ricardo Martin- Brualla. Nerfies: Deformable neural radiance fields. ICCV, 2021a.
11

---

Published as a conference paper at ICLR 2023
Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Brrtin- and Steven M Seitz. Hypernerf: A digeeo-d representation for topologically varying neural radiance fields. In ACM Conference and Exhibition on Computer Graphics and Interactive Techniques in Asia (SIGGRAPH Asia), 2021b.
Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francese Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Putlern Recognition, pp. 10318-10327, 2021.
Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Shordone, Patrick Labatut, and David Novotny. Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction. In International Conference on Computer Vision, 2021.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234-241. Springer, 2015.
Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2304-2314, 2019.
Mehdi SM Sajjadi, Daniel Duckworth, Aravindh Mahendran, Sjoerd van Steenkiste, Filip Pavetić, Mario Lučić, Leonidas Guibas, Klaus Greff, and Thomas Kipf. Object scene representation transformer. arXiv preprint arXiv:2206.0692. 069222 2022a.
Mehdi SM Sajjadi, Henning Meyer, Etienne Pot, Urs Bergmann, Klaus Greff, Noha Radwan, Suhani Vora, Mario Lučié, Daniel Duckworth, Alexey Dosovitskiy, al. Scene representation transformer: Geometry- novel view synthesis through set-latent scene representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2229-6238, 2022b.
Vincent Sitzmann, Semon Rezchikov, Bill Freeman, Josh Tenenbaum, and Fredo Durand. Light field networks: Neural scene representations with single-evaluation rendering. Advances in Neural Information Processing Systems, 34, 2021.
Mohammed Suhail, Carlos Esteves, Leonid Sigal, and Ameesh Makadia. Light field neural rendering. arXiv preprint 12.09687, 2021.
Mohammed Suhail, Carlos Esteves, Leonid Sigal, and Ameesh Makadia. Generalizable patch-based neural rendering. arXiv preprint arXiv:2207. /0662, 2022.
Ayush Tewari, Ohad Fried, Justus Thies, Vincent Sitzmann, Stephen Lombardi, Kalyan Sunkavalli, Ricardo Martin-Brualla Tomas Simon, Jason Saragih, Matthias Nießner, et al. State of the art on neural rendering. In Computer Graphics Forum, volume 39, pp. 701-727. Wiley Online Library, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems (NeurIPS), 2017.
Dor Verbin, Peter Hedman, Ben Mildenhall, Todd Zickler, Jonathan T Barron, and Pratul P Srini- vasan. Ref-nerf: Structured view-dependent appearance for neural radiance fields. arXiv preprint 12.09921 2021.
Dan Wang, Xinrui Cui, Septimiu Salcudean, and Z Jane Wang. Generalizable neural radiance fields for novel view synthesis with transformer. arXiv preprint arXiv:2206.0537. 2022.
Peng Wang. Lingjie Liu. Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint arXiv:2106. 10689, 2021a.
Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul P Srinivasan, Howard Zhou, Jonathan Barron, Ricardo Martin- Brualla, Noah Snavely, and Thomas Funkhouser. Ibrnet Learning multi-view image-based rendering. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021b.
12

---

Published as a conference paper at ICLR 2023
Yuqing Wang, Zhaoliang Xu, Xinlong Wang, Chunhua Shen, Baoshan Cheng, Hao Shen, and Huaxia Xia. End-to-end video instance segmentation with transformers. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021c.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600-612, 2004.
Suttisak Wizadwongsa, Pakkapon Phongthawee, Jiraphon Yenphraphai, and Supasorn Suwajanakom. Nex: Real-time view synthesis with neural basis expansion. In IEEE Conference Computer Vision and Puttern Recognition (CVPR), 2021.
Dejia Xu, Yifan Jiang, Peihao Wang, Zhiwen Fan, Humphrey Shi, and Zhangyang Wang. Sin- nerf: Training neural radiance fields on complex scenes from single image. arXiv preprint arXiv:2204.00928 2022.
Zhenpei Yang, Zhile Ren, Qi Shan, and Qixing Huang. Mvs2d: Efficient multi-view stereo via attention-driver 2d convolutions. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8574 8584, 2022.
Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan. Mvsnet: Depth inference for unstructured multi-view stereo. In Proceedings of the European conference on computer vision (ECCV), pp- 767-783, 2018.
Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. Advances in Neural Information Processing Systems, 34, 2021.
Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one or few images. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun. Nerf++: Analyzing and improving neural radiance fields. arXiv preprint arXiv:2010.07492 2020.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018.
Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun. Point transformer. In Proceedings of the IEEÉ/CVF International Conference on Computer Vision, pp. 16259-16268, 2021.
Ellen D Zhong, Tristan Bepler, Bonnie Berger, and Joseph H Davis. Cryodrgn: reconstruction of heterogeneous cryo-em structures using neural networks. Nature Methods, 18(2):176-185, 2021.
Luowei Zhou, Yingbo Zhou, Jason Corso, Richard Socher, and Caiming Xiong. End-to-end dense video captioning with masked transformer. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2a.
Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. arXiv preprint 108 2018b.
13

---

Published as a conference paper at ICLR 2023
A PRELIMINARIES
Self-Attention and Transformer. Multi-Head Self-Attention (MHA) is the key ingredient of transformers (Vaswani et al. 2017). Data is first tokenized into sequences and a pairwise score is computed to weight the relation of each token with all the others in given input context. Formally, let X E RNxd represent some sequential data with N tokens of d-dimension. self-attention layer transforms the feature matrix as below:
Attn(X) = softmax(AA)f((X)) where A., = (x
E [M]
(4)
where A E NR is called the attention matrix, softmax(() operation normalizes the attention matrix row-wise, and ( represents a pair-wise relation function, most commonly the dot- a (X,,, x = fx().) (X;))// where fo()), fx((); are called query, key, and value mapping functions.
a standard transformer, they are chosen as fully-connected layers. This self-attention is akin to aggregation operation. Multi-Head Self-Attention (MHA) sets group of self-attention blocks, and adopts a linear layer to project them onto the output space:
MHA(X) = (Xt Attn2(X)
Attn (x) Wo
(5)
Following an MHA block, one standard layer of transformer also adopts Feed-Forward Network (FFN) to do pose feature transformation, as well as skip connection and layer normalization to stablize training. The whole transformer block can be formulated below:
X = MHA(LayyrNoom X)) + X, Y = =FFN(LayrrNomml x)) + X
(6)
Neural Radiance Field. NeRFs (Mildenhall
2020) converts multi-view images into radiance
field and interpolates novel views by re-rendering the radiance field from new angle. Technically, NeRF models the underlying 3D scene as continuous radiance field JF (xx 0) + (c. σ) parameter- ized by Multi-Layer Perceptron (MLP) θ, which maps spatial coordinate æ E R together with the viewing direction [-x, , TT² to a color e E R& plus density a E I,+ tuple. To form an image, NeRF performs the ray-based rendering. where it casts ray o E IR through each pixel (towards direction d E RR',, and then leverages volume rendering (Max 1995) to compose the color and density along the ray between the near-far planes:
(o, d) from the optical center
" C(rl0)= = I"' T(t)o((())(((((() d)dt, where T(t) exp (-fcotejds) fn.
where r(() = o + Id, in and I are the near and far planes respectively. practice, the Eq. is numerically estimated using quadrature rules (Mildenhall et al., (020).. Given images captured from surrounding views with known camera parameters, NeRF fits the radiance field by maximizing the likelihood of simulated results. Suppose collect all pains of rays and pixel colors the training set
D [(r,, C.) where N is the total number of rays sampled, and Ć, denotes the ground-truth color of the i-th ray, then we train the implicit representation O via the following loss function:
(()) = E (r ((rl))--
B IMPLEMENTATION DETAILS
Attention. Computing attention between every pair of inputs has O(N2) memory complexity, which is computational prohibitive when sampling thousands of points the same time. Nevertheless, we note that view transformer only needs to read out one token as the fused results of all the views. Therefore, we propose to only place one read-out token X, E Rd in the query sequence, and let it iteratively summarize features from other data points. This reduces the complexity for each layer up to O(N). We initialize the read-out token as the element- wise max- pooling of all the inputs: Xo- = max Fi (x) 0), ... Fv (II (z). ) 0)). Rather than adopting a standard dot-product attention, we choose subtraction operation as the relation function. Subtraction attention has been shown more effective for positional and geometric relationship reasoning (Zhao) let al.) 2202 I Fan et al.] (022). Compared with dot dot-product that collapses the feature dimension into scalar, subtraction attention computes different attention scores for every channel of the value matrix, which increases diversity in feature interactions. Moreover, we augment the attention map and value
Memory-Efiie Effficient Cross- Cross-View
14

---

Published as a conference paper at ICLR 2023
matrix with { [Ad] provide relative spatial context. Technically, we utilize a linear layer Wp to lift A to the hidden dimension. We illustrate view transformer in Fig. 2a To more specific, the modified attention adopted in view transformer can be formulated as:
View-Atln(X: X) diag A + A fv + +)), where A₁ - fo(Xo) [K(())
where A, E R denotes the j-th column of A, A = Add
Ady] Wp E RNxd, fq. fk- and Jv are parameterized by MLP. We note that by applying diag(.), read out the updated query token Xo. See Alg. for the implementation in practice.
Network Architecture. To extract features from the source views, we use U-Net-like architecture with a ResNet34 encoder, followed by two up-sampling layers as decoder. view transformer block contains a single-headed cross-attention layer while the ray transformer block contains multi-headed self-attention layer with four heads. The outputs from these attention layers are passed onto corresponding feedforward blocks with Rectified Linear Unit (RELU) activation and a hidden dimension of 256. A residual connection is applied between the pre-normalized inputs (LayerNorm) and outputs at each layer. For all our single-scene experiments, we alternatively stack view and ray transformer blocks while our larger generalization experiments use blocks each. All transformer blocks (view and ray) are of dimension 64. Following Vasswani et al. (2017); Mildenhall et al. (2020); Zhong al. (202T), we convert the low-dimensional coordinates high-dimensional representation using Fourier components, where the number of frequencies is selected as 10 for all our experiments. The derived view and position embeddings are each of dimension 63.
Algorithm 1 Cross View Attention: Tireh Pseudocode
X. coordinate aligned features (Nays Npus D)
X, + epipolar view (eatures(NNmey Npe. Nviews D)
Ad relative directions of source views target views (Nmys Nprss Nviews, JA. + functions that parameterize MLP layers
SQ
Q = (
K {x(x,)
V = fv(X,)
P = Ad)
A - Q i; NoNo,, :]
A softmax( A, dim -2)
0 ((V P). s).sumddim 2)
0 = (o
Pseudocode. We provide simple and efficient pytorch pseudo-code to implement the attention operations in the view, ray transformer blocks in Alg. I respectivel.. We do not indicate the feedforward and layer normalization operations for simplicity. As seen in Alg. 3 we reuse the epipolar view features X, to derive keys, and values across view transformer blocks. Therefore, one could further improve efficiency by computing them only once while also sharing the network weights across view transformer blocks or simply put fview (.) represents the same function across different values of i. This can be considered analogous to unfolded recurrent neural network that updates itself iteratively but using the same weights.
C TENTATIVE EXTENSIONS
C.I AUTO-REGRESSIVE DECODING
The final rendered color is obtained by mean-pooling the outputs from the ray transformer block, and mapping the pooled feature vector to RGB via an MLP layer. can be understood that the target pixel's color is strongly dependent on the closest point from the ray origin and weakly related to the
15

---

Published as a conference paper at ICLR 2023
Algorithm 2 Ray Attention: PyTorch-like Pseudocode
Xe coordinate aligned features( Nays NNpu D)
2 + point coordinates (after position encoding) (aooddng)(Nasss Naws: Npts. D)
d target view direction (after position encoding) Nayss Nus: D)
fo: J /x. Jv. Jv, JA: + functions that parameterize MLP layers
Xo = fr(concat(A) (oo:at(Xo::00 d)
Q fX(
K = (X))
V - (Xo)
A = matmml(( Q KTK )/VD
A softmax A, dim -1)
0 matmul(V A)
O = fo(O)
Algorithm 3 GNT: PyTorch-like Pseudocode
X epipolar view features( Nays; Npss Nviews; D)
Φ > point coordinates (after position encoding) Nays. Npus. D)
target view direction (after position encoding) (Na NVpts D)
Ad > relative directions of source views wrt target views(NMmss, Nppss Nviews, 3)
(1)
view> (()) functions that parameterize view transforms, ray transformers layer I respectively
Jigb + functions that parameterize MLP layers
1= 0
X. = max Novm (X,)
while < N Mayers do
Xo f view
(!))
(o,, X,, Ad)
Xo Jeay (X d. x)
1 1=10 1 +
end while
O Norm Xo)
RGB figh (mean
Nanu Nymu Xxo)
farthest. Revisiting Eq. 7 volumetric rendering attempts to compose point-wise color depending on the other points in far to near fashion. Motivated by this, propose an auto-regressive decoder to better simulate the rendering process. Transformers have shown great success in auto auto-regressive decoding, more specifically in NLP (Vaswani al., 2017). We borrow similar strategy and replace the simpler MLP-based color prediction with series of transformer blocks with self, cross attention layers.
In the first pass, the decoder is queried with positional encoding of the farthest point (>(#x)) )) to generate an output feature representation of the same. In the next step, the output token is concatenated with the second farthest point (y(=x-1)) to query the decoder. This process repeats until all the points in the ray are queried in far-to-near fashion. In the final pass, the encoded view direction (> (d)) is concatenated with the per-point output features in the previous passes to query the decoder and the output token corresponding to the view direction is extracted. The extracted token is mapped to RGB via MLP layer. This entire process is summarized in Fig. g The auto-regressive procedure closely resembles the volumetric rendering equation which iteratively blends and overrides the previous color when marching along ray from far to near.
Transformer-basee decoders used in language iteratively predict output tokens only during inference, ie. they trained in non-autoreereessvv fashion due to the availability of ground truth output tokens in each step. Transferring the same to neural rendering is not possible, as we do not have access to the groundtruth color for each point sampled along the ray. Hence, we require loop to
16

---

Published as a conference paper at ICLR 2023
Auto Regresegreesive Neural Rendering
0, t (d
Self Attention
Cross Attention
(OleN)} (0(en)))(0(xx-1))
-- (= (0(2))
(0(xn)
(O(e1)).
X
Pred.
Figure 8: Architecture of auto-regressive ray decoder with sampling strategy in a far to near fashion.
Table 4: Comparison of autoregressive GNT against SOTA methods for single rendering the
LLFF dataset.
Orchids
Horns
Models
PSNRT
SSIM
LPIPS
Avg.l.
PSNR
SSIM1
LPIPS
Avg.
LLFF
18.52
0,588
0.313
0.141
23.22
0.840
0.193
0.064
NeRF
20.36
0.641
0.321
0.121
27.45
0.828
0.268
0.058
NeX
20.42
0.765
0.242
0.102
28.46
0.934
0.173
0,040
NLF
21.05
0.807
0.173
0.084
29.78
0.957
0.121
0.030
GNT
20.67
0.752
0.153
0.087
29.62
0.935
0.076
0,028
GNT AutoReg
21.05
0,736
0.181
0.090
28.20
0.908
0.114
0.037
GNT Fine
20.69
0.752
0.153
0.087
29.59
0.934
0.076
0,028
auto-regressively decode features even during training. This reduces the computational efficiency of the proposed strategy, especially as the number of points sampled along the ray increases. Therefore, we introduce a caching mechanism to store the per-layer outputs of the previous tokens and only compute the attention of new token in the current pass. This does not overcome the iterative loop during each forward pass but avoids redundant computations, which helps improve the decoding speed drastically when compared the naive strategy. Due computational constraints, we are only able to train GNT + AutoReg with much fewer rays sampled per iteration (500) when compared to other methods as discussed Sec. 4.2 Tab. discusses single scene optimization results on the LLFF dataset and we can clearly see that the GNT + AutoReg improves the overall performance when compared to existing baselines, and improving the PSNR scores in complex scenes (Orchids) when compared to own method without the decoder. However, this is not consistent across all scenes and metrics. This could be because of the fewer number of rays sampled and we expect our results would improve when scaled to similar settings. Nevertheless, this shows that the learnable decoder predicts per-point RGB features effectively without any supervision from the volumetric rendering equation.
C.2 ATTENTION GUIDED COARSE-TO-FINE SAMPLING
GNT's ray transformer learns point-to- to-nt correspondences which helps model visibility and occlusion or more formally wise density a. Motivated by this hypothesis, we estimate the depth maps from the extracted attention maps and qualitatively analyze the same Sec. 4.5 Therefore, we could conclude that the learedd point-wise importance values can be considered equivalent to point- wise density or a. To further test our claim, we attempt to use the learned point-wise correspondence values sample "fine" points, which then queried to GNT to render higher-quality image. Due to the set-like property attention, we directly query the fine points to the same network without using separately trained "fine" network unlike other NeRF methods (Mildenhall et al. 2020 Barron
17

---

Published as a conference paper at ICLR 2023
(a) Coarse
(b) Fine
(c) Ground Truth
Figure 9: Visualization of ray attention extracted during coarse, fine sampling where each color indicates the distance of each pixel relative to the viewing direction. The sampled fine points inferred from the learn attention values help GNT capture more fine- geaggeai details. Red indicates far while blue indicates near.
al. 2021 Wang et al. 202lb; Liu al. 2022). Please note that we follow the same training strategy from Sec. 4.1 and only sample coarse-fine points during evaluation. Tab. compares "GNT + Fine" against other methods, and we can clearly see that outperforms other SOTA methods in complex scenes like Orchids, performing even better than own method without fine sampling. However, the performance improvements are not significant across all scenes and we attribute this to the lack of training with the coarse-fine sampling strategy and expect results to improve further. In Fig- 9 we visualize the estimated depth values obtained from the learned attention maps during both coarse, and fine stages. We clearly that the fine depth map is able to better estimate differences between nearby pixels which results in a higher resolution output.
D ADDITIONAL RESULTS AND ANALYSIS
Tables 6 and 7 linelude breakdown of the quantitative results presented in the main paper into per-scene metrics. Our method quantitatively surpasses original NeRF and achieve on-par results with state-of-the-ar methods. Although we slightly underperform NLF (Suhail et al. 2021) some scenes, argue that the comparison is not fair because NLF requires much larger batch size and longer iterations. We also include videos to demonstrate our results in the project page
Breakdown of Table 1
Comparison with SRT (Sajjadi et al. 2022b). SRT (Sajjadi et al. 2022b) is another pure trans- former based generalizable view synthesis baseline. In contrast GNT, SRT barely utilizes attention blocks to interpolate views without any explicit geometry priors. We directly evaluate our cross-scene trained GNT in Sec. [4.3 on NMR dataset (Kato al., (018)) without further tuning. In addition to
SRT, we also include other generalizable novel view synthesis methods LFN (Sitzmann al.) 2021) and PiXeINeRF (Yu al., 202T) which are compared with SRT in Sajjadi et al. (2022b). All the results presented in Tab. 5 Overall, we find GNT can largely outperform all baselines in all the metrics. We note that the pre- pre-training data of SRT include the samples from NMR dataset (Kato et al., 2018), which is way to the evaluation set. After all, our superior performance indicates our GNT can generalize better than SRT. We argue this might be because multi-view geometry is strong inductive bias for novel view interpolation. That being said, a pixel on the novel view should be roughly consistent with its epipolar correspondence. Enforcing such constraints explicitly can significantly improve trainability and data efficiency. Nevertheless, we admit relieving multi-vies view geometry and learning data-driven light transport prior from scratch can potentially render more sophisticated opties, which we leave for future exploration.
massive than GNT's pre-training datasets and has narrower domain gap
Comparison with GPNR (Suhail et al. 2022). The concurrent work GPNR (Suhail et al. 2022) also utilizes fully attention-based architecture for neural rendering. Below, we summarize several key differences: Embeddings: GPNR leverages three forms of positional encoding (including light field embeddings) to encode the information of location, camera pose, view direction, etc. In contrast, GNT merely utilizes image features (with point coordinates). In this sense, GNT enjoys a neat design space and potentially suggest such handcrafted feature engineering may not be necessary as
18

---

Published as a conference paper at ICLR 2023
Table 5: Comparison with LFN, PixeINeRF, and SRT on the NMR (Kato et al. 2018 dataset.
Models
PSNR1
SSIM
LPIPS
Avgl
LFN Sitzmann et al. 2021
24.95
0.870
PixeINeRF (Yu et al.
2021
26.80
0,910
0108
0,041
SRT (Sajjadi et al. 20226)
27.87
0.912
0.066
0.032
GNT
32.12
0.970
0.032
0.015
(a) MipNeRF
(b) NLF
(c) GNT
(d) Ground Truth
(a) NeX
(b) NLF
(c) GNT
(d) Ground Truth
Figure 10: Qualitative results for single-scene rendering. the Trex scene from LLFF (first row) and Materials scene from Blender (second row), GNT's learnable renderer is able to model physical phenomenon like reflections.
they can be learned through cross-scene training. Aggregation schemes: GPNR has three-stage aggregation: 1) visual feature transformer to exchange information between patches across views, 2) epipolar aggregator transformer combines features along the epipolar line for each reference view, 3) reference view aggregator transformer fuses the epipolar information across multiple views. Such aggregation scheme extends NLF (Suhail al. 2021) paper, which indicates GPNR rendering pipeline is more likely to simulate a light field-based rendering. Instead, GNT leverages the two-stage aggregations, which naturally correspond to the online scene representation and ray-based rendering in generalizable NeRF. In this sense, the rendering pipeline of GNT looks more like radiance field-base rendering. Performance: We have direct comparison with GPNR on cross-scene generalization experiments (Tab. 2). Our results show that GNT can outperform GPNR on all the tested datasets with simpler architecture and lighter feature engineering. Especially on the Shiny dataset (Tab. 2b) GNT significantly outperforms GPNR by <3dB in PSNR.
(a) GNT
(b) Ground Truth
(a) GNT
(b) Ground Truth
Figure 11: Qualitative comparison between images rendered by GNT and Ground truth image to
discuss limitations. Epipolar correspondence for boundary pixels can be missing sometimes, which causes minor stripe artifacts.
19

---

Published as a conference paper at ICLR 2023
Table 6: Comparison of GNT against SOTA methods for single scene rendering on the NeRF Synthetic
Dataset (scene-wise).
Models
Lego
Chair
Drums
Ficus
Hotdog
Materials
Mic
Ship
LLFF
24.54
28.72
21.13
21.79
31.41
20.72
27.48
23.22
NeRF
32.54
33.00
25.01
30.13
36.18
29.62
32.91
28.65
MipNeRF
35.70
35.14
25.48
33.29
37.48
30.71
36.51
30.41
NLF
35.76
35.30
25.83
33.38
38.66
35.10
35.32
30.94
GNT
34.59
34.60
28.32
32.71
38.43
32.73
35.66
31.76
(a) PSNR1
Models
Lego
Chair
Drums
Ficus
Hotdog
Materials
Mic
Ship
LLFF
0.911
0.948
0.890
0.896
0.965
0.890
0.964
0.823
NeRF
0.961
0.967
0.925
0.964
0.974
0.949
0.980
0.856
MipNeRF
0.978
0.981
0.932
0.980
0.982
0.959
0.991
0.882
NLF
0.989
0.989
0.955
0.987
0.993
0.990
0.992
0.952
GNT
0.984
0.986
0.966
0.986
0.989
0.984
0.993
0.906
(b) SSIM
Models
Lego
Chair
Drums
Ficus
Hotdog
Materials
Mic
Ship
LLFF
0.110
0.064
0.126
0.130
0.061
0.117
0.084
0.218
NeRF
0.050
0.046
0.091
0.044
0.121
0.063
0.028
0.206
MipNeRF
0.021
0.021
0.065
0.020
0.027
0.040
0.009
0.138
NLF
0.010
0.012
0.045
0.010
0.009
0.011
0.008
0.084
GNT
0.012
0.013
0.030
0.012
0.012
0.017
0.005
0.100
(c) LPIPS
Models
Lego
Chair
Drums
Ficus
Hotdog
Materials
Mic
Ship
LLFF
0,049
0,027
0.069
0.065
0.020
0.069
0.031
0.076
NeRF
0.018
0.016
0.043
0.020
0.017
0.025
0.013
0.047
MipNeRF
0.009
0.009
0.036
0.011
0.009
0.019
0.006
0.035
NLF
0.007
0.007
0.029
0.008
0.005
0.007
0.006
0.025
GNT
0.008
0.008
0.020
0.009
0.005
0.010
0.004
0.027
(d) Avgl.
20

---

Published as a conference paper at ICLR 2023
Table 7: Comparison of GNT against SOTA methods for single scene rendering on the LLFF Dataset (scene-wise).
Models
Room
Fern
Leaves
Fortress
Orchids
Flower
T-Rex
Horns
LLFF
24.54
28.72
21.13
21.79
18.52
20.72
27.48
23.22
NeRF
32.70
25.17
20.92
31.16
20.36
27.40
26.80
27.45
NeX
32.32
25.63
21.96
31.67
20.42
28.9
28.73
28.46
NLF
34.54
24.86
22.47
33.22
21.05
29.82
30.34
29.78
GNT
32.96
24.31
22.57
32.28
20.67
27.32
28.15
29.62
PSNRT
Models
Room
Fern
Leaves
Fortress
Orchids
Flower
T-Rex
Horns
LLFF
0.932
0.753
0.697
0.872
0.588
0.844
0.857
0.840
NeRF
0.948
0.792
0.690
0.881
0.641
0.827
0.880
0.828
NeX
0.975
0.887
0.832
0.952
0.765
0.933
0.953
0.934
NLF
0.987
0.886
0.856
0.964
0.807
0.939
0.968
0.957
GNT
0.963
0.846
0.852
0.934
0.752
0.893
0.936
0.935
(b) SSIMT
Models
Room
Fern
Leaves
Fortress
Orchids
Flower
T-Rex
Horns
LLFF
0.155
0.247
0.216
0.173
0.313
0.174
0.222
0.193
NeRF
0.178
0.280
0.316
0.171
0.321
0.219
0.249
0.268
NeX
0.161
0.205
0.173
0.131
0.242
0.150
0.192
0.173
NLF
0.104
0.135
0.110
0.119
0.173
0.107
0.143
0.121
GNT
0.060
0.116
0.109
0.061
0.153
0.092
0.080
0.076
(c) LPIPS}
Models
Room
Fern
Leaves
Fortress
Orchids
Flower
T-Rex
Horns
LLFF
0.039
0.086
0.110
0.041
0.141
0,058
0.069
0.064
NeRF
0.028
0.073
0.112
0.036
0.121
0.055
0.056
0.058
NeX
0.025
0.057
0.077
0.027
0.102
0.037
0.038
0.040
NLF
0.016
0.053
0.062
0.022
0.084
0.030
0.029
0.030
GNT
0.017
0.055
0.061
0.021
0.087
0.042
0.031
0.028
(d) Avgl
21

---

Published as a conference paper at ICLR 2023
E DEFERRED DISCUSSION
Discussion on Occlusion Awareness. Conceptually, view transformer attempts to find correspon- dence between the queried points and source views. The learned attention amounts likelihood score that a pixel on the source view is an image of the same point in the 3D space, i.e., no points lies between the target point and the pixel. NeuRay (Liu et al., 2022) leverages the cost volume from MVSNet (Yao et al. 2018) to predict per per-pixel visibility and shows that introducing occlusion information is beneficial for multi-view aggregation generalizable NeRF. We argue that instead of explicitly regressing the visibility, purely relying on epipolar geometry-constrained attention can automatically learn how to infer occlusion, given prior works in Multi- Multi-View Stereo (MVS) (Yang al. 2022 Ding et al., 2021). In view transformer, the U-Net provides multi-scale features to the transformer, and the attention block acts as a matching algorithm, which selects the pixels from neighboring views that maximize view consistency. We defer empirical discussion Sec. 4.5
Discussion on Depth Cuing. The ray transformer iteratively aggregates features according to the attention value. This attention value can be regarded the importance of each point to form the image, which reflects visibility and occlusion reasoned by point-to-point interaction. Therefore, we can interpret the average attention score for each point as the accumulated weights in volume rendering. In this sense, we can infer the depth map from the attention map by averaging the marching distance i; with the attention value. This implies our ray transformer learns geometry-aware 3D semantics both feature space and attention map, which helps it generalize well across scenes. We defer visualization and analysis to Sec. 4.5 NLF (Suhail et al. 2021) proposes a similar two-stage rendering transformer, but it first extracts features the epipolar lines and then aggregates epipolar features to get the pixel color. We doubt this strategy may fail to generalize as epipolar features lack communication with each other and thus cannot induce geometry- grouundd semantics.
F LIMITATIONS
Although our method achieves strong single-scene performance and achieves SOTA generalization, we discuss certain limitations of our method. The view transformer relies on epipolar constraints so that can only aggregate information from valid epipolar lines. Therefore, non-epipolar scenes and complex light transport might not be captured by the view transformer. Although we adopt feature extractor with large receptive fields to encode global light transport and our view transformer empirically works well on complex lighting effects, what is captured by the image encoder remains unclear. Moreover, epipolar correspondence for the boundary pixels sometimes are missing, which causes some minor artifacts (see Fig.
22


Reviews:
[{"review_id": "kL85uhGGW5F", "reviewer": "Reviewer_TeHV", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents the first NeRF reconstruction method, Generalizable NeRF Transformer (GNT), based on transformer. It introduces the view transformer which predicts coordinate-aligned features by aggregating information from epipolar lines on the neighboring views with attention and the ray transformer which renders novel views using attention to decode the features from the view transformer along the sampled points. Experimental results show that the GNT can successful reconstruct a NeRF model with more details.", "summary_of_the_review": "I am happy with this submission and I think the technical novelty of this work is sufficient to be accepted in ICLR. At the same time, I feel that this paper can be even better with further ablation study and comparisons (check my comments in weakness).", "strength_and_weaknesses": "Strength: This is, to my knowledge, the first transformer based NeRF reconstruction method. While the proposed method is not groundbreaking, it is a reasonable design and an advancement to study how NeRF reconstruction can be achieved by transformer.\n\nWeakness: I would expect more ablation study and comparisons of the proposed GNT with some more recent methods for NeRF reconstruction not limited to the vanilla NeRF and the MipNeRF. Indeed, there are many recent advancement in NeRF for speeding up the reconstruction, or use partial depth, or spherical harmonic coefficients for NeRF reconstruction. These methods are not referenced nor compared in the paper. Such addition references, comparisons and ablation study should be able to further strengthen the contribution of this work. In the current submission, this paper just read like this is another alternative for NeRF reconstruction, but the advantages of GNT over other recent NeRF reconstruction methods are not obvious.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "XcPQfXdpOL", "reviewer": "Reviewer_i8zE", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": " This paper addresses the problem of generalizable Nerf using a view transformer for generalization and a ray transformer for ray-based rendering.  For generalization, the view transformer uses multi-view geometry inductive bias to predict coordinate aligned features by aggregating information along the epipolar lines on other views. For rendering, the ray transformer renders novel views using features from the view transformer along the ray points during ray marching. Such designs allow authors to generalize the Nerf in the previously unseen scenes.", "summary_of_the_review": "Please, refer to the weakness regarding the failure cases and limitation of the proposed method.", "strength_and_weaknesses": "++ The addressed problem in this paper is a problem of high interest.\n\n++ The proposed method is intuitive and meaningful.\n\n++ The paper is well written and easy to follow.\n\n++ The provided experimental results are comparable, and in fact exciting.\n\n++ The experimental evaluations provided in the supplementary material further illustrate the effectiveness of the proposed method.\n\n-- I do not find any major problem with the proposed method. However, a discussion regarding the failure cases and limitations is missing. The paper can benefit from demonstrating the examples where the proposed method lacks to generalize. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "PLxjUF8EdtD", "reviewer": "Reviewer_qn8f", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a neural architecture GNT that uses transformers for novel view synthesis from multi-view images. Specifically, a view transformer aggregates feature from feature maps of input images to obtain features of sampled 3D points in the scene. Then a ray transformer learns to aggregate features of points along each ray and decode code from them. Experiments have shown that GNT can achieve comparable results for single-scene overfitting settings. And achieve better results on cross-scene generalization settings compared with other baselines.", "summary_of_the_review": "Overall, the submission presents marginal improvements over existing method. However, the main concern is the lack of technical contribution, and that the empirical evaluation is not convincing.\n\n---------\nUPDATED: after rebuttal I think my concerns have been partly addressed. I changed my score to weak accept.", "strength_and_weaknesses": "+ The paper demonstrates that pure transformer architecture with geometry prior can learn to perform the task of novel view synthesis.\n+ The model has a better performance compared with other baselines in the cross-scene generalization settings.\n\n- My major concern about the paper is the lack of technical novelty compared with previous works. There are a number of works that leverage transformers to learn the pipeline of volume / light field rendering such as [1][2][3][4]. Specifically, NeRFormer [1] also uses the transformer for aggregating features of every 3d point from feature maps of multi-view images using epipolar constraints. GNT simply replaces the volumetric rendering part with a transformer, which introduces limited novelty.\n- The benefits of the proposed architecture is not empirically convincing. GNT performs worse than NLF[4] in the single-scene settings. On cross-scene settings, the improvement is also very marginal. I would like to know the computing cost (training time inference time) of GNT compared with GNT with volumetric rendering ablation.\n- The authors claim that the GNT can better handle challenging view-dependent effects. However, no quantitative results (e.g. on the Shiny dataset) can support this point. And in Figure 4, results of the strongest baseline NLF is not shown. This makes it hard to evaluate the effectiveness of GNT for view-dependent effects.\n\n[1] Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction. ICCV 2021.\n[2] Scene Representation Transformer: Geometry-Free Novel View Synthesis Through Set-Latent Scene Representations. CVPR 2022.\n[3] IBRNet: Learning Multi-View Image-Based Rendering. CVPR 2021.\n[4] Light Field Neural Rendering. CVPR 2022.\n\nA few questions:\n1. In Table 3, why the Epipolar Agg. → View Agg variant is significantly worse than NLF when they share a similar two-stage aggregation strategy?\n2. What is the training/inference time of GNT in single-scene and cross-scene settings?\n\nMinor:\n- Section 4.3 Datasets section, “IBRNetto” -> “IBRNet to”\n- In Figure 2, symbols of input should have explanations. Otherwise it is very hard to understand the figure.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "OlOT0Q130b7", "reviewer": "Reviewer_xoKn", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a novel approach based on two transformers to reconstruct NeRF. The view transformer leverages multi-view geometry (epipolar geometry) to guide the attention-base scene representation. Next, the ray transformer renders novel views using attention to decode from the representation learned by the view transformer. The proposed GNT outperforms other state-of-the-art methods in both single-scene and cross-scene scenarios.", "summary_of_the_review": "This is one of the first papers using the attention mechanism in most components for the task of NeRF.\nTwo novel transformers are introduced to learn a neural representation generalized across scenes and better render refractions and reflection. Finally, The proposed GNT outperforms other state-of-the-art methods in single-scene and cross-scene scenarios.", "strength_and_weaknesses": "Strength\n1. View transformer enables the learning of a neural representation generalized across scenes.\n2. Replacing explicit volume rendering formula with a learned ray transformer. This might contribute to better render refractions and reflection.\n3. Outperforming other state-of-the-art methods in both single-scene and cross-scene scenarios.\n\nWeakness\n1. GNT includes two transformers. It is not clear how stable is the training with respect to random seed or changing some hyperparameters.\n2. Some clear artifact at the bottom parts in the Horns example.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster