Laser: Latent Set Representations for 3D Generative Modeling
Authors: Pol Moreno
Adam R. Kosiorek
Heiko Strathmann
Daniel Zoran
Rosalia Galiazzi Schneider
Björn Winckler
Larisa Markeeva
Theophane Weber
Danilo Jimenez Rezende
Venue: Submitted to ICLR 2023 (2023)
Keywords: generative models
nerf
computer vision
3D scenes
novel view synthesis
variational auto-encoder

Abstract: NeRF provides unparalleled fidelity of novel view synthesis---rendering a 3D scene from an arbitrary viewpoint. NeRF requires training on a large number of views that fully cover a scene, which limits its applicability.
While these issues can be addressed by learning a prior over scenes in various forms, previous approaches have been either applied to overly simple scenes or struggling to render unobserved parts.
We introduce Laser-NV---a generative model which achieves high modelling capacity, and which is based on a set-valued latent representation modelled by normalizing flows.
Similarly to previous amortized approaches, Laser-NV learns structure from multiple scenes and is capable of fast, feed-forward inference from few views. 
To encourage higher rendering fidelity and consistency with observed views, Laser-NV further incorporates a geometry-informed attention mechanism over the observed views.
Laser-NV further produces diverse and plausible completions of occluded parts of a scene while remaining consistent with observations.
Laser-NV shows state-of-the-art novel-view synthesis quality when evaluated on ShapeNet and on a novel simulated City dataset, which features high uncertainty in the unobserved regions of the scene.

Under review as a conference paper at ICLR 2023
LASER: LATENT SET REPRESENTATIONS FOR 3D GENERATIVE MODELING
Anonymous authors Paper under double-blind review
ABSTRACT
Neural Radiance Field (NeRF) provides unparalleled fidelity of novel view synthesis synthesi--reddering a 3D scene from an arbitrary viewpoint. NeRF requires training on a large number of views that fully cover scene, which limits its ap- plicability. While these issues can be addressed by learning a prior over scenes in various forms, previous approaches have been either applied to overly simple scenes or struggling to render unobserved parts. We introduce Latent Set Representations for NeRF-VAR (LASRR-NV--A- generative model which achieves high modelling capacity, and which is based on set- set-valued latent representation modelled by normalizing flows. Similarly to previous amortized approaches, LASER-NV learns structure from multiple scenes and is capable of fast, feed-forward inference from few views. To encourage higher rendering fidelity and consistency with observed views, LASER-NV further incorporates geometry-informer attention mechanism over the observed views. LASER-NV further produces diverse and plausible com- pletions of occluded parts of scene while remaining consistent with observations. LASER-NV shows state-of-the-ar novel-view synthesis quality when evaluated on ShapeNet and on a novel simulated City dataset, which features high uncertainty the unobserved regions of the scene.
INTRODUCTION
Probabilistic scene modelling aims learn stochastic models for the structure of 3D scenes, which are typically only partially observed (Eslami et al., 2018, Kosiorek et al.] 2021 Burgess et al.) (019). Such models need to reason about unobserved parts of scene in way that consistent with the observations and the data distribution. Scenes usually represented latent variables, which are ideally compact and concise, yet expressive enough to describe complex data.
Such 3D scenes can be thought of as projections of light rays onto image plane. Neural Radiance Field (NeRF, (Mildenhall et al., 2020)) exploits this structure explicitly. It represents scene as a radiance field (a.k.a. scene function), which maps points in space (with the corresponding camera viewing direction) to color and mass density values. We can use volumetric rendering to project these radiance fields onto any camera plane, thus obtaining images with CNN, this rendering process respects 3D geometry principles. NeRF represents scenes as parameters of an MLP, and is trained to minimize the reconstruction error of observations from single scenc-resultin in unprecedented quality of novel view synthesis.
image. Unlike directly predicting
For generative modelling, perhaps the most valuable property of NeRF the notion of 3D geometry embedded in the rendering process, which does not need to be leamed, and which promises strong generalisation to camera poses outside the training distribution. However, since NeRF's scene representations are high dimensional MIP parameters, they are not easily amenable to generative modelling (Dupont et al. 2022). NeRF-VAF (Kosiorek al. (02T) embeds NeRF in a generative model by conditioning the scene function a latent vector that inferred from a set of 'context views". then uses NeRF'S rendering mechanism to generate outputs. While NeRF-VA VAE admits efficient inference of compact latent representation, its outputs lack visual fidelity. This is not surprising, given its simple latent structure, and the inability to directly incorporate observed features. In addition, NeRF- VAE does not produce varied samples of unobserved parts of scene.
number of recent deterministic methods (Yu et al. 2021 Trevithick & Yang 2021 Wang et al.. 2021) uses local image features to directly condition radiance fields in 3DD. This greatly improves

---

Under review as a conference paper at ICLR 2023
p(Z sample
Latent Set Representation
[Z .2.
Zx
render( 2 V,C')
Multi-view geometry
Unobserved region
Encodee
Scone function
Laser-N NV
colomr demstry
sample
d. Query pair.
render( P.CC)
Contest V= (I",
Target camera C
Figure 1: Left: LASER-NV infers set-valued latent Z from the context V that consists of N image and camera pairs (I", C²). On querying the scene function at a point x; with direction d,, the latents combined with local features H" that are back-projected from the context views-prodducng color and density. Right: Rendering a novel viewpoint may include observed (green, an example query point the left) and unobserved parts (red) of the scene. Conditioned the context V, LASER-NV allows sampling multiple scene completions that are consistent with the context views (green arrows) while providing varied explanations for the unobserved parts of the scene (red arrows). We show two such samples for the same target camera C Also see the gif in supp. material showing fly-through for four prior samples conditioned on the same views.
reconstruction quality of observed parts of the scene. However, these methods are still unable to produce plausible multimodal predictions for the unobserved parts of a scene.
In this work. address NeRF-VAE's shortcomings by proposing Latent Set Representations for NeRF- VAE (LASER-NV). To increase modelling capacity, LASER-NV uses an urbitrarily-sizer set of latent variables (instead of just one vector) modelled with normalizing flows. To further enable producing samples which are consistent with observed parts, we make the generative model conditional on set of context views (as opposed to conditioning only the approximate posterior). LASER-NY offers superior visual quality with the ability to synthesise multiple varied novel views compatible with observations. Figure shows LASER-NV's key components and abilities. We include gi f in supp. material showing thy-throughs of additional prior samples, see Section 4.5 for details.
Our contributions are as follows:
We introduce novel set-valued latent representation modelled by purpose-built permutation- invariant normalizing flows conditioned on context views. We show that increasing the number of latent elements improves modelling performance, providing simple way to trade off computation for quality without adding new model parameters. We also verify that increasing latent dimensionality in NeRF-VAE offers no such benefits. In contrast with deterministic scene models in the literature, our probabilistic treatment over the latent set allows covering multiple models when predicting novel views.
We develop novel attention mechanism to condition the scene function on the set-valued latent as well as additional local features computed from context views. We show that including local features further improves visual quality.
We evaluate LASER-NV on three datasets: category-agnostic ShapeNet dataset, Multi- ShapeNet, and on a novel "City" dataset that contains a large simulated urban area and poses significant challenges the unobserved parts of the scene. Our model overcomes some the main limitations of NeRF-VAF and also outperforms deterministic NeRF models novel view synthesis in the face of uncertainty.
benchmark for novel view synthesis due high uncertainty in
2

---

Under review as a conference paper at ICLR 2023
2 BACKGROUND: NERF & NERF-VAE
Neural Radiance Field (NeRF, (Mildenhall et al. 2020) represents 3D scene as a scene function F(x, d) of a point coordinate z E R and direction d R3, and outputs the colour c E R and volume density 2 O. This function is parameterized using neural network (typically fully connected MLP). To obtain a pixel colour, the scene function is evaluated at many points along the corresponding ray in the volumetric rendering process.
This volumetric rendering integral is in practice approximated using numerical integration; see (Mildenhall et al., 2020 Blinn 1982) for details. We use = render C) to denote the image R#xW*3 x3 for the rays of camera C' and scene
rendering process that outputs the image function F. The camera = (K, R, t) is specified by its intrinsic parameters K E RR3x3 and extrinsic parameters given by rotation R R and translation t E R In its original formulation, the parameters NeRF are optimized by minimizing an error function (typically the mean-squared error) between rendered images and the ground-truth images across many views of single scene.
While NeRF learns high-fidelity scene representations when many views are available, it does not learn prior over scenes, does not provide compact scene representations, and does not admit efficient inference (i.e. fast estimation of the scene parameters from a few input views). To address these issues, VAE (Kosiorek al.) (202)) embeds NeRF as a decoder in a variational auto-encoder (VAE, (Kingma& & Welling 2014; Rezende et al. 2014)). Each scene is represented by a latent vector (instead of the set of parameters of an MLP as in NeRF) which can be either sampled from Gaussian prior p(z) or from the approximate posterior q(z C) inferred from set of context views and associated cameras C. The latent z used to condition an MLP that parameterizes the scene function, Fo z) (a;, d:) + Cj. T; Here, parameters 0 are shared between scenes, while is scene specific. NeRF- VAE uses a Gaussian likelihood p( | x, C') with the image rendered from the sampled scene function as mean, and fixed standard deviation.
3 LATENT SET REPRESENTATIONS FOR NeRF- VAE
LASER-NV is conditional generative model, conditioned {I",C")
of input views V C. -- Given these views the model infers a latent representation that can be used for novel view synthesis.
In order to represent large-scale and complex scenes we use Latent Set Representations (LASERS), a set of latents of the form 2 = {= where K is hyperparameter. In the NeRF-VAF, the scene function MIP is directly conditioned on the inferred latent. In contrast, LASER-NV'S scene function conditioned on the latent set Z and features H {H" Jn=1 J extracted from N input views, and integrates information from both. This results in the form Fe ( Z, H) (æt, d4) + C₁ Ti which we
detail further in Section 3.2
We obtain the input features H by separately encoding the input views using a convolutional neural net (CNN) that maintains spatial structure. These feature maps are subsequently used in the condi- tional prior, the posterior, and the resulting scene function, all of which we describe next. Further architectural details provided in Apppendix A
3.1 CONDITIONAL PRIOR AND POSTERIOR
Given set of context views V = (Tm C consisting of images I" and corresponding cameras C", LASER-NV defines a conditional prior p( (2V) over a latent set of size K, parameterized by a permutation-invariant normalizing flow- a distribution model that allows for both fast and exact sampling and efficient density evaluation (Reennde & Mohamed 2015 Papamakarios et al. 2021).
Flows are defined by invertible mapping f of random variables, starting from base distribution Pol ()). When f is composed of multiple invertible mappings fi.... fm. the resulting density is given by
M
det f. (m 1), D)
p(2 V) po (F
II
(1)
821 (m)
3

---

Under review as a conference paper at ICLR 2023
who parameterize the transformations in each layer using transformers, preserving permutation invariance. Specifically, we propose novel design to condition the flow distribution on the features H. extracted from the context views. To achieve that, we use split coupling layers (Dinh et al. 2017), where for each layer, we first apply self-attention to the set of latents, followed by voss-atiention attend the context features. The output of the second transformer defines the parameters for an invertible affine transformation of the random variables. Implementation details of our normalizing flow is in Appendix A.I
Our flow design closely follows that of Wirnsberger al. (2020 2021
Since LASER-NV is trained as a VAE, we define an approximate posterior g(Z(u u U V), where u= = [I', It=1 are target views that the model is trained to reconstruct, denoted as posterior context. The posterior and conditional prior are both distributions over the latent space, where the posterior is conditioned on the union of prior and posterior context. We model the posterior distribution using the same architecture as the conditional prior, but with separate set of learned parameters.
3.2 DRCODER & SCENE FUNCTION
When evaluating the scene function, e. computing colour and density for point x;; d, LASER-NV has access to both the latent set 2 as well as the features H extracted from the context views V. We now describe how integrate those two, which consists of querying the latent set for each point, and using 3D projection to extract relevant features.
Querying the Latent Set While the latent set representation Z contains information regarding the entire scene, when querying for specific point 2;, we would like to extract only relevant information from 2. We propose using a transformer cross-attention model to compute a single feature vector from Z. In the first layer, queries are computed from the positional encoding of æ;, and keys and values are computed from the elements of Z. subsequent layers, queries are computed from previous layers' output. We denote the output of the latent querying latent features ž, each corresponding to a query point x,,
Local Features When synthesizing novel view given a few input views, a number of existing F-RF-based methods uses 3D projection to determine which features from the context views are useful for explaining particular 3D point (Yu al. [02 Trevithick & Yang 2021] Wang et al. 2021). LASER-NY incorporates this idea, and in particular follows the design of pixeINeRF (Yu al. T as it is shown to achieve strong results in a wide range of domains.
1= described above, along with their corresponding camera matrices {C") {C") J==1- Given a query 3D point x; and direction d, we obtain the point p R2 in the image space of the nh context view using the known intrinsic parameters. The corresponding local features are then bilinearly interpolated h
We use the encoded input views "
(p"). We refer to h, as local focalfeature features since they are spatially localized based on the projected coordinates, as opposed to being a global aggregation of all context features. We finally obtain hạ for each context view by separately processing the features h" using a residual MLP that has as an additional conditioning on the image-space projected points and directions.
Integrating Latents and Local Features In contrast to pixeINeRF, which does not have latents, LASER-NV integrates the both local and the latent features. It does with a transformer:
N
= MoMa; U is_ visibbib,,, C")) U (2:))
1=1
The attention queries are computed from latent features Σ; (see above), and attend to the combined set only those context views from which this
of local features and to ž. Note that, for point x,, point is visible. Using a transformer to process encoded features from multiple views with epipolar constraints has been proposed in prior work on Multi-view Stereo, see e.g. (He et al. 2020 Xi et al. 2022 Wang et al. 2022 Ding et al. (022).
The final step in evaluating the scene function is use f: to condition residual MLP similar to NARF-VAE resulting in LASER-NVE scene function F. 2, H) (x,, d:) + C, or which is used to volume-render image outputs.
4

---

Under review as a conference paper at ICLR 2023
3.3 LOss
Similarly to NeRF-VAE, our objective function is given by the evidence lower-bound:
M
L(, Rzza
Σ
logpl
l C*, C22 - (T(((2 ( (UV) ( V))
(3)
where I' and C are the images and cameras from the posterior context views U. We use Gaussian likelihood term for p(r C 2, with a fixed variance.
,,,
4 EXPERIMENTS
4.1 DATASETS
We first evaluate LASER-NV on novel dataset based on a set of synthetie, procedurally generated, environments consisting of urban outdoor scenes. This dataset contains posed images of a virtual city with objects of multiple scales, such as house blocks, roads, buildings, street lights, etc (but no transient objects such as cars or people). As such, it is an extremely challenging scene representation benchmark, especially when representations are inferred from few views ruly-resulting in partial observability and high uncertainty. In this situation, when evaluated on unobserved parts of scene, we want a model to produce predictions that are varied, self-consistent, and aligned with the data distribution. Each scene consists of neighborhood block, and randomly chosen designated point of interest (road intersection or park) is selected within. Images are then rendered by placing the camera random nearby positions and viewpoints from the point of interest. The dataset has 100K training scenes, and 3200 test scenes. We provide further details and dataset samples in Appendix B
To test LASER-NV's ability to produce near-deterministic scene completions in low-uncertainty setting, we compare it against a number of previous methods on the ShapeNet NMR dataset (Kato) al. 2018), which consists of several views of single objects from 13 different ShapeNet categories. Finally, we evaluate on the more challenging MultiShapeNet-Hae which tests the models capabilities in the presence of large number of cluttered objects of varying size with detailed textures and realistic backgrounds.
(MSN- (Sajjadi et al., 2021).
4.2 MODELS & EVALUATION
On the City dataset, we compare LASER-NV with Multi-view Conditional NeRF (MVC-NCRF) and NeRF-VAE; see Figure 7 in Appendix A for model diagrams. MVC-NeRF (Figure 7c is variant of LASER-NV FFigure 7 which does not use LASER-N- -N''s latent set representations but instead only relies on deterministically rendering new views using local features. It therefore closely resembles the design of piXeINeRF but uses the same architecture for the image encoder and scene function Fa as LASER-NY NY to ensure they are comparable. On ShapeNet, we further show quantitative results of a number of non-NeRO novel view synthesis methods: Scene Representation Networks (SRN, Sitzmann et al. (219)), Differentiable Volumetric Rendering (DVR. Niemeeyr et al. (2020). ). and Scene Representation Transformers (SRT, [Sajjadi et al. (2021)). During training, LASER-NY NVV uses random subset of 4 of the 23 target views as posterior context viewe[) MVC-NeRF and NAR trained to directly predict the 23 views given single context view. At test time, LASER-NV samples the latents from its conditional prior whereas NeRF-VAE uses its posterior (given that its prior is unconditional). Importantly, all models in this evaluation use one input view and predict 23 target views.
At evaluation time, feed each model with set of context views, and evaluate reconstructions of those same views, and predictions of novel views of the same scene. We evaluate reconstruction ability via likelihood estimates, and standard image similarity metrics PSNR, and SSIM (Wang
et al. 2004). For novel view predictions we report the Fréchet Inception Distance (FID, Heusel
al. 2017)) between the marginal distribution of samples and the evaluation datase This metric
While using the 23 views posterior context would he the standard up (Bayer et al. 2021), find that using randomly-sampled views much cheaper computationally and yet sufticient for training.
P IImage similarity metrics such as PSNR and SSIM are not useful when there exists large multimodal space of possihle predictions the case in the City.
5

---

Under review as a conference paper at ICLR 2023
highlights the incapability of deterministic models to generate varied and plausible samples in face of uncertainty. Note that since NeRF-VAL VAE unconditional, we use the learned approximate posterior as the conditional distribution. In the City, thus train separate model (one to use for reconstruction metrics that has posterior context views and another one with views for predictions). For ShapeNet results all models use one context view.
On MSN-Hard, we train LASER-NV with 5 prior context views and 3 target views. We report the test PSNR of I novel view given 5 input views, and compare with the results reported in (Sajjadi al. (2022). The methods of comparison include two amortized non-NeRF models based on light fields, Scene Representation Transformers (SRT) and Object SRT (OSRT (Sjjaadi et al. (2022) )) and two amortized NeRF methods, pixeINeRF and Volumetric OSRT (VOSRT).
4.3 ABLATIONS
We further ablate LASER-NV to isolate contributions of individual model components. To quantify the effects of conditioning the prior we compare LASER-NV to a conditional version of NeRE-VAE Note that the original NeRE-VAB used an unconditional prior (Kosiorek et al. 2021). We also compare version that does not condition the scene function with local context features, cf.
LASER-NT
Equation (2) and Figure 75 in Appendix A denoted by LASER-NV (NO GEOM.). LASER-NV, LASER-NV V (NO GFOM.), N-RF-VAR variants, and MVC-NeRF share architectures for the image encoder and the scene function Fo.
4.4 TRAINING AND IMPLEMENTATION DETAILS
We use the hierarchical sampling technique of (Mildenhall et al. 2020) colour, which involves learning two separate sets of scene function parameters as well as optimizing a corresponding colour likelihood term for each step. For experiments with the City, we leverage ground-truth depth maps using the method proposed by [Stelzner al. (202T) in order to train all the models with fewer scene function evaluations per ray. Not using ground-truth depth significantly increases training time while only marginally lowering reconstruction PSNR, see seepApenddxx for discussion and Fig. in Appendix B for visualisations. Importantly, when evaluating our models we revert to volumetric rendering. All generative models are trained by optimizing the evidence lower bound; where anneal the KL term in (Eq. (3) All experiments use the Adam (Kingma & Ba 2014) optimizer. We provide full details of model architectures in in[ppendix A and training procedures in Appendix E
when rendering a pixel
4.5 RESULTS
Results the City dataset are shown in Table In terms of reconstructing observed input views, both LASER-NV and MVC-NeRF, due to the use of explicit geometrical knowledge, result in excellent reconstructions. NeRF-VAE, however, needs to reconstruct input views purely from its latent represen- tation, resulting in low fidelity reconstructions. For prediction of unseen views, the output distribution of VA is closer the actual evaluation data (lower FID), in contrast MVC-NeRF which outputs a single prediction, and NeRF-VAL which produces low quality outputs that do not vary much. We show representative example reconstructions and predictions in Fig. We also include gi supp. material showing fly-throughs of model samples.
Both MVC-NeRF and LASER-NV outperform existing published methods when predicting novel views of these single-object scenes. We further observe low quality predictions of NeRF- VAB in example predictions in FFg. IT Finally, |Fig. 12 shows hand-selected example where the input view has ambiguity and how LASER-NV can sample plausible variations. As in the City experiment, LASER-NV produces plausibly varied predictions of the object in contrast to the other methods.
Results on ShapeNet are shown in Table 2
Results on MSN-Hard are shown in Table LASER-NV clearly outpeforms the NeRF-based models, and improves over OSRT and the original SRT. Only modified SRT shows higher PSNR (25.93) over LASER-NV (24.45). Note that SRT does not learn to estimate densities nor depth maps and
"o create this visualisation we trained the model on modified City dataset with birds-cye views.

---

Under review as a conference paper at ICLR 2023
Reconstruction
Prediction
I Log-I Bog-Liken Likelihood
PSNRT
SSIM
FID
MVC-NeRF
28.63
0.910
91.92
Cond. NeRE-VAE
3.51
23.94
0.709
50.98
NeRF-VAE
3,50
23.92
0.708
52.12
LASER-NV (NO GEOM.)
3,82
27.13
0,870
24.19
LASER-NV
3.83
29.61
0.923
22.54
Table 1: Results in City. The color likelihood is computed via importance sampling (10 samples); note that MVC-NeRF is deterministic and thus we do compare it on this metric. Reconstruction: We report performance of reconstruction of the 2 context views. Prediction: We evaluate the models on previously unobserved views using the FID score. MVC-NeRF provides single deterministic prediction whereas for the other models we average over 10 independent samples.
Content view
Context view
view
view
Ground-uuul
MVC-NNRR
NeRE VAE
LaSee NV
Reconstnacion
Reconsrruction
Prediction
Prediction
Prediction
Prediction
Figure 2: Representative reconstructions of observed views and predictions of partially unobserved novel views. The models are conditioned on two context views (top left). The first two columns show reconstructions of those views. Note how NeRF-VAR struggles to reconstruct both views accurately, while the other models do well. We show predictions of two novel views (top right), where the first view is partially observed in the context (highlighted in cyan color) and the second view is fully unobserved. LASER-NV'S predictions are consistent with the observed part of the context, diverse, and plausibly so. NeRF- VAE'S predictions lack quality and diversity. MVC-NeRF's only produces single prediction, which is not very plausible.
deterministic, Fig. 3 shows a prediction example using LASER-NV, and further examples are in Appendix C which demonstrate high fidelity colours and densities.
4.6 TRAIN- AND TEST-TIME SCALING DATA EFFICIENCY
Latent set representations allow trading computation for increased model capacity. We train LASER- NV on the City dataset with increasing latent set sizes K, and compare conditional NeRF-VAR VAF also trained with increasing latent dimensions" Al Figure 4 shows reconstruction log- leglliilisssds as
Note that this increases the number of NeRF-VAE's parameters.
7

---

Under review as a conference paper at ICLR 2023
DVR*
SRN#
SRT*
pixelNeRF*
NeRF-VAE
MVC-NeRF
LASER-NV
PSNR f
22.70
23.28
27.87
26.80
25.36
28.03
27.92
SSIM
0.860
0.849
0.912
0.910
0.875
0.925
0.923
Table 2: Category-agnoste ShapeNet evaluation by reconstructing 23 views from one input view for all models. We further report numbers from the respective papers DVR* (Niemeyer et al. 2020 SRN* (Sitzmann et al. 2019), SRT* (Sajjadi et al. 2021), and pixelNeRF* Yu et al. 2021
NeRF
Light field
pixeINeRF
VOSRT
LASER-NV
OSRT
SRT
SRT++
PSNR t
21.97
21.38
24.45
23.33
23.54
25.93
Table 3: Novel view PSNR of one view from 5 input views on HardH PixelNeRF results are
from (Sajadi al. (2021) and the remaining from Sajjadi et al. (2022) SRT++ is improved SRT (Saijadi et al. 2022).
Inputs
GT
Prediction
Depth
Figure 3: Novel view syntesis with LASER-NV on MSN-Hard.
function latent capacity. While LASER-NV consistently improves its reconstructions with larger latent sets, cond. NeRF-VAE quickly saturates at 256 latent dimensions, at worse reconstructions compared to LASER-NV. Further, in Appendix D we provide an analysis of test-time scaling- we show that increasing the latent set size at test time does not result in clear change of performance. Finally, in Figure 14] we show that LASER-NV performs better at test time than NeRF-VAE and MVC-NeRF for different training dataset sizes.
RELATED WORK
Neural scene representations Inferring neural scene representations using deep generative model of scenes is an active rescarch area, and we mention some of the most closely related approaches below. NeRF-VAE learns a generative model of 3D scenes which is only shown to work relatively simple scenes. When applied to more complicated data (see Fig. 2). it produces blurry reconstructions and novel views. We posit that this is caused by insufficient capacity of its latent representation 2. Generative Query Network (Eslami et al. (2018) similarly learns distribution over scenes, but its decoder is CNN. Subsequent work (Rosenbaum et al. 2018) improves its representational capacity by using an attentive mechanism when conditioning the decoder from context an analogous way to how LASER-NT improves over NeRF-VAE. By implementing NeRF in our decoder we gain the benefits of more robust generalization capabilities to novel viewpoints compared the convolutional counterparts (Kosiorek et al., (021). Simone (Kabra et al. 2021) is convolutional generative model that performs high quality novel view synthesis and while also performing unsupervised segmentation of input videos. The authors in (Dupont al. 2022) propose directly modelling the underlying radiance fields functions, which they refer as 'functa'. Both our method and theirs learn prior over scene functions with model that has a large representational capacity. [Bautista et al. (2022) propose two-stage method to learn generative model of 3D scenes. As LASER-NV. their model can conditionally sample radiance fields. However, they focus on scenes with dense views, whereas LASER-NV works with sparse views.
8

---

Under review as a conference paper at ICLR 2023
Train colour log likellhood
Train colour log likellhood
3.8
3.8
Wettiood
Latent dim 48-142 128
Letent dim- 1024
10g
Latent dim 24 128
100
Latent dim 512
dim 256
Letent dim 128
Letent dim 120
3.0g
300
Training step
le5
Training step
1e5
(a) LASER-NV
(b) Conditional NeRF- VAE
Figure 4: Training reconstruction performance in City of LASER-NV with an increasing latent set size (each with 128 dimensions) vs a cond. NeRF-VAE with increasing latent dimensionality.
Deterministic novel view synthesis NeRF relies on optimizing the parameters of an MLP scene function, separately for every scene. A numher of methods in the literature propose amortizing the mapping of input views the scene function using multi-view geometry, see e.g. IBRNet (Wang) et al. 202T), General Radiance Fields (GRF) (Trevithick & Yang, (021) and pixelNeRF (Yu et al.
2021. These methods extract features of the context views by using projective geometry to select which features to attend to, and are shown to generalize well and carry out high-fidelity predictions.
PixeINeRF is designed to work well in a solely view-centered coordinate system. MVSNerf (Chen
et al. (021) lifts context features to a 3D representation with strong novel-view synthesis performance. Scene Representation Transformers (SRT) (Sajjadi et al., 202T) use global features obtained from context views using generalization of the Vision Transformer (Dosovitskiy et al. (020), which bears similarities to LASER-NV. SRT bypasses the use of 3D geometry entirely, yet shows remarkable
performance on real world data.
NeRF for large-scale scenes A number of NeRF methods have been developed to scale up to city-wide scenes. They work by optimizing multiple separate scene functions for separate blocks of scene, which are then dynamically used when rendering (Tancik et al. 2022 Turki et al. (02T). Alternatively they design a scene function that represents the scene at multiple levels of scale (Martel
et al. [202]] Xiangli et al. 2021). Similarly, Mip-NeRF (Barron et al. (2021) allows prefiltering the
scene function inputs to better capture coarse and fine details. These methods are orthogonal to ours since they do not learn conditional NeRF model sparse views nor address the uncertainty in predictions.
CONCLUSION
We propose LASER-NV, a conditional generative model of neural radiance fields capable of efficient inference of large and complex scenes under partial observability conditions. While recent advances in neural scene representation and neural rendering lead to unprecedented novel view synthesis capabilities, producing sharp and multi-modal completions of unobserved parts of the scene remains a challenging problem. We experimentally show that LASER-NY can model scenes of different scale and uncertainty structure, and isolate the usefulness of each contribution through ablation studies. While NeRF serves as a strong inductive bias for learning 3D structure from images. LASER-NY also inherits some of its drawbacks. Volumetric rendering with neural radiance fields computationally costly and can limit the model from real-time rendering. It remains to be seen whether fast NeRF implementations (Müller et al. 2022) can be of help. Furthermore. accurate GT camera information is required for learning and novel view synthesis; (Moreau et al. 2022) recently made progress on localization methods using NeRF. While learning generative scene model of real scenes remains an open problem, we believe this work is an important step in that direction. Finally, interesting direction for future research is incorporating object-centric structure and dynamics into LASER- NV, which might be useful for for downstream reasoning tasks (Sajjadi et al. 2022 Ding et al. 2021)

---

Under review as a conference paper at ICLR 2023
REFERENCES
Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin- Brralla, and Pratul P. Srinivasan. Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields. In International Conference on Computer Vision, 2021.
Miguel Angel Bautista, Pengsheng Guo, Samira Abnar, Walter Talbott, Alexander Toshev, Zhuoyuan Chen, Laurent Dinh, Shuangfei Zhai, Hanlin Goh, Daniel Ulbricht, Afshin Dehghan, and Josh Susskind. GAUDI: Neural Architect for Immersive 3D Scene Generation. In Advances in Neural Information Processing Systems, 2022 arXiv/2207.13751
Justin Bayer, Maximilian Soelch, Atanas Mirchev, Baris Kayalibay, and Patrick van der Smagt. Mind the Gap when Conditioning Amortised Inference Sequential Latent- Varriabe Models. In International Conference Representation Learning, 2021 StXXv//2101.00445
Christopher M. Bender, Kevin O'Connor, Yang Li, Juan Jose Garcia, Junier Oliva, and Manzil Zaheer. Exchangeable Generative Models with Flow Scans. In AAAI Conference on Artificial Intelligence, 2020.
James F. Blinn. Light Reflection Functions for Simulation of Clouds and Dusty Surfaces. In SIGGRAPH, 1982.
Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. MONET: Unsupervised scene decomposition and repre- sentation. arXiv/1901.
2019.
Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su. Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo. International Conference Computer Vision, 2021 arXiv/2133115595
David Ding, Felix Hill, Adam Santoro, Malcolm Reynolds, and Matt Botvinick. Attention over learned object embeddings enables complex visual reasoning. Advances Neural Information Processing Systems, 34, 2021.
Yikang Ding, Wentao Yuan. Qingtian Zhu, Haotian Zhang, Xiangyue Liu, Yuanjiang Wang, and Xiao Liu. Transmvsnet: Global context-aware multi-view stereo network with transformers. In Conference on Computer Vision and Pattern Recognition, 2022.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. In International Conference Representation Learning, 2017 arXiv/1605.08800
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly. et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.1 11t.2999./1199.... 2020.
Emilien Dupont, Hyunjik Kim, M. Ali Eslami, Danilo Jimenez Rezende, and Dan Rosenbaum. From data to functa: Your data point is a function and you should treat it like one. ArXiv, arXiv/2201. 2022.
SM Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S Morcos, Marta Gar- nelo, Avraham Ruderman, Andrei A Rusu, Ivo Danihelka, Karol Gregor, et al. Neural scene representation and rendering. Science, 16494)) 2018. 360.
Yihui He, Rui Yan, Katerina Fragkiadaki, and Shoou-I Yu. Epipolar transformers. In Conference on Computer Vision and Pattern, 2020.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge local nash equilibrium. Advances in Neural Information Processing Systems, 30, 2017.
Rishabh Kabra, Daniel Zoran, Goker Erdogan, Loic Matthey, Antonia Creswell, Matt Botvinick, Alexander Lerchner, and Chris Burgess. Simone: View-invariant, cemporally-abstracatee object representations via unsupervised video decomposition. Advances Neural Information Processing Systems, 34, 2021.
10

---

Under review as a conference paper at ICLR 2023
Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. Neural 3d mesh renderer. In Conference Computer Vision and Pattern, 2018.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv/1412.6989, 2014.
Diederik P Kingma and Max Welling, Auto-encoding variational bayes. In International Conference on Learning Representations, 2014.
Adam R. Kosiorek, Heiko Strathmann, Daniel Zoran, Pol Moreno, Rosália G. Schneider, Sovna Mokr'a a, and Danilo Jimenez Rezende. Nerf-vae: geometry aware 3d scene generative model. In International Conference Machine Learning, 2021 arXiv/2104.00587.
Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, and Simon Lucey. BARF: bundle-adjusting neural radiance fields. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, 2021.
Julien N. P. Martel, David B. Lindell, Connor Z. Lin, Eric Chan, Marco Monteiro, and Gordon Wetzstein. Acorn: Adaptive coordinate networks for neural scene representation. ACM Trans. Graph., 10:58:1-58:13,, 2021.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes neural radiance fields for view synthesis. In European Conference on Computer Vision, 2020.
Arthur Moreau, Nathan Piasco, Dzmitry Tsishkou, Bogdan Stanciulescu, and Arnad de La Fortelle.
Lens: Localization enhanced by nerf synthesis. In Conference on Robot Learning. 2022.
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding, ACM Trans. Graph., 41(4):102:1-102::11 July 2022.
Michael Niemeyer, Lars M. Mescheder, Michael Occhsle, and Andreas Geiger. Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, 2020.
George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan. Normalizing flows for probabilistic modeling and inference. Journal of Machine Learning Research, 22(57): 2021.
Razvan Pascanu, Tomás Mikolov, and Yoshua Bengio. the difficulty of training recurrent neural networks. In International Conference Machine Learning, 2013 28.
Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International Conference Machine Learning, 2015.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International Conference on Machine Learning, 2014.
Dan Rosenbaum, Frederic Besse, Fabio Viola, Danilo J. Rezende, and S. M. Ali Eslami. Learning models for visual 3d localization with implicit mapping. CoRR, arXiv/1807.031111 2018.
Mehdi S. M. Sajjadi, Henning Meyer, Etienne Pot, Urs M. Bergmann, Klaus Greff, Noha Radwan, Suhani Vora, Mario Lucic, Daniel Duckworth, Alexey Dosovitskiy, Jakob Uszkoreit, Thomas A. Funkhouser, and Andrea Tagliasacchi. Scene representation transformer: Geometry ffree novel view synthesis through set-latent scene representations. arXiv/2111.13113 2021.
Mehdi S. M. Sajjadi, Daniel Duckworth, Aravindh Mahendran, Sjoerd van Steenkiste, Filip Paveti's Mario Luvei'c, Leonidas J. Guibas, Klaus Greff, and Thomas Kipf. Object scene representation transformer. 2022 Sktv/2200.06922..
11

---

Under review as a conference paper at ICLR 2023
Vincent Sitzmann, M. Zollhöfer, and G. Wetzstein. Scene representation networks: Continuous 3d-structure-aware neural scene representations. In Advances in Neural Information Processing Systems, 2019. urXiv//900.01618.
Karl Stelzner, Kristian Kersting, and Adam Kosiorek. Decomposing 3d scenes into objects via unsupervised volume segmentation. In arXiv, 2021 arXiv/2104.01 148.
Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul P. Srinivasan, Jonathan T. Barron, and Henrik Kretzschmar. Block-nerf: Scalable large scene neural view synthesis. ArXiv, arXiv/2202.05260 2022.
Alex Trevithick and B. Yang. GRF: Learning general radiance field for 3d scene representation and rendering. In International Conference on Learning Representations, 2021. arXivv2010004595.
Haithem Turki, Deva Ramanan, and Mahadev Satyanarayanan, Mega-nerf: Scalable construction of large-scale nerfs for virtual flythhoouhh. CoRR, arXiw/2112.10703, 2021.
Arash Vahdat and Jan Kautz. NVAE: deep hierarchical variational autoencoder. arXiv/2007.038898 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 2017. 30.
Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul Srinivasan, Howard Zhou, Jonathan T. Barron, Ricardo Martin-Brualle Noah Snavely, and Thomas A. Funkhouser. IBRNet: Learning mieti-view image-bace based rendering. In Conference on Computer Vision and Pattern Recognition. 2021.
Xiaofeng Wang, Zheng Zhu, Fangbo Qin, Yun Ye, Guan Huang, Xu Chi, Yijia He, and Xingang Wang. Mvster: Epipolar transformer for efficient multi-view stereo. arXiv preprint arXiv:2204.07346. 2022.
Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4):600-61,, 2004.
Peter Wirnsberger, Andrew J. Ballard, George Papamakarios, Stuart Abererombie, Sébastien Racanière, Alexander Pritzel, Danilo Jimenez Rezende, and Charles Blundell. Targeted free energy estimation via learned mappings. The Journal of Chemical Physics, 2020.
Peter Wimsserger, George Papamakarios, Borja Ibarz, Sébastien Racanière, Andrew Ballard, Alexander Pritzel, and Charles Blundell. Normalizing flows for atomic solids. arXiv preprint 1xxiv:2/11.000 10.066, 2021.
Junhua Xi, Yifei Shi, Yijie Wang, Yulan Guo, and Kai Xu. Raymvsnet: Learning ray-based Id implicit fields for accurate multi-view stereo. In Conference on Computer Vision and Pattern Recognition, 2022.
Yuanbo Xiangli, Linning Xu, Xingang Pan, Nanxuan Zhao, Anyi Rao, Christian Theobalt, Bo Dai, and Dahua Lin. Citynerf: Building nerf at city scale. ArXiv, arXiv/21 2021.
Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one or few images. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021,
12

---

Under review as a conference paper at ICLR 2023
A MODEL ARCHITECTURES
A.1 NORMALIZING FLOWS FOR LASER
We wish model our latent set representation 2 { 2x) using a flexible distribution that is efficient both to evaluate well sample from. Furthermore, the distribution should ideally be invariant to re-ordering the elements of the set. Normalizing flows (Rezende & Mohamed] 2015) allow specifying distribution with precisely these properties. Formally, is given by:
2 (Z) Z(m) fm (2(m-1),)) v). m 1, I....,,,, M, with Z = Z
(4)
where (-(-) is base distribution, fm are invertible transformations applied to an intermediate random variable, and Z is the final LASER after M flow layers. V is an additional set of context vectors that condition the flow transformation. The resulting probability density of 2 is
det 5f (z (5m(2(m-1)))) V)
p 12) po (0)
II m=2
(m
The need to compute the Jacobian of intermediate flow transformations puts constraints on the design of those transformations, as they need to be computationally efficient. We use a split-coupling flow similar to Ve (Dinh al., (017), and we follow similar architecture Wirnsberger et al. 2220 2021 in adapting it to sets. The method Flow Scans (Bender et al.] (2020) first introduced the general idea of adapting split-coupling to work sets, and Wirnsberger et al. 2020 2021) use a transformer for increased flexibility. A split-coupling transformation partitions the set 2 into two sets 2,, Z2 by splitting each element of the original set across channels (we split in half). At a high level, the transformation of one part is computed as a function of the other part. Finally, the original set is recovered by elementwise concatenation of both transformed parts. More formally, let be invertible function, split-coulling transform Im that maps 2 (mm-1) 2 (m) is given by three operations:
the ones used in
Z = 9 Z (m-1)) fm (z(m-1) v))
(m)) 9 2 (m-1) fm (m) 1))
(6)
((m) = (concat(2 (
(m)
(m)
k,
k. 11=
where 1 indicates dimensions of split 1 of element k. and V is conditioning context. Note how the parameters of the transformation g of one partition are derived from the other partition. To do so in permutation-equuvarian manner, the functions t are composition of two transformers: self-attention module is applied followed by a cross-attention conditioned on context features, defined in Appendix A.2 More specifically, when transforming split 1 as function of split 2, we have:
v (m 1) CrossAttm (SelfAttm
1/m-1)).1)
We finally apply the function g. in our the affine function Dinh et al. 2017). from the parameters obtained for every element in xb> m.
(m)
(m-1) exp(W Th Wa (m (m-1)) + W (m-1) for I.... K, 22
scale 22
(8)
#k.1
scale and W split-coupling flows, we find that including an additional linear invertible transformation after each fm be helpful. To do so, we use the Linear Permutation Equivariant (ffR") ), proposed in the Flow Scans work, the purpose of which is to allow capturing interdependencies in the dimensions of the set elements. The only difference that we parameterize each sub-block of the linear transformation using a full unconstrained matrix (instead of a diagonal matrix).
where Wm
bias are linear weights. When composing our complete flow with M layers of
In summary, our flow chains together the sequence of mappings f fle pling from this flow involves sampling from the base distribution followed by applying the chain of functions.
fir , J. JM Thus, sam-
13

---

Under review as a conference paper at ICLR 2023
Our normalizing flow differs in number of ways compared with Wirnsberger et al. (2020) 2021) First, our flow layers are additionally parameterized by cross-attention module in order to condition the distribution the context set; second, we don't I use rational-quadien the simpler affine coupling layer to work well. Notice that the size of the resulting set depends only the hyper-paramett K, that is the number of elements sampled from the base distribution. It does not depend on the context size, i.c. features extracted from input images.
splines, instead we found
use diagonal Gaussian base distribution, and the number of flow layers be M = 8 for the prior flow, and M = 4 for the posterior flows. Our self-attention and cross-attention are composed L layers, and 256 hidden units. Furthermore, we invert the chain of mappings (which is possible due to invertibility) for the prior flow as it we found it to be more numerically stable during training.
In our experiments,
A.2 ATTENTION MODULES
V/+1
Y i+1
Layer numu
layer
linear, ReLu
liner, ReLu
linear, ReLu
linear, ReL.u
layer norm
Jayer norm
Multi-head attention
Multi-head attention
keys
values
queries
values
quentes
lloear
linear
linear
Iinerr
linear
linear
context
Sell-attention block
Cross-attention block
Figure 5: Multi-head self-attention and cross-attention blocks. When combining multiple blocks, the outputs y) for layer become the inputs for layer
We use transformer attention Vaswani et al.] (2017) in various components of LASER-NV. We distinguish two types of attention modules: self-attention, where keys, queries and values are computed from the same input, and cross-attention, where some context is used compute keys and values, and queries computed from a different input). An attention module can be composed of multiple attention blocks, where each attention block is shown in Fig. 5] We use the standard multi- head dot-product attention with softmax weights. We define SelfAttn(y) L) be self-attention module with input and L block&Jann and, similarly, CrossAttn y c, L) to be a cross-attention module with context Note how SelfAttn and CrossAttn do include positional encoding that are often used in transformers.
A.3 IMAGE ENCODER
To encode the context images,
convolutional neural network based the ResNet architecture
from Vahdat & Kautz (2020) (also described in Kosiorek et al. (2021)) to independently encode context elements into
feature maps. The residual blocks that compose the encoder are shown in Fig. For all models and datasets we use I. = 4. Each context view consists of an image in RGB space with concatenated information of the camera position and direction.
For N context views, our last layer outputs Nhw vectors that form the conditioning set used to parameterize our distributions over Furthermore, the feature maps at different layers are also combined into feature stack to use as the context local features described in Appendix A.4
We often drop the 7. input argument for notational clarity.
14

---

Under review as a conference paper at ICLR 2023
H
channels 32
channels = d
Res block (2)
conv(1, 1)
guuuezze-an-exciiane
Res block (1)
2
H
batch norm, ReLuu
conv (3, s)
Res block (2)
(3
batch swish
batch norm, ReL.u
Res block (1)
x4
conv(3, 1)
conv(7, 1)
Block 0
batch norm, swish
Block 0
Res block(s)
concat(T",
tt" (t.d") d")
channels
channels df
Figure 6: ResNet architecture based on the encoder of NouveauVAIi Vahdat & Kautz 2000 Vahhaa
A.4 SCENE FUNCTION ARCHITECTURES
Fig. 7 provides a high-level overview of the information flows going into the scene function of each model, and describe implementation details of their components below.
[Z], Z ZK|
{ [11 Z2
ZK }
Latent Attention
Latent Attention
(I" C" N 1=1
Multi-view geometry
X, d
X,, d Query point
Query point
Outpur MLP
Output MLP
c; d;
C, 02
(a) LASER-NV scene function
(b) LASER-NV (NO GEOM.) scene function
Z
(I", JTCC
n-1
Multi-view geometry
d
Latent MLP
d Query point
Query point
Output MLP
Output MLP
c O2
(c) MVC-NeRF scene function
(d) NeRF- VAE scene function
Figure 7: Scene function architecture for LASER-NV (a), its ablations (b) and (c), and NeRF-VAE (d).
Latent attention query point x; attends to Z as per Fig. (a) using a (ooss-att(77(x))) Z, L). where (ax:)) is the circular encoding function of positions (see Appendix A.4). We the same architecture across datasets, setting = 3 layers, 512 hidden units, and 4 attention heads.
Local features from multi-view geometry We follow PixeINERF's design of stacking the features of the encoded images different levels of resolution, which allows the conditioning features to
15

---

Under review as a conference paper at ICLR 2023
capture context structure at different scales. For input images of resolution W x H, we stack the features of the outputs from all layers after the first layer:
H = stack( E TR " x
](=1):
where i = 2'. When stacking, we upscale the feature maps of all the feature maps of layers E (2... to the size of the feature map with largest resolution (I 1) using bilinear interpolation. The resulting set of stacked feature maps (one for each context view) is then used to condition the scene function.
As we described in Section 3.2, our multi- view geometry module is in part based on pixeINeRF's processing of context views. Particularly, process local features with exactly the fi ResNet detailed in Yu et al. (2021): positions and directions are projected to the view-space using the cameras intrinsic parameters, and then processed with 3 residual blocks of 512 hidden units and ReLu activation. When computing the projection, we also define the binary variable t, based on whether the projected coordinates are within the image plane.
The output features are integrated with our latent features via Mg which is a cross cross-attention module. Note that context features with of are not attended (we modify the pre-softmax logits of those context features by adding large negative number to the logits). In our experiments Ma has 3 layers and 512 hidden units.
Scene function output MLP models, as shown in Fig. 7 This component follows the residual MI.P architecture of the scene function of NeRF-VAF VAF. Namely, it takes as input the circular encoded positions y(::) and directions r(d)), and at every layer it linearly projects the conditioning features (i.e. z for NeRF-VAE, or fi for LASER-NV) and sums it to the activations. The MLP is composed of two parts: the first has 4 layer of 256 hidden units each and outputs the densities, and the second part has an additional 4 layers and outputs the colours. All layers use a swish activation function.
We now describe the final component of the scene functions of all
The circular encoding function y is given by
rp)) (sin(2 (min()))))))) (20 sin (2 xp). (2 where different constants Lmin and Lmax for positions and directions and for each dataset (listed in/Appendix E}.
(10)
A.5 NeRF-VAE AND CONDITIONAL NeRE-VAE
Our implementation NeRF- VAE follows the same overall architecture that of Kosiorek et al. (2021). but we use the image encoder described in Fig. 6 6jnstted. The approximate posterior is computed a diagonal Gaussian distribution, the means and log-variances of which are given by an MLP e as a function of the pooled feature maps H (pooled with a simple averaging over feature maps). The ML. has layers of 256 hidden units and ReLu activation function. The scene function is composed of an MLP that processes the latent vector z (one linear layer of 256 units, followed by ReLu non-linearity), followed by the output MLP described in Appendixx A.4
A.6 BACKGROUND SCENE FUNCTION
The volumetric rendering integral of a ray r is given by
c(T)NeRF= = 1.' T(t))(((t))(((())))) d) dt, with T(t) = exp (-/"" (r((s))
(1I)
If the ray is infinite (camera's 's far plane t, + oo) the colour weights (rt))()(t)) would always sum to one. For a truncated ray (finite ty), this happens only if the ray passes through solid surface. This heuristic allows to model distant backgrounds by putting them at the end of the integration range, but doesn't work the camera position changes significantly compared to the ray range Iy - b, not normalized, the final weight T(ty) tell us about the probability of light hitting particle beyond the ray.
In that case, model the background an infinite dome around the scene and model it with light-field that depends only on the camera direction. That is, we compute the colour as
c(r)
c((r NERE + T(ty)s
(12)
16

---

Under review as a conference paper at ICLR 2023
For the City dataset, in order to model the sky and sun, fog (2, d4) is learned neural network. We use a cross- attentioo module with 2 layers to attend to Z. For NeRF-VAE and cond. NeRF-VAE we instead use 2 layered MLP with ReLu activations. For the ShapeNet fig is constant colour (white).
We address the challenging backgrounds of MSN-Hard with a more expressive attentive background scene function that conditions both the latents 2 as well context image features this is in a similar fashion to how the NeRF scene function of LASER-NV integrates its latents with attention to local context features. Given an encoded ray direction 7(d:) and context feature maps U H", cross-attention module attends to the latents: h fbg (7(()))))) 2), and the output used to attend to the set of context features U H with a second cross- cross-attention module h - Jhy (h U H Finally, an MLP Sbg computes the background colour by taking as input the obtained background features and additionally conditions on the encoded directions by adding linear projection of the directions to the activations of every layer. This MLP is composed of 2 layers of 512 hidden units each with a ReLu nonlincarity, and outputs the RGB background colours using a sigmoid nonlinearity.
B CITY DATASET
Figure 8: Examples of generated cities.
Figure 9: Examples of different viewpoints (column) for each scene (row).
The aim of the City dataset is to allow generating visually complex scenes with large variability in its structure and appearance. Each generated city consists of large area spanning approximately 270m x 270m, which consists of 2x2 grid of blocks, where each block is composed of 4 lots. A lot can be a building or green patch (park). The city 2x2 grid is surrounded by an outer ring of buildings. Roads separate all blocks in the scene. Each building is randomly generated with variation in size, appearance (textures, materials, colours), architectural styles (residential, industrial, commercial). Fig. 8 shows examples of generated city scenes. Once city is built, specify range of points of interest, iec. road intersections and parks and use each these points as the starting placement for a scene (data point) of our dataset. Each scene comprised of posed images rendered from randomly
17

---

Under review as a conference paper at ICLR 2023
chosen viewpoints the point of interest. More specifically, render 8 randomly chosen nearby points, and 4 random viewpoints for each point, making a total of 32 generated views per scene. Each generated view is annotated with the ground-truth render, camera parameters and depth maps. Our generated training dataset contains 100,000 scenes generated in total from approximately 2,600 uniquely generated cities. We test 3,200 scenes generated in total from 10 uniquely generated cities.
C ADDITIONAL VISUALIZATIONS
Capturing detail different scale In Fig. 10 we can see a higher resolution how our model is expressive enough to capture texture and geometry details at different scales (e.g. far-away buildings and nearby objects such as the lamp post and the colonnade).
Ground-truhh
Recunstraction
Inferred depth
Figure 10: Higher resolution reconstructions of LASER-NV reveal detail captured at different scales (e.g. lamp posts, and nearby buildings vs buildings at large distance).
Novel view synthesis with ShapeNet NMR In Fig. 11 we compare predictions for number of objects that reveal how NeRF-Ve VAE produces blurry reconstructions compared MVC-NeRF and LASER-NV. When an image does not reveal the full shape of an object, shown in Fig. 12 LASER- NV is able generate multiple plausible variations (in contrast deterministic which generates single blurry prediction).
Input
MVC-NeRE
NeRF VAE
Laser-NV
Input
GT
MVC-NeRF
NeRF VAE
Laser-NV
Figure 11: Samples of category-agnostic ShapeNet for ablations of LASER-NV. MVC-NeRF and LASER-NV both result in high-fidelity predictions, while the NeRF-VAE's outputs are blurrier.
18

---

Under review as a conference paper at ICLR 2023
Laser-NV
MVC-NeRF
Input
GT
Sample
Depths
Sample 2
Depihs 2
Sample 3
Depths
Prediction
Depihs
Figure 12: When predicting novel view from an ambiguous input, LASER-NV generates plausible variations, whereas MVC-NeRF results single blurry prediction.
MSN-Hard predictions Figure 13 shows LLAER--V'SS predictions on MSr-Hard including depth maps estimated from NeRF's densities.
GT
Prediction
Depth
Figure 13: Representative examples of LASER-NV predictions on MSN-Hard,
D ADDITIONAL RESULTS
Data efficiency We evaluate how data hungry LASER-NV is compared to NeRF-VAE and MVC- NeRF. (Figure 14 we report the reconstruction colour log- probaabbbyl estimated on validation set of the City dataset. For LASER-NV, we observe little degradation between using 50K scenes and 12.5K unique scenes. For DeRF-VAE we notice a larger drop when reducing the size from 25K to 12.5 scenes, showing clearer signs of overfitting. Finally, since MVC-NeRF severely underfits the City dataset for all training set sizes, we do not see any clear performance drop.
understand how different latent sizes at test time impact performance. Figure 15) shows reconstruction performance for different values of K test time, for various values of used K during training. As we can see, LASER-NV does not significantly leverage larger latent set size than trained with. As expected, we do see performance degrading when using smaller sets at test time, and this effect becomes stronger for models trained with larger capacity.
Test-Time Scaling of the Latent Set We further
Training LASER-NV without ground-truth depth We train all our models using ground-truth depth on the City dataset. This reduces memory and computation requirements during training. We
19

---

Under review as a conference paper at ICLR 2023
Reconstruction log likelihood
Laser-NV 50k scenes
3.75
Laser-NV 25k scenes
3.50
Laser-NV
12.5k scenes
NeRF
50k scenes
Cand NeRF VAE
3.00
Cond NeAF VAE 12.5k scenes
2.75
MYC-NeRF 50k scenes
2.50
MVC NeRF
25k scenes
2.25
MVC.NeRF
12.5k scenes
2.0
0.2
0.4
0.6
0.8
1.0
step
le6
Figure 14: Validation reconstruction performance of different method as function of number of scenes throughout training.
Trained with K 6
Trained with K = 12
Trained with 24
Trained with K 48
PSNR
24
24
22
22
22
24
16
24
32
0.9
0..9
0.9
0.9
5S1
0.8
0.8
0.8
0.8
1
16
24
32
8
16
32
48
64
24
32
48
16
24
64
Test latent size
Test latent set sibc
Test latent s2t size K
Test latont set size
Figure 15: Reconstruction performance of LASER-NV as function of the latent set size used at test time for different models trained with increasing sizes K.
note, however, that it also possible to train LASER-NV without using ground truth depths. Once trained, find that both the model trained with depths and the model trained without achieves slightly higher average colour log-likelihood (3.87) compared to LASER-NV (of 3,83) (importance- weighted estimate using 10 samples). We hypothesize that this is because the model trained without depths is trained solely using the rendering method used at test time.
As any NeRF-related method, LASER-NO NV can output estimated depth maps easily. We show example reconstructions and inferred depth maps of a trained LASER-NV on the City in Fig. 16 Note how the model is capable of capturing fine detail in textures and shapes (e.g. lamp post).
Computational efficiency of LASER-NV vs NeRF-VAE For model conditioned inference (inferring the latent distribution) and rendering of single image for LASER-NT (with 24 slots) take 0.39 GFLOPs and 1.80 GFLOPs respectively. NeRF-VAR VAF instead takes 0.37 GFLOPs and 0. GFLOPs. The main reason for more expensive rendering in LASER-NV is the introduction of PixelNeRF-like conditioning. model with a flow posterior but without such conditioning uses 0.47 GFLOPs to render a single image. Attention, however expensive, is evaluated only once per scene (to compute the prior/posterior). The PixeINeRF features are cheap per point, but they are evaluated for every 3D point used in rendering, therefore dominating the overall cost. Note that the above numbers do not reflect training latency. At training time we can guide the rendering process by using depths, which allows evaluate the scene function only twice for every pixel. We also
input views.
20

---

Under review as a conference paper at ICLR 2023
Ground-truth (view 1)
Reconstruction (view 1)
Inferred depths (view 1)
Ground-truth (view 2)
Reconstruction (view 2)
Inferred depths (view 2)
Scene
Scene
125
Figure 16: Reconstructions of LASER-NV trained without the use of ground-true -trutt depths.
subsample images and we never reconstruct the whole target view(s). This makes training faster, and means that LASER-NV is only about 4x slower per training iteration than NeRF-VAE. Having said that, we noticed that NeRE-VAE does not benefit from longer training (its performance saturates), while the performance increase from increasing the model size (to equalize the FLOPs) negligible.
Robustness to camera noise We carry out experiment MSN-Hard in which train and test with camera poses with incremental levels of Gaussian noise. We replicate the set up in Sajjadi et al. (202T) and compare their reported results using SRT and PixelNeRF to LASER-NT -V.. Results in Fig. 17 show that while LASER-NO outperforms the other models when there no noise, its performance drops more quickly compared to SRT, in similar fashion to PexeINeRF. However, LASER-NV is more robust than PixeINeRF for all levels of noise. We believe that the use of NeRF and local features makes these models less robust noisy cameras. That said, there recent methods that allow mitigating noise sensitivity with NeRF (Lin et al. 2021 that can be incorporated.
24
SRT
22
Laser-NV*
PSNR
20
PixeINeRF
18
16
14
0.00
0.02
0.04
0.06
0.08
0.10
Camera Noise
Figure 17: PSNR of models trained and evaluated under cameras with different levels of noise. The PSNR values of LASER-NV* are preliminary training values models that have not trained to convergence (340K steps out of 1000K).
TRAINING DETAILS
Hierarchical rendering Instead of optimizing separate sets parameters for the coarse and fine rendering passes, we find that it suffices to only keep separate parameters of Fg. This allows sharing parameters (and reduce computation) when querying the scene function on the coarse sampled points with respect to latent attention and multi-view geometry components of models.
Training with depth (City) Because we have RGB-D data in the City dataset, we can use depth as supervised signal following the method in Stelzner al. (2021). This method involves replacing the
21

---

Under review as a conference paper at ICLR 2023
pixel colour likelihood, for given ray r. with the ray colour and depth likelihood p(t(r), c(r) | 2). where t(r) is the ray's ground-truth depth. This alternative likelihood requires only two scene function evaluations per ray. In practice, likelihood terms to the loss (each weighted by 0.5). At every training set, similar to how NeRF-VAE is trained, the likelihood terms estimated by only reconstructing subset of rays (i.e. pixels) of all the target images. We find that using 64 rays for estimating the (expensive) image log-likelihooe and 512 to estimate the ray colour and depth likelihood term, works well in the City. Additionally, we restrict output densities to be between 0 and 10 by using weighted sigmoid in the output of the scene function.
find that get the best quality renders when combining both log-
Additional regularization The loss function for ShapeNet experiments includes density regular- ization term based on the Ll-norm of the densities, with a weight of 0.01. For experiments, we scale the model parameter updates (Pascanu et al. 2013 Sec. 3.2), such that loss gradient norms have a maximum value for improved stability.
We use the following selected for each dataset based on preliminary exploration using the training dataset and validation split.
Hyper-parameterr
across all models. Some of which were
Hyperparameter
City
Shapenet NMR
MSN-Hard
Learning rate
0.0003
0.0002
0.0001
Batch size
96
96
32
# of rays per scene
512
512
384
of context views
2
# of target views
8
23
3
# of posterior context views
10
4
8
# of coarse points
256
32
64
# of fine points
64
64
32
Camera Exear
0.05
0.1
0.01
Camera for
270
3.7
19
Positions circular encoding Lmin
-8
-2
-5
Positions circular encoding Lmux Directions circular encoding Lmin
8
8
10
0
0
0
Directions circular encoding Lmax
4
4
8
LASER-NV set size K
24
8
32
LASER-NV latent vector dimensionality
128
128
128
Table 4: Common hyper-parameters
22


Reviews:
[{"review_id": "y5N4BpjgYPj", "reviewer": "Reviewer_b7vH", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a conditional generative model based on NeRF-VAE. The differences are that the proposed methods use a set of latent vectors to encode the scenes instead of one and it incorporates multi-view geometry local features for better feature learning. It also applies normalizing flow to model conditional prior and conditional posterior. Experiments show that the method outperforms baselines on both reconstructions of observed views and predictions of unobserved novel views.", "summary_of_the_review": " I am concerned about the novelty of this paper in spite of the performance improvement. I think this paper is below the bar of ICLR.", "strength_and_weaknesses": "Strengths:\n- The proposed latent set representation is useful. According to the experiments, the latent set representation has a good scaling ability.\n- The proposed methods have a better performance in predicting unobserved views compared to NeRF-VAE.\n\nWeaknesses:\n- The current City dataset is relatively simple. Can the method work on more complex scenes, such as multiple ShapeNet objects (like PixelNeRF) or real dataset?\n- I am concerned about the technical novelty of this paper. The idea of set latent scene representation is actually not a new thing, like Scene Representation Transformer, CVPR 2022. Moreover, aggregating multi-view local features is a standard technique in computer vision, and has been widely used in many existing works, including PixelNeRF, IBRNet etc. The proposed frameworks seem to be a combination of existing techniques in spite of the performance improvement. I am afraid the novelty is below the bar of ICLR.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "qF3QgBTiaZ", "reviewer": "Reviewer_kWNb", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a latent set representation for 3D generative modeling. The proposed method includes a normalizing flow based context encoder, a NeRF scene function, and a VAE based renderer. By using a set of latent variables rather than one latent variable, the proposed method can make multiple different predictions for unobserved regions in novel view synthesis. The experiments show that the proposed method outperforms baseline methods including MVC-NeRF and NeRF VAE.", "summary_of_the_review": "The paper is novel. However, the experiments on real data are missing. So it is unclear how sensitive it is to camera poses. Thus I'm giving the borderline reject rating. I would like to change my rating if my concern listed above is well addressed.", "strength_and_weaknesses": "+ The idea of using the latent set representation is novel.\n\n+ The design of multiple latent variables makes it possible to make multiple different predictions in unobserved regions.\n\n+ LASER outperforms MVC-NeRF and NeRF VAE in experimental analysis. LASER produces a sharper result.\n\n- The experiments are done on synthetic data only, where the camera pose is accurate. However, in real cases, there is always error in camera poses. It would be good if the author could show the result on real data (like in the original NeRF paper) so that we could understand its robustness to pose errors.\n\n- The explanation on obtaining \"z\" for rendering is unclear. It would be good if mathematical equations are provided in Sec. 3.3 \"Querying the Latent Set\" so that the readers could better understand how \"z\" is computed and where the randomness comes from.\n\n- Although the results are better than baseline results, they are still blurry, e.g., in Fig. 3. It should be analyzed as limitations. However, the paper did not analyze the limitations of the proposed method.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "lBeJXsIxgT", "reviewer": "Reviewer_TTqh", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper tackles the problem of scene generation, generating consistent images of scenes from multiple viewpoints conditioned on few images. The main technical contribution of the work is proposing a set-valued latent representation using normalizing flows, built on top of NeRF-VAE. The proposed model is tested on synthetic city dataset and ShapeNet dataset, while having competitive performance with respect to recent methods.", "summary_of_the_review": "My biggest concern lies in the novelty of the method. I would like to hear from other reviewers before making the final decision.\n\n\n\n---------------------------\nAfter Rebuttal:\nI appreciate the authors for their response.\nI'm in favor of accepting the paper, due to it's multimodality and high performance.\nI have raised my score accordingly.", "strength_and_weaknesses": "**Strength**\n- Performance\n\nBy looking at the figures and tables of the manuscript, the work clearly outperforms NeRF-VAE [1], producing less blurry images from multiple views. However, I would like to ask the authors why SRT [2] was not used as a baseline.\n\n- Good experimental design\n\nI enjoyed showing that increasing the latent vector size of NeRF-VAE does not result in a good performance, thereby the design choice of this work is better. The ablations by showing with performance with/without either latent set and local features make the paper stronger.\n\n\n**Weaknesses**\n- Novelty\n\nMy main concern lies in the novelty of the work. The work seems to be a mixture of NeRF-VAE and SRT. Please see the novelty section below for details.\n\n\n- Intuition behind the set data structure for the scene latents\n\nAlthough it is clear from Table 1, that the scene latents increase the performance of the method, I would like to ask the authors the intuition behind the sets as the choice for the latent data structure. Other works such as Neural Sparse Voxel Fields [3] either have an explicit latent data structure, where the latent code explicitly encodes local information or alot of other generative modeling works show the controlability of latent code by interpolating them. I do not see either of the intuition behind the set representation nor other attributes of set-valued latent representation and would like to hear from the authors the reason for choosing particularly this data structure besides the performance.\n\n- Proposed model requires depth information\n\nAlthough the authors have reported the log-likelihood score of the method in the appendix without depth information, reporting all the scores in Table 1 without depth information will strengthen the paper.\n\n[1] Kosiorek et al. NeRF-VAE: A Geometry Aware 3D Scene Generative Model. ICML, 2021\n\n[2] Sajjadi et al. Scene Representation Transformer: Geometry-Free Novel View Synthesis Through Set-Latent Scene Representations. CVPR, 2022\n\n[3] Liu et al. Neural Sparse Voxel Fields. NeurIPS, 2020\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "FFS-avx2xn", "reviewer": "Reviewer_k538", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed a VAE-NERF model for novel view synthesis:the latent code z is represented by a permutation invariant set and conditional prior p(z|v) as a permutation-invariant normalizing flows; the query k/v pairs are modeled from latent z to decode the scene and images. The results look plausible and unobserved parts can also be generated.", "summary_of_the_review": "The paper proposed a VAE-NERF model for novel view synthesis:\na latent set z is learned by a permutation invariant set and conditional prior p(z|v) as a permutation-invariant normalizing flows; the query k/v pairs are modeled from latent z to decode the scene and images. The results look plausible and unobserved parts can also be generated.\n\nThe idea is interesting and novel, naturally combining normalizing flow to model NERF-VAE. The experimental result looks satisfactory in synthetic single objects and urban scenes.", "strength_and_weaknesses": "+ The  permutation-invariant set normalizing flow is an interesting idea, which naturally enables representation learning with transformers.\n\nSome questions:\nIs there any physical meaning for latent codes z? are they different objects? or different position?\nThere are existing work combining generative modeling (GAN/VAE/diffusion-process) with NERF, any insight why the proposed method is better?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject