Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper tackles the problem of scene generation, generating consistent images of scenes from multiple viewpoints conditioned on few images. The main technical contribution of the work is proposing a set-valued latent representation using normalizing flows, built on top of NeRF-VAE. The proposed model is tested on synthetic city dataset and ShapeNet dataset, while having competitive performance with respect to recent methods.
strength_and_weaknesses: **Strength**
- Performance

By looking at the figures and tables of the manuscript, the work clearly outperforms NeRF-VAE [1], producing less blurry images from multiple views. However, I would like to ask the authors why SRT [2] was not used as a baseline.

- Good experimental design

I enjoyed showing that increasing the latent vector size of NeRF-VAE does not result in a good performance, thereby the design choice of this work is better. The ablations by showing with performance with/without either latent set and local features make the paper stronger.


**Weaknesses**
- Novelty

My main concern lies in the novelty of the work. The work seems to be a mixture of NeRF-VAE and SRT. Please see the novelty section below for details.


- Intuition behind the set data structure for the scene latents

Although it is clear from Table 1, that the scene latents increase the performance of the method, I would like to ask the authors the intuition behind the sets as the choice for the latent data structure. Other works such as Neural Sparse Voxel Fields [3] either have an explicit latent data structure, where the latent code explicitly encodes local information or alot of other generative modeling works show the controlability of latent code by interpolating them. I do not see either of the intuition behind the set representation nor other attributes of set-valued latent representation and would like to hear from the authors the reason for choosing particularly this data structure besides the performance.

- Proposed model requires depth information

Although the authors have reported the log-likelihood score of the method in the appendix without depth information, reporting all the scores in Table 1 without depth information will strengthen the paper.

[1] Kosiorek et al. NeRF-VAE: A Geometry Aware 3D Scene Generative Model. ICML, 2021

[2] Sajjadi et al. Scene Representation Transformer: Geometry-Free Novel View Synthesis Through Set-Latent Scene Representations. CVPR, 2022

[3] Liu et al. Neural Sparse Voxel Fields. NeurIPS, 2020

clarity,_quality,_novelty_and_reproducibility: - Clarity

The paper is clearly written and easy to understand and the figures make the paper easier to understand.


- Novelty

The work seems to be a mixture of NeRF-VAE and SRT which is my biggest concern. Although the taking advantages of other methods does not necessarily reduce the novelty of the work, using the latent code of NeRF-VAE, thereby having multimodality and using set-representation latent code of SRT is straightforward with no huge novelty in the problem setting.

- Quality

The experimental design of the paper is well, although some more ablation studies could help.

- Reproducibility

Code is not uploaded. Wil the code and city datasets be uploaded? I think the city dataset will be used alot in the community.

summary_of_the_review: My biggest concern lies in the novelty of the method. I would like to hear from other reviewers before making the final decision.



---------------------------
After Rebuttal:
I appreciate the authors for their response.
I'm in favor of accepting the paper, due to it's multimodality and high performance.
I have raised my score accordingly.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposed a VAE-NERF model for novel view synthesis:the latent code z is represented by a permutation invariant set and conditional prior p(z|v) as a permutation-invariant normalizing flows; the query k/v pairs are modeled from latent z to decode the scene and images. The results look plausible and unobserved parts can also be generated.
strength_and_weaknesses: + The  permutation-invariant set normalizing flow is an interesting idea, which naturally enables representation learning with transformers.

Some questions:
Is there any physical meaning for latent codes z? are they different objects? or different position?
There are existing work combining generative modeling (GAN/VAE/diffusion-process) with NERF, any insight why the proposed method is better?
clarity,_quality,_novelty_and_reproducibility: The paper is clear and easy to follow.  Combining permutation-invariant normalizing-flow and transformer is novel and interesting.

Transformer is popular and also used in other novel view synthesis papers such as IBRNet.
There are existing work combining generative modeling (GAN/VAE/diffusion-process) with NERF, such as NERF-VAE, GRAF, SGN, ...
Representing a scene by piecewise implicit functions, grids or octrees has also been proposed before, which is related to the proposed set-based reprsentation.
summary_of_the_review: The paper proposed a VAE-NERF model for novel view synthesis:
a latent set z is learned by a permutation invariant set and conditional prior p(z|v) as a permutation-invariant normalizing flows; the query k/v pairs are modeled from latent z to decode the scene and images. The results look plausible and unobserved parts can also be generated.

The idea is interesting and novel, naturally combining normalizing flow to model NERF-VAE. The experimental result looks satisfactory in synthetic single objects and urban scenes.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents a conditional generative model based on NeRF-VAE. The differences are that the proposed methods use a set of latent vectors to encode the scenes instead of one and it incorporates multi-view geometry local features for better feature learning. It also applies normalizing flow to model conditional prior and conditional posterior. Experiments show that the method outperforms baselines on both reconstructions of observed views and predictions of unobserved novel views.
strength_and_weaknesses: Strengths:
- The proposed latent set representation is useful. According to the experiments, the latent set representation has a good scaling ability.
- The proposed methods have a better performance in predicting unobserved views compared to NeRF-VAE.

Weaknesses:
- The current City dataset is relatively simple. Can the method work on more complex scenes, such as multiple ShapeNet objects (like PixelNeRF) or real dataset?
- I am concerned about the technical novelty of this paper. The idea of set latent scene representation is actually not a new thing, like Scene Representation Transformer, CVPR 2022. Moreover, aggregating multi-view local features is a standard technique in computer vision, and has been widely used in many existing works, including PixelNeRF, IBRNet etc. The proposed frameworks seem to be a combination of existing techniques in spite of the performance improvement. I am afraid the novelty is below the bar of ICLR.

clarity,_quality,_novelty_and_reproducibility: The paper is clear and the experiments are detailed. I think it is reproducible.
summary_of_the_review:  I am concerned about the novelty of this paper in spite of the performance improvement. I think this paper is below the bar of ICLR.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a latent set representation for 3D generative modeling. The proposed method includes a normalizing flow based context encoder, a NeRF scene function, and a VAE based renderer. By using a set of latent variables rather than one latent variable, the proposed method can make multiple different predictions for unobserved regions in novel view synthesis. The experiments show that the proposed method outperforms baseline methods including MVC-NeRF and NeRF VAE.
strength_and_weaknesses: + The idea of using the latent set representation is novel.

+ The design of multiple latent variables makes it possible to make multiple different predictions in unobserved regions.

+ LASER outperforms MVC-NeRF and NeRF VAE in experimental analysis. LASER produces a sharper result.

- The experiments are done on synthetic data only, where the camera pose is accurate. However, in real cases, there is always error in camera poses. It would be good if the author could show the result on real data (like in the original NeRF paper) so that we could understand its robustness to pose errors.

- The explanation on obtaining "z" for rendering is unclear. It would be good if mathematical equations are provided in Sec. 3.3 "Querying the Latent Set" so that the readers could better understand how "z" is computed and where the randomness comes from.

- Although the results are better than baseline results, they are still blurry, e.g., in Fig. 3. It should be analyzed as limitations. However, the paper did not analyze the limitations of the proposed method.
clarity,_quality,_novelty_and_reproducibility: Clarity: As mentioned above, some mathematical equations should be included to make it clear.

Quality: The result outperforms the baseline methods on synthetic data. However, it is unclear how it performs on real data.

Novelty: The latent set representation is novel.

Reproducibility: It could be reproducible if the clarity problem is resolved.
summary_of_the_review: The paper is novel. However, the experiments on real data are missing. So it is unclear how sensitive it is to camera poses. Thus I'm giving the borderline reject rating. I would like to change my rating if my concern listed above is well addressed.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a  set-valued latent representation using normalizing flows  in a multi view transformer prediction model. By using a latent set representation instead of a single vector as Nerf-Vaes, the results are way better, and can generate multi-modal predictions. The reviewers raised concerns regarding comparisons with SRT which also uses a set-valued latent representation, where the latent are in 1-to-1 correspondence with patches in the multi-view images, and the authors added the requested experiments. The reviewers further asked about experiments on real world data such as in the StreetView benchmark, but the authors said that dynamic entities are part of the dataset and they cannot use it for this purpose. Overall, though reviewers welcomed the additional experiments and explanations, would still like to see application of the model in this real world dataset, even if the results are not the state of the art, to illuminate on the generalization ability of the proposed set representation. For this reason, the authors are encouraged to add the requested experiments and submit to a future venue. 
justification_for_why_not_higher_score: Lack of real world results.
Decision: Reject