Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a generative model for unsupervised object-centric 3D scene understanding. The proposed model is claimed as "causal" as it is baked in with a multi-object Neural Radiance Field with explicit latent variables for color, shape, and object positions. This model is trained with ELBO in Variational Bayes. To demonstrate that the introduction of the "causal" structure can help generalize the inference of latent variables to OOD scenes, i.e. scenes not covered in the support set of the training data, the authors further propose an MCMC sampling method for test time only. The proposed model shows superior performance in GQN dataset where objects are iid and ARROW datasets where there is a specific correlation between objects.  
strengths_and_weaknesses: \+ This paper is very well written. I liked the narrative in which the authors first describe the representation and the generative models with probabilistic language, then introduce the inductive bias of NeRF, and then move on to introduce training and inference. This makes the method section very easy to follow. 

\+ The introduction of the scene variable is fairly interesting. Existing methods for object disentanglement mostly assume iid prior over objects. The hierarchical design of the generative model echoes the "conditional independence" in causality, making the assumption in models more generic. 

\+ Though couples of existing works use NeRF as a scene generator, the explicit modeling of the latent variable and the MCMC sampling in OOD scenes are novel contributions. In particular, the observation that the variational encoder from Variational Bayes training is the critical bottleneck for OOD generalization is very sharp. 

\+ As far as I know, generalizing to novel viewpoints is something not covered by the existing literature on multi-object representation learning. 



\- Though I liked the idea of using MCMC sampling for inference in OOD images, I have some concerns about its concrete realization. First, to account for the distribution shift in the latent causal factors, the proposed method needs to manually replace the learned prior of latent variables with uninformative distributions. Then is it fair to compare with other existing models since here the model is actually informed of which causal factor is intervened in the distribution shift? Second, the combination of Langevin dynamics and Metropolis-Hastings is interesting, but why the proposal of each object can be "considered" independently? 

\- The comparison with the baselines is not completely fair. First, Slot Attention model does not use either the viewpoint information or multiple views of the same scene. Maybe authors should try a variant of it imitating how GQN extends VAE? I am also curious why the trained Slot Attention model does not generalize to objects with OOD number of objects since it should be successful according to the original paper. Second, NeRF-VAE is not encouraged to have structured latent representation, so we kind of know a priori it will fail the composition test, maybe the authors should try some disentangled VAEs such as beta-VAE? Third, the correlation in the ARROW dataset that the arrow object is always pointing to a special object is known a priori and the proposed model seems to perform well in this dataset by incorporating this knowledge into the model. In reality, we don't really know what kind of correlation there are among objects, how would the proposed model still be generic? 
questions: Apart from the questions I raised in the weakness part above, I would appreciate it if the authors would like to answer the following:

1. Since the way of inference in OOD images is different in the proposed model and existing models (MCMC vs Variational Encoder), it is unclear which component between the explicit generative model and the MCMC inference plays a more critical role in the success of OOD generalization. Maybe the authors would like to apply their MCMC samplers to the trained decoder of Slot Attention, uORF, and NeRF-VAE? I believe this ablation can be beneficial for readers who are interested in OOD generalization. 
2. If time allows, authors may also want to compare these models in datasets where there are more authentic textures, for example, the CLEVR-TEX dataset? 
3. Authors claimed to use uniform distribution to replace the learned prior when generalizing to OOD images, how is that implemented and what is the rationale here? More specifically, what is the range of the support space, and why so? 

I will be happy to raise my rating if authors would like to resolve some of my concerns. 

--------------------

Post discussion: Thanks authors for providing results on experiments I requested, I have raised my rating. 
limitations: Apart from the limitation authors listed in Sec 4.5, I believe the authors may also want to talk about how statistical assumptions in the proposal distribution in MH sampling may affect the OOD inference, And how can the assumption of knowing there is indeed a distribution shift can be fulfilled in reality.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors proposed a VAE-based NERF-like object-centric framework to model 3D-scenes. The model represents objects as separate NERFs. Moreover, authors make architectural choices that reflect causal independent mechanisms, so this enables OOD generalization. Plus, a novel MCMC inference scheme is proposed to infer out-of-distribution scenes. The model outperforms other models in OOD setting and performs competitively in In-distribution set-up.
strengths_and_weaknesses: Strengths

- the idea and necessity of a novel MCMC inference scheme are intuitive and well-explained
- the model outperforms baselines in OOD setting and performs competitively in In-distribution set-up.
- the model is OC and generative (that is novel as far as I understand) compare to its OC counterpart (uORF), so I believe it’s easier to edit a scene object-wise.
- authors make architectural choices to reflect the causal independent mechanisms.

Weaknesses

- there is no ablation study on model design -- this is my main concern
- the architectural choices that make the model more causal are not explicitly stated in one place in the main text, so it’s hard to follow what exactly these choices are. And there is no ablation study on these choices either. 

Overall, in my opinion, the authors implemented the right priors to make OOD generalization possible and the paper seems novel to me. 

I am willing to increase my score if a proper ablation study will be performed.

questions: Ablation study questions 

Main questions
1) Can you explicitly state all architectural choices you made to enable independent mechanisms and perform an ablation study on these choices? 
2) Do you need a two-step generation? The general latent layout of the scene is generated firstly and then latents of objects are generated based on this layout. Can you ablate it? 
3) Did you apply a new MCMC procedure to your baselines (slot-attn, uORF, NERF-VAE) for OOD setup? If not -- why? 
4) Can you ablate your MCMC procedure for OOD setup and show what scores the model obtains when one uses an encoder to obtain latents?

Small issues
1) I did not understand figure 1, can you maybe elaborate on this picture a little bit more.
2) I suggest to the author to include a graphical model in the main text to increase readability. 

limitations: -
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a generative model of images that factors appearance of the image into background and multiple objects — each object is associated with its own NERF based appearance model drawn from a global prior and together can be used to forward render an associated image in a differentiable way.  Latent object appearance models are inferred using a VAE-like framework with neural-net encoders.

Finally by replacing certain learned priors by uniform priors, the authors show that their models are able to work in OOD settings quite accurately.  Results in this paper are all on synthetic baselines.


strengths_and_weaknesses: Strengths:

A full instance-level generative model of image appearance is arguably one of the “holy grails” of computer vision — and the fact that this paper takes a step in this very ambitious direction and achieves reasonably good results (on synthetic data) is a strength.  

There are two possible families of comparison — NERF based models which generally do not have a probabilistic object based decomposition of the scene, and 2d models of image composition which do capture object based decompositions, but are not able to learn from multi-view data in the same way. Compared to other works along the same lines (e.g. slot attention), the authors also show strong performance in their model’s ability to generalize to out-of-distribution data.  

Another strong point of the paper is the attention paid to reliable optimization.  For example, the authors find that modeling positions using a categorical variable helps (compared, e.g, to using naive spatial transformer models which would model pose using a continuous variable).  The idea is similar to the way that Faster R-CNN works for object detection --- use a discrete grid to predict at an "anchor grid" of positions, then use ROIAlign (which is basically an axis aligned STN) to crop features.


Weaknesses:

There are some major things that are not modeled in this work but would be important in the long run — these include orientations (or a more general space of transformations beyond point-to-point translation) and semantics (there is not a way that the concept of “chair” can be learned by training on many images).  Also, even though images can be “out-of-distribution” in the sense of having different numbers of objects, being placed in different locations or having novel compositions, I don’t believe that generalizing to novel types of objects is handled well in this approach.  

Another weakness of the approach (mostly having to do with the fact that it requires MCMC) is the running time, which the authors acknowledge to be a limitation.  I would recommend mentioning more specifics about time requirements.


questions: * I cannot figure out how are shape and color separated in this model — it’s possible that I am missing something, but shape and color variables seem to be treated symmetrically throughout, so what forces one variable to map to shape and the other to map to color?
* Related to this question, it would be useful to provide further factorization in Equation 2 to explain the joint distribution of shape, color and position of a single given object.
* I wonder if there is a principled way to decide when it is okay to replace a learned prior with a uniform distribution.  In other words, when is it okay to extrapolate from the dataset?  Clearly the answer cannot be “all the time” — could the authors shed some light on how to make this decision appropriately?

limitations: yes.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents a slotted generative model with a NeRF rendering decoder. Each slot parameterizes a scene function and the decoder mixes the scene functions while rendering (summing densities and averaging colors). The model is trained as a 2-stage VAE - first training the slotted model VAE and then a "global" VAE which models the marginal posterior distribution of the first model. The slot latents are factorized into "appearance" and "position" and the latter is one-hot encoded (relaxed with Gumbel softmax) and used as a convolution kernel to "shift" the scene function in scene space while rendering.
The model receives one or more input views of a scene and is trained as a VAE, much like NeRF-VAE but with slots.
An inference procedure is also proposed to work in the case of OOD inputs where the amortized encoder is not expected to work.
The method is demonstrated to work nicely on two synthetic datasets - GQN and ARROW.
There are some experiments that show that the model is able to learn factorized representation and these can be manipulated.
strengths_and_weaknesses: Originality:
  While mostly an amalgamation of existing methods this is an interesting model and the fact that it works, even on synthetic data, is non-trivial. The one-hot encoding of position is a nice touch and I suspect makes the model work better. 
  I am, however, not convinced by the "Causality" claim of the model - I am not a causality person, but I feel this is a stretch - this model is not more "causal" than IODINE, for example - it does, maybe, reflect the underlying generative process better (through cameras for example) but I feel this is a bit of stretch (see more below)

Quality:
  The paper is quite thorough, and all in all quite good. I feel the choice of comparing only to NeRF based baselines is a weakness - much of the rendering process, especially in simple scenes like these, can be learned well by 2D models (up to NVS) and these should have been compared to (for example, GQN and IODINE). I would want to see analysis of the global latent, and how it affects things (the authors do mention this under "limitations")

Clarity:
  The paper is written nicely and quite clearly, but I feel several elements are not described well - especially the role and structure of the global latent, see below. 

Significance:
  All in all I feel this is an important contribution to the community with some non-trivial advances.
questions: * How is the conditional model p(z_s|z_g) structured? I looked at the supplementary material and it wasn't clear - how is the mapping for a single latent to multiple slots done? is it just an MLP which scales with the number of slots?

* Follow-up to the above: can you change the number of objects without affecting the number parameters in the model? Consequently - can you instantiate a model with a different number of slots than used in training?

* What prevents the object location to be learned in a non-canonical position and compensate with the appropriate shift through the inferred position? There's nothing constraining the object to be learned around the origin.

* Have you tried the actual learned encoder on OOD data instead of using MCMC inference? I know this is claimed not to work in the paper but I didn't see an experiment showing this.
* Follow-up to the above: have you tried amortized iterative inference for evaluation of OOD? (IODINE works on unseen combinations of color and shape, for example)
limitations: I think the authors addressed the model limitations well in the paper - the above question directly refer to some of the missing points I feel.
review_title: null

Meta-review: metareview: This paper proposes a NERF-based object-centric VAE generative model, which it argues is a more "causal" generative model than prior attempts. While the approach is somewhat elaborate, it is described well, and involves a novel and quite well-motivated combination of several previously proposed components.

The MCMC sampler that improves the VAE encoder for out-of-distribution settings is an interesting additional contribution beyond the generative model, even though it is naturally computationally inefficient.

While the experimental settings are somewhat toy (all synthetic data), the proposed method is shown to enjoy substantial gains over some baselines representative of the closest related work, and the new baselines and ablation studies added during the response period are helpful. 

After quite an engaging response period, the four thoughtful reviewers all agreed that the merits outweigh the shortcomings. I concur and recommend acceptance. 
confidence: Certain
Decision: Accept