Unsupervised Causal Generative Understanding of Images
Authors: Titas Anciukevičius
Patrick Fox-Roberts
Edward Rosten
Paul Henderson
Venue: NeurIPS 2022 Accept (2022)
Keywords: unsupervised learning
generative models
object centric models
out-of-distribution generalization
domain shift
causality

Abstract: We present a novel framework for unsupervised object-centric 3D scene understanding that generalizes robustly to out-of-distribution images. To achieve this, we design a causal generative model reflecting the physical process by which an image is produced, when a camera captures a scene containing multiple objects. This model is trained to reconstruct multi-view images via a latent representation describing the shapes, colours and positions of the 3D objects they show. It explicitly represents object instances as separate neural radiance fields, placed into a 3D scene. We then propose an inference algorithm that can infer this latent representation given a single out-of-distribution image as input -- even when it shows an unseen combination of components, unseen spatial compositions or a radically new viewpoint. We conduct extensive experiments applying our approach to test datasets that have zero probability under the training distribution. These show that it accurately reconstructs a scene's geometry, segments objects and infers their positions, despite not receiving any supervision. Our approach significantly out-performs baselines that do not capture the true causal image generation process.

Unsupervised Causal Generative Understanding of Images
Titas Anciukevičius
Snap Inc., University of Edinburgh titas. www.uueeiiccsuugeeeee com
Patrick Fox-Roberts
Edward Rosten Snap Inc.
Paul Henderson University of Glasgow
Snap Inc.
Abstract
We present novel framework for unsupervised object-centric 3D scene under- standing that generalizes robustly to dist---iisttitutionn images. To achieve this, design causal generative model reflecting the physical process by which an image is produced, when a camera captures a scene containing multiple objects. This model is trained reconstruct multi-view images via latent representation describing the shapes, colours and positions of the 3D objects they show. It ex- plicitly represents object instances as separate neural radiance fields, placed into a 3D scene. We then propose an inference algorithm that can infer this latent representation given single out-of-distribution image input even when it shows an unseen combination of components, unseen spatial compositions or a radically new viewpoint. We conduct extensive experiments applying our approach to test datasets that have zero probability under the training distribution. These show that accurately reconstructs scene's geometry, segments objects and infers their positions, despite not receiving any supervision. Our approach significantly out-performs baselines that do not capture the true causal image generation process.
Introduction
Most machine learning approaches make the assumption that at test time, they are applied to data
drawn from the same distribution as seen during training [14 73 21 33 This means the generalization guarantees of statistical learning theory apply [27]. However, this does not apply to images drawn from a different distribution recent works have shown this for images taken from unfamiliar viewpoints [4] 4 8 shifted by few pixels [7], and showing scenes with an unseen
26 70). It has been suggested that this is because they learn spurious shortcuts (36] to achieve low training loss, but which do not capture the true causal relationship. However, when deployed, machine learning methods often encounter observations drawn from a previously unseen distribution.
composition objects [1] 110 114
36
37
In this work, consider the task of transforming single observed image into detailed represen- tation of the scene it depicts, providing explicit information about its 3D structure such as object locations, shapes and appearances. We focus on the challenging setting where test time, we see images depicting scenes that have zero probability in the training distribution (Fig. T]. We adopt an unsupervised approach to learning, avoiding the need for manual annotation of object masks, 3D positions, and similar we require only a dataset of posed multi-view images.
Our approach to build generative model (Sec. jointly over image pixels and the 3D world they depict, that can be robustly inverted to infer the latent factors that gave rise to an input image [166] [29 (2). If such model is support generalization
-iist-i-iitiittionn (OOD) data, should reflect the underlying causal model of the environment from which the data arose [102] [16
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Segmentation
Depth
NVS
Imput
Depth
Generation
Input
Segmentation
Depth
NVS
Input
Segmentation
Depth
Input
Segmentation
Depth
pooD position
poOD composition
ptrain
POOD
pooD
count
Our causal generative model learns to represent images from one probability distribution pirain but also allows interventions the model represent set of different distributions. This allows both (i) inference object segmentations, depth-maps, and novel views (NVS) for input images drawn from different distributions (e.g. taken from radically different camera viewpoints pooD niews. containing unseen compositions of objects pOOD composition or unseen number of objects Poon)); (ii) generating plausible 3D scenes containing multiple objects ('Generation', top center).
Figure I:
In practice this means the conditional distributions in the model should correspond (as far as possible) to independent mechanisms [113 (66 in the world. Thus, when mechanisms physical processes in the environment change, we expect much of our model to remain applicable; this has been called sparse mechanism shift [116]. For example, even if objects are no longer placed according to the same patterns as at training time, we still expect the appearances object types to remain consistent.
In general it is impossible to recover such causal model purely from observational data [102] [16]. We therefore embed in our model the physical knowledge that the world is composed 3D objects of different shapes, which may appear at different locations, imaged by camera subject to the laws 3D geometry and perspective projection. Each object has separate appearance and position representations, ensuring disentanglement, so the same object can be represented invariantly in different locations. The final image synthesized by volumetric rendering of the latent 3D shapes placed in a 3D scene space according to their latent positions. This structure means the model is compositional: having learnt about one type of object one context, it can also represent and perform inference about similar object in different contexts.
We perform inference on OOD data by intervening on our model 102
i.c. replacing certain conditional distributions (or mechanisms) that are no longer appropriate and then doing posterior inference, To ensure that the inference method itself supports OOD generalization we cannot employ amortized variational inference 665 109), as encoder network is in general not robust to changes of distribution [89 36). Instead, we develop novel Markov chain Monte-Carla (MCMC) inference scheme, that finds posterior samples for a given test image without any encoder network.
Targeting OOD generalization requires important design choices in our model, that differ from other generative models of 3D scenes (97 122 [72.. First, we use a non-learnt rendering mechanism, as this guaranteed to generalise correctly to OOD data (e.g. novel viewpoints): contrasts with prior works that learn the rendering process (e.g. with CNN mapping rendered features to pixels [97] 94)). Second, we use an explicit disentangled representation of the assumed underlying causal variables (e.g. object shapes and positions), which allows performing interventions and counterfactual inference; this contrasts with works that model the scene without disentangling the proper causal variables, such as spatial mixture models 33 [22]. Third,
separate the mechanisms (conditionals) for per-object appearances and for scene layouts, so the latter can be intervened on without affecting the former; this contrasts with methods that have single global decoder [72 94
To evaluate our approach (Sec. 4 we create challenging under the corresponding training distribution, yet share aspects of its structure. We show that our model can generalize unseen numbers of objects, unseen compositions, and radically new camera viewpoints all significantly better than existing works. It successfully localizes objects in the 3D scene, reconstructs their 3D shapes, and hence predicts depth-maps and instance segmentation masks.
datasets that have zero probability

---

To summarise, our core contribution is the first unsupervised framework for inference of explicit object-centric 3D scene representations, that generalizes out-of-distribution scenes. Our secondary contributions are: (i) experimental evidence that current methods do not generalize to images with radically new camera viewpoint, a different number of objects, or an unseen composition of objects; (ii) novel generative model of 3D scenes based on multi-object radiance fields with explicit object positions and volumetric rendering; and (ii) a novel MCMC inference scheme exploiting the structure of our model, that allows inferring 3D scenes from a single OOD image.
2 Related Work
Recently numerous works have observed that modern learning- bbsnd computer vision methods fail to generalise ditoo-ddistribution data 4 [
46 48 [5
71
[I
110
[42
14
36
37
92
87
10
123 26 141 (6) and constructed various henchmarks 68
Others have tried to improve OOD generalisation, the most relevant by aiming to learn relations that are invariant across [04] or that maximize sparsity of interactions [3 38
47
8
42
62
136
131.
training domains [1] 2 10O 74
118
These typically assume that multiple and mainly address the supervised setting (e.g. image classification). Like the latter, our focus is on enabling OOD generalisation, but in contrast to them, we address unsupervised object-centric generative modelling. Moreover, we do not rely on multiple training datasets to discover causal factors, but instead directly incorporate universal knowledge such as separation of the world into objects.
fnferentll-distributee datasets are available during training,
Our work is also connected to the vision inverse graphics paradigm 41 9 63 140
27
90
In this setting, it is assumed that we have access to (maybe parametric) 3D models of objects, and wish find suitable pose and other parameters to explain an input image [33 75 53
112
52
Like our work, these typically use a test-time optimisation; unlike ours, they do not attempt to learn priors on object layout nor shapes instead assuming these are known priori.
Neural implicit scene representations [144] learn continuous representation of a 3D scene from 2D images using neural rendering, either by explicit volumetric rendering [88] 85 132 135] 98 or with CNN post-processing of rendered features [20]. These initial works fitted individual scenes without learning common characteristics, therefore requiring many images input. This
was addressed by sharing models across different scenes [38 125 77 35
103 96
55). allowing
single inference of novel views from one or few images. All these methods model scene entity without decomposing it into individual objects, meaning manipulating the scene (e.g. moving single objects) is not possible. In contrast, [99] divides scene into objects, but requires detailed manual annotations to do so, and does not support inference from few images; (28) relies on weaker supervision in the form of ground-truth object masks. [139 122 discover such decomposition automatically (though with depth supervision for [122). ) however they rely on spatial mixtures to assign each point in space to an object, without any explicit, controllable representation of object positions. This entanglement of latent position and appearance means they are not guaranteed to generalise to OOD combinations of position and appearance [81]. Finally, [43] supports composition of multi-object scenes using neural scattering functions but these must be learnt from multiple views of single objects (a form of weak supervision), with no probabilistic model over appearances nor layouts.
Other approaches extend neural rendering to the generative setting [15 97
94 95
19
25
allowing sampling objects or scenes a priori. However, these do not allow us to perform inference in the OOD setting, as they have components that do not reflect the causal, compositional structure of the world (e.g. a monolithic latent space lacking object-centric representations, or a learnt neural renderer that will not generalise to OOD viewpoints nor compositions). In contrast, there are object-centric generative models that can sample plausible images and perform inference - but only in 2D, without reasoning over latent 3D scene representation, and therefore without supporting 3D tasks such as depth prediction. Some use full-image spatial mixture model [3 93
67 B1 32 60 or alpha
stacking [126] 128]: others model images as composed of smaller patches or sprites 34 57 [5) learning both the distribution of appearances and compositions: others use compositional energy-
based models 22 80 A 3D extension of these latter is proposed by (45) but requires videos for inference at test time and does not support OOD generalisation. Other methods take purely
discriminative approaches to unsupervised segmentation [82 61 [66. In the supplementary, we
provide table comparing the capabilities of our method to closely related works.
3

---

3 Method
Our goal is to infer an explicit object-centric representation of 3D scene from one or multiple images, even when they show scenes lying outside the distribution observed during training. We achieve this by learning a compositional generative causal model jointly over multi-view images and the scenes they depict, as described in Sec. 3.1 For training (Sec. (.2) we use only posed multi-view images (x1, v1):..., (XN, Vx) drawn from training distribution ptrain (x, v) without any annotations such as depth-maps. bounding-boxe or segmentation masks. At test time, input images are drawn from a distribution pOoD xx v) with support disjoint from the training distribution. Performing inference over the generative model using our proposed framework (Sec.3..) yields an explicit object-centric representation, including 3D object shapes, positions and appearances.
3.1 Compositional Generative Causal Model
We model multi-view set of N images (x1... x J as caused by a single 3D scene S being rendered from viewpoints (V1... "v....N), by function C(S, v). We now describe this scene representation, then the generative process by which it is sampled.
Scene representation. The scene S = (Sbg- {(s) S s(00))) ))==1 is composed of a 3D background
component Sbg describing the background's shape and color, and 3D objects indexed = 1 ... O with shape and color described by sPPP and explicit 3D positions s Each s explicitly represents the 3D appearance an object as a neural radiance field (NeRF) (88] in a canonical space (c.g. with the object centered at the origin). The positions s!e0 speciffy objects placement in the global 3D scene space. While most prior work parametrizes 3D object positions as coordinates, we represent them by 1-hot vectors choosing from set of plausible candidate locations to
the center position of the object. As we explain in Sec. [.3] this makes gradient-bart based optimization easier.
Generative process for S. We first sample high-level latent scene embedding 2 ~ (O. I), that will model correlations between objects and learn the typical composition of scene [57 5) The individual object appearances are specified by Gaussian latent variables z; zhape and 2, that respectively encode the shape and color of the 7th object; they are conditioned on z with mean and log- variance given by fully-connected network Ço (zz) with weights 8. [ The position z!!T is specified by a categorical variable, with logits given by Ez()). We similarly introduce latents chape and Zig to encode the shape and color of the background. For brevity, we will write bog
z" = - shape zool shape zool 2....0) The latents z are mapped to the scene S by a function Zbbg Sg. This sets spem equal to 2, and derives the object NeRF representations si from z; shape and 2, as described in the next paragraph. The probability of an image x,, given its camera viewpoint
Vn is then
po (x,, Vv)) = // [x(x C(So(z"), vz),, o²) po(n") (20) pz())) dz
where C(S v) renders the scene described by from viewpoint v, and fN represents a factored Gaussian likelihood over the H W pixels of the image, with fixed standard deviation a. The probability of a composition z² of objects and background in a scene is given by
(z 22) = Po po(2
shape |z (2
col (2ig
x² II i=1
po (z. shope (z²) po (2.00) 2²) po(2?00 pos 2
(2)
We emphasise (-(-) models different distributions for each object and variable; it does not assume the different scene variables constituting z" are I.I.D. Hence, (z Iz²) can model any relationship among object locations, shapes and colors, which is necessary to sample scenes with plausible relationships among scene components. The graphical model is illustrated in supplementary Fig. 9 in
Sec. 7
Rendering the scene S. The rendering process C(S, v) outputs an image x for a camera viewpoint v. given our explicit compositional representation of scene S. Recall S contains a 3D background app pos
component Sōg and a set of object components {s s J for brevity we will identify the
See the supplementary for all network architectures

---

background as component O, with S fixed to the origin. We extend multi- component neural 88 support explicit placement of objects in the 3D scene according to the position variables s spee Specifically, the latent codes z; hapee and z for the th object parametrize
radiance fields (NeRF)
a learnt function fā (q": 2; zmaape, z!"²), that maps points q' in the canonical space of the object to a color E [0, 1| and density a E I We place each object at its 3D position sim by convolving its density and color functions with one-hot location indicator:
Si(a) - 1]] (q') S6 (5(a-10 - q": :4
shape col) dq = (cs(g))
(3)
where q is a position in scene space, and s.(*(q) is an indicator function with a unit impulse if point is chosen as the object center position by the h-hot indicator s : As in [139] we divide the scene space into foreground/backgroum regions and only render the corresponding components in each.
Given the placed object densities oy and colors c, we calculate the color of each pixel in the image by casting a ray r(t) = Xo fd IR from the pixel in direction through a camera at position Xo- summing the contributions from different objects [85] 86
where T(t) = exp (-
(- (-1' (r((t)))
C(S, v)(r) = S. T(t) ((ir(t))..(r(00)))
Continuous relaxation of object placement. To allow gradient- training and inference, our generative process must be differentiable. We therefore relax the categorical position variable to Gumbel-Softnax [54] 84]. This approach ensures we always receive non-zero gradients of the image with respect every possible object position, easing optimisation. This is in contrast models based on spatial transformers 99 134 78). which can get stuck in local minima if the model has a poor initial prediction, as the gradient of pixels wrt position is zero if the predicted and true positions do not overlap. Note that for a discretized representation (e.g. voxels [79] or triplanes [18), the object placement operation Eq. can be efficiently implemented convolution operation in the Fourier domain - exploiting the Fourier transform's 's sifting property [16] 5
3.2 Training
We train our generative model from dataset of images containing K views for each scenes. The model includes three learnable components, with parameters 0: (i) f (q: z shapee zzal that represents a 3D object as a function from position to color and density conditioned on the object appearance embedding;
shape zoglor) that similarly represents the 3D background; (iii) Ço and Ço. that map the global scene latent z to parameters of the object and background latents z". We train the model using autoencoding variational Bayes (65 [09]. The posteriors over Gaussian latent variables are all diagonal Gaussians (parametrized by mean, and log-variance for z²), ). whilst for positions the posterior is Gumbel- -Sottmax (parametrized by logits). We use encoder networks to parametrize these variational posteriors. ency ((xn, vn ( parametrizes (x>" vn (. for efficiency, we pass it only subset of M < K images. It encodes each observed image and its viewpoint (Xn: Vn) independently then sums the results (as (35)) before outputting the posterior parameters; this ensures the encoder is invariant to the ordering of images. enc" (z") parametrizes q(z" (z"), and takes the lower-level latent code z" as input.
(ii) (ho (q; bbg 2bg
For stable training, we adopt two-stage approach. We first train the model to reconstruct X,.... via the object-level latent space z", ignoring the scene-level latent 2n, i.e. maximizing the following loss:
L gg [(x.. Va J
Σ log fx C(Sa(z'), vn). o²)
(S)
After this has converged, we learn the sevee-level latent space by maximizing
L F (291zz)
po [2)]
DxI. [go 2²" || (2())
(6)
We use Adam for optimization above expectations by a single sample. We also further approximate L by rendering only a random subset of pixels per minibatch. More implementation details are in the supplementary material.
64 (-weighting of KI. terms (50). and approximate each of the
5

---

3.3 Inference for out-of-distribution (OOD) images
At test time, we assume images are sampled from a distribution disjoint from the training distribution. This means that directly performing posterior inference under our model (which has learnt the training distribution, and ideally assigns zero probability to OOD test images) is not sound. We therefore make appropriate interventions on our model [102]. taking advantage of its causal nature. For example, when the distribution of object arrangements is different test time, we replace the learnt prior pe (z*) on object arrangements with an uninformative uniform prior. Moreover, the variational encoder networks ence and ency used during training are not suitable for use test time, due to domain shift in their inputs. Therefore, our framework instead directly samples the posterior distribution of latent variables given an observed image, using Markov chain Monte-Carke Carlo (MCMC) inference 669
Our novel MCMC scheme alternates Langevin dynamics (LD) D2 130 and Metropolis-Hastingg (MH) 44 steps, to infer the latent scene variables (z", 2²) from a single observed image x* with viewpoint v*. The MH steps encourage the Markov chain make large jumps between modes of the posterior, while the LDD steps generate high-probabilliyy samples with less exploration. Each LD step ascends the gradient of
log [x"(x;; C(So(z)), v*), 2²) + log 2²) + logg((2) x log p(z", x² x", v*)
Each MH step first picks an object slot i uniformly at random, then samples a new latents for that object from a proposal distribution p(z,hape ),d)), accepting/rejeccing it according to the usual MH criterion 44 The proposal distribution ÿ approximates & Ei= pi=:p(zz,ap plz:hape 25o0) using a Gaussian mixture model fitted by rnpectation-maxiiissee
[23]; it thus captures the distribution of object latent codes while disregarding the ordering of object indices. Note that the compositionality of our model increases efficiency of the chain, as it allows MH proposals that affect only one object while keeping other variables fixed in contrast a monolithic latent embedding, which would require accepting or rejecting global modifications to the entire scene. Thus, each MH step need not revert progress made on other variables: e.g. if the background is perfectly inferred but objects are not, then an MH proposal may change only an object. leaving the background intact. Also, it allows caching computation and only re-rendering parts of the scene that need to be considered for a proposed change (e.g. just background). In contrast, MCMC on non-structured models must render the entire scene from scratch.
4 Experiments
We conduct experiments on two synthetic datasets, using our model and three baselines. We also include ablations of our model, without MCMC inference, and with an unstructured latent space. We first evaluate performance in the standard setting where the test-set distribution matches the training distribution. Then, we evaluate generalisation to OOD data, by using several OOD test splits for each dataset, which have zero probability under the training distribution. We first describe these datasets and test splits (Sec. 4.1. the tasks, metrics and baselines used for evaluation (S)c. and finally the performance of our proposed model, the baselines and ablations (Sec. 4.3) Implementation details for all models (hyperparameters, hardware, etc.) are given in the supplementary material.
4.1 Datasets
GQN. We render images of rooms containing several objects (cubes, cylinders, spheres), based on the 'rooms ring camera' dataset of [35]; similar datasets were used in [45] 33), but in all cases without OOD test splits. In the training split, the camera viewpoints are circular path around the center of the room, with the camera pointed at the center at fixed elevation angle. Textures for the walls and colors for the objects walls have the same texture as each other, with the fourth different. There are 3-4 objects present; these are placed near the side of the room identified by the odd texture. We define the following OOD test splits: (i) position: the 3-4 objects are now placed near a different wall; (ii) composition: unseen combinations of background and object textures; (iii) number of objects: 1/5/6 objects instead of
selected randomly from finite set, with some combinations held out. Three
2 Detectton of the distribution shift could done automatically with the generative model 13 108
We also conducted early experiments with black- box variational inference [107]. but this performed poorly due becoming trapped in local minima. MCMC is more able explore diverse modes of the posterior

---

GQN
ARROW
per per-image
per-scene
per-imiage
per-seene
PSNR
D.MRR 1. ARI 1 mSC PSNR D.MRE! & PSNR D.MRR.. ART
mSC
PSNR
D.MRE
Test
Ours
24.1
0.031
0.81
0.88
20.8
0.058
27.1
0.100
0.71
0.82
26.8
0.107
(wl MCMC)
24.5
0.031
0.91
24.7
0.031
27.0
0.101
0.70
0.82
27.0
0.102
Ours (w/o structure)
32.1
0.016
27.8
0.030
22.2
0.990
20.1
0.990
8-VAE
20.6
24.3
IODINE
26.2
0.50
0.54
27.7
0.63
0.55
Slot Att.
30.5
0.94
0.67
28.3
0.48
0.17
Slot Att. (MCMC)
27.8
0.83
0.56
29.6
0.35
0.14
EORFs
27.0
0.027
0.59
0.049
35.1
0.44
0.202
NAEF-VAE"
32.0
0.016
27.8
0.033
25.3
0.991
25.3
0.991
OOD
Ours
21.8
0.034
0.68
0.89
18.3
0,069
26.7
0.139
0.57
0.81
26.2
0.137
Ours (wro MCMC)
11.4
0.170
0.44
0.55
11.5
0.176
21.0
0.100
0.40
0.61
21.0
0.159
Ours (wlo structure)
17.1
0.221
15.4
0.250
20.6
0.992
0.990
8-VAB
15.6
21.1
IODINE
19.7
0.44
0.53
25.0
0.42
0.42
Slot AuL.
20.3
0.66
0.56
22.8
0.26
0.14
Slot AtL (MCMC)
23.7
0.68
0.54
27.2
0.32
0.17
OORF"
14.7
0.287
0.45
0.45
14.1
0.308
22.7
0.132
0.38
0.41
22.6
0.131
NeRE-VAE
15.9
0.271
14.9
0.301
19.4
0.992
19.4
0.992
Table 1: Quantitative results discriminative tasks, comparing performance for different methods on an in-distribution test set and OOD data. Dashes indicate the method does not support the task. Best results are shown in bold.
4; (iv) viewpoint: camera positions and elevations are randomly sampled. Exact details of the data generation process are given in the supplementary material.
ARROW. We render images using a modified version of the CLEVR dataset (59] similar those in [57]. These have four objects, of which one is always an arrow, two of which are the same as each other, and fourth that different. The arrow always points at the odd (fourth) object. Object colors are randomly sampled. The camera has a random azimuth and shallow elevation. We define the following OOD test splits: (i) position: the four objects are positioned in a line, and the arrow no longer points to the odd object; (ii) composition: all objects are the same shape and color, with no arrow present; (iii) number of objects: 1/5/6 objects instead of four; the arrow still points at one odd object; (iv) viewpoint: camera looks down steeply on the objects.
4.2 Evaluation
Tasks & metries. We consider the following discriminative tasks, all taking single image as input: instance segmentation, measured by adjusted Rand index (ART) and mean segmentation covering (mSC) 33, which in the PER-SCENE setting measures how well the model infers segmentation maps for novel viewpoints; depth prediction, measured by the mean relative error between predicted and true depths (D.MRE); and pixel reconstruction, measured by peak signal-noise ratio (PSNR), which in the -IEGIMMGE setting measures how faithfully our latent representation can autoencode OOD images, and in the PER-SCENE setting how well the model performs novel view synthesis (NVS) We report these metrics according to two different protocols: (i) IMAGIM,GE, where we calculate the metrics only on the input image passed to the model; and (ii) PER-SCENE, where we calculate the metrics jointly over 10 images of the scene, taken from different viewpoints, but still having received only of these as input. Thus, the latter setting measures how well the model predicts appearance, depth and segmentation from novel viewpoints. Finally, we evaluate a priori image generation, measured by the Fréchet Inception distance (FID) (4) and kernel Inception distance (KID) 15
between sampled and ground-truth image distributions.
Baselines. We compare our approach to five existing works. Slot Attention 82 is recent unsupervised object segmentation model, with a spatial mixture representation. It is purely 2D and not generative. We additionally experiment with MCMC inference on the trained Slot-Attention decoder (instead of using their encoder), to see whether the decoder generalises to OOD data. uORF [39 decomposes 2D images into 3D components represented NeRFs. Unlike ours, it is not generative, and does not explicitly represent position separate from appearance. NeRF-VAE
t The setting for this experiment is different, as at test- time-timenee the encoder requires the same number of images used during training, whereas every other experiment inputs just image
7

---

Slot-Attenton Recon.
OORF
Nar--VE
Input
Recon.
Segment.
Depths
NVS
NVS2
Rocon.
Recon.
Composition
Composition
Number
Position
Viewpoint
Viewpoint
Figure 2: Qualitative results on various tasks, using our model and the baselines, on OOD data. Each row shows the input image, then (col. 2-6) outputs from model: the reconstruction, instance segmentation, depth map, and two novel viewpoints. We see that our model predicts high-quality segmentations and depth-maps, and that the viewpoints plausible and consistent with the input. The final three columns show reconstructions (the easiest task) by the baselines. Here we see that they fail to generalise OOD data they have specialised their training dataset, and learnt shortcuts (36) that interpret the input images as if they were drawn from it. Thus, they fail to map objects in unfamiliar viewpoints or contexts to appropriate latent representations.
generative method over 3D NeRFs, but which does not separate individual objects in its latent space. IODINE is discriminative 2D method which performs iterative amortized inference over a spatial mixture model. 3-VA 50 is an unstructured VAE aiming to learn disentangled representations, which has been hypothesized to help with OOD generalization. Note that the baselines are either discriminative approaches that do not support accounting for distribution shifts, model the scene with one latent variable which cannot be intervened on to model different distribution scenes. Full details are in the supplementary Sec. I0
4.3 Results
In-distribution data. We first evaluate how each model performs the distribution of images was trained on. Results on the discriminative tasks are given in Tab. [ the top four rows show performance the test split, which is drawn from the same distribution as the training data (all standard deviations are in the supplementary). Qualitative results are displayed in Fig. 2 all methods successfully reconstruct input images (high per-image PSNR). The 3D-aware methods predict depths with a small relative error (low per-image D.MRE), with NeRF-VAE- slightly better on GQN, and ours on ARROW. Our method performs similarly to uORF* with ours slightly better on segmentation but slightly worse for novel view synthesis. For GQN, Slot Attention performs best on segmentation under ARI scores, with ours better according to mSC and for both metrics ARROW. Results for image generation are given in Tab. 2 and Fig. 3 (see the supplementary for images from HeRF-VAE") Note that Slot Attention, IODINE and uORF* are excluded, they cannot generate images a priori. Our model has clearly learnt the distribution of both datasets, including the space of likely object and background appearances. Quantitatively, ours performs similarly to NeRF- VAF* on GQN (with FID slightly better and KID slightly worse), and out-performs it on both metrics for ARROW. Fig.
We see that
4 we also show that our semantically-inerpreaaal latent space allows users edit the scene shown an image, by counterfactual inference over the model.

---

GQN
ARROW
FID
KID
FID
KID
Ours
80.3
0.053
141.4
0.167
Ours w/o no (z" (2)
200.4
0.180
275.7
0.365
NeRF-VAE
84.2
0.047
182.7 0.190
Table 2:
Quantitative results on image genera- tion for our method, ablation, and baseline.
Figure 3: Examples of images sampled from our model for GQN (top) and ARROW (bottom)
Move objects
Sumple new objects
Sample or nomove bockground
Gezerat
Move objects
Sample new objects
Sample background
Figure 4:
Editing scenes using model, by counterfactual inference. We condition the input scene, then modify (intervene on) certain latent variables and re-render the result
OOD data. We now consider the more challenging setting where test data is drawn from a different distribution than the training set. Quantitative results in this setting are given in the bottom half of Tab. [ This shows the mean over all OOD splits; full breakdown is given in the supplementary. We see that in general, our method significantly out-performs the baselines on these splits - showing the benefit of our causal model and inference scheme. In particular, our method performs best on all combinations of vision tasks and datasets, except for depth estimation on ARROW where RRRF* is slightly better. Slot-Attentin with MCMC and IODINE with iterative amortized inference successfully reconstruct input images during their test-time optimization, but this comes at the cost of lower segmentation performance. In contrast, our method performs well on both reconstruction and segmentation. Thus, the best methods on the nn-distribution test split are not the best on OOD data. They leverage features that predictive of the output in the training set (36) and easy to learn 133 [06], but which are not reliable across different distributions 91]. Our qualitative results (Fig. 2 reinforce this interpretation we that our method still predicts accurate segmentations and mapth-maps for OOD data, and novel viewpoints look plausible. Notably, in all cases, it successfully reconstructs the input image via its latent space and synthesises new viewpoints, in spite of never having seen such an image during training. In contrast, the baselines struggle with images that are different to their training distribution e.g. mispredicting object colors (Ist row, uORF* and NeRF-VAE*), predicting an over-smoothed average scene (3rd row, NeRF-VAE*), or failing to separate walls and ceiling when they in an unfamiliar pose (5th row, Slot Attention).
In the supplementary material (S)c.[6). we provide a more detailed analysis of how model performance generalizes various axes of out-of-distribution images. Particularly noteworthy is that the 2D baseline Slot Attention 82 has the most significant drop in reconstruction performance (measured by PSNR) on images taken from -ittr---istribution camera viewpoints (Tab. 3) In contrast, our 3D-aware model has a much smaller drop in performance. Similarly important are the qualitative results in Fig. [ showing that the non-compositional baseline model NeRF-VAE 772 fails the straightforward task of reconstructing the input image - it predicts scenes similar to ones seen in the training dataset even when the OOD images are clearly different (e.g. still predicting the same object positions as in the training set despite objects being at the opposite side of the room). In contrast, our method which uses test-time optimization accurately reconstructs out-of- distririinttion input images.
4.4 Ablation Study
Ablating MCMC inference. We perform an ablation study the effects of our novel MCMC scheme. We compare it with the common approach of amortised inference (i.c. an encoder network predicts the posterior parameters). This is denoted in Tab. Tas as Ours (w/o MCMC). We see that amortised inference is indeed critical bottleneck for out-of-distibbtiion generalization: though performs well when the test distribution is identical the training distribution, its performance drops significantly on ditto--distribution images, while MCMC holds up.
Ablating generative model structure. We analyse the effects of using our compositional model compared to a non-compositional model, which has one global latent variable rather than one per object (similar to NeRF-VAB but otherwise with the same architecture and inference ours. This

---

is denoted in Tab. [ Ours (w/o structure). We see that our proposed compositional generative model significantly improves -ittr--diitritutionn generalization over the unstructured ablation: though both perform similarly on IID test data, the unstructured model performs significantly worse on out-of-distribution images.
A Ablating pe (x" [z"). We analyse the effects of our high-level prior over scene variables. In Tab, 2 we evaluate samples generated by the model, comparing FID and KID with an ablated model that samples z" from fixed prior. The poorer scores from the ablated model demonstrate that our hierarchical approach with a high-level prior necessary to correctly model the density of scenes: our model samples plausible scenes as it can model relationships between objects, while the ablated model performs much worse.
4.5 Limitations and Societal Impacts
Currently our method has several limitations.
The scene-level prior distribution pe 220) uses an unstructured latent space and decoder thus, this particular component of the model not interpretable. It would be worthwhile to use a structured prior, that explicitly models sparse relations among objects.
We have only tested it on synthetic data; it would be interesting to conduct experiments on natural images. This would require modelling the image-formation process more faithfully, e.g. by modelling light transport and non-Lambertian materials.
Our MCMC inference scheme is computationally expensive, as we render a complete image at each step. The efficiency could be increased by using stochastic gradient for the LD transitions.
Our model lacks variables to explicitly model object presence if an object slot unused, learns latent representation of an empty, zero-density object.
It would be preferable to use less dataset-specific prior information, e.g. candidate cells for objects.
Optimizing the model required multiple stages of training, and optimising Eq. was unstable. It would be worthwhile to investigate joint training as in [57].
We do not anticipate any negative societal impacts from this work. While all generative models have the potential to be used for creating fake content, our method requires further development to work on realistic images. On discriminative tasks, support for OOD inference should reduce the susceptibility of models to dataset bias; thus, we see a significant long-term benefit to such methods.
Conclusion
We have presented a new object-centric generative model that captures the causal process by which images are produced, and incorporates universal physical knowledge such as 3D geometry. We have shown that it is possible intervene on our model then perform Monte-Carlo inference, in order to process OOD images. On instance segmentation and novel view synthesis in this OOD setting, it significantly oet-performs three state-of-the-arr approaches.
Acknowledgments and Disclosure of Funding
We thank Professor Christopher K. I. Williams for detailed feedback on the draft and Professor lasonas Kokkinos and Hakan Bilen for fruitful discussions. We also thank the NeurIPS reviewers for extensive feedback, discussion and experiment suggestions. This work was mostly done during T.A.'s 's internship at Snap Inc. T.A. was partly supported by an EPSRC Doctoral Training Partnership.
10

---

References
[1] Abbas and Deny. Progress and limitations of deep networks to recognize objects in unusual poses. arXiv preprint 22x1:2207.000344 2022.
[2] K. Ahuja, E. Caballero, D. Zhang. J.-C. Gagnon-Audet, Y. Bengio, I. Mitliagkas, and I. Rish. Invariance principle meets information bottleneck for out-of-distribution generalization, Ad- vances in Neural Information Processing Systems, 34, 2021.
[3] K. Ahuja, K. Shanmugam, K. Varshney, and A. Dhurandhar. Invariant risk minimization games, International Conference on Machine Learning, pages 145-155. PMLR, 2020. 3
[4] M. A. Alcorn, Q. Li, Z. Gong, C. Wang. L. Mai, W.-S. Ku, and A. Nguyen. Strike (with) pose: Neural networks are easily fooled by strange poses of familiar objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages
2019. 19
[5] T. Anciukevičius, C. H. Lampert, and P. Henderson. Object-centric image generation with factored depths, locations, and appearances. arXiv preprint arXiv:2004.006-2. 2020.
28
[6] M. Arjovsky, L. Bottou, I. Gulrajani, and D. Lopez-Paz. Invariant risk minimization. CoRR, abs/19977000 0098992999,,, 2019. 3
[7] A. Azulay and Y. Weiss. Why do deep convolutional networks generalize so poorly to small image transformations? J. Mach. Learn. Res., 20:184:1-184:2: 2019. 1 19
[8] A. Barbu, D. Mayo, J. Alverio, W. Luo, C. Wang, D. Gutfreund, J. Tenenbaum, and B. Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. Advances in neural information processing systems, 32, 2019. 1 T9
[9] H. Barlow. Cerebral Cortex Model Builder. In Matters of intelligence, pages 395-406. Springer, 1987.
[10] M. A. Bautista, W. Talbott, S. Zhai, N. Srivastava, and J. M. Susskind. On the generalization of learning-based 3D reconstruction. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2 2180-2189, 2021. 3
[1]] S. Beery, G. Van Horn, and P. Perona. Recognition in terra incognita. In Proceedings of the European conference on computer vision (ECCV), pages 456-473, 2018. D
[12] J. Besag. Comments "Representations of knowledge in complex systems" by U. Grenander and M.I. Miller. Journal of the Royal Statistical Society, Series B, 56:591-592, 1994. 6
[13] C. M. Bishop. Novelty detection and neural network validation. IEEE Proc. on Vision, Image and Signal processing. 141-222217-222, 1994. 6
[14] C. M. Bishop. Pattern Recognition and Machine Learning. Springer, 2006. I
[15] M. Binkkwski, D. Sutherland. M. Arbel, and A. Gretton. Demystifying MMD GANs. In International Conference on Learning Representations, 2018.
[16] R. N. Bracewell and R. N. Bracewell. The Fourier transform and its applications, volume 31999. McGraw-hill New York, 1986.
[17] C. Burgess, L. Matthey, N. Watters, R. Kabra, Higgins, M. M. Botvinick, and Lerehner. MONet: Unsupervised Scene Decomposition and Representation. CoRR, abs/ 1ttp/991..11111 1.01.11390 1390, 2019. 28
[18] E. R. Chan, C. Lin, M. Chan, K. Nagano, B. Pan, S. De Mello, O. Gallo, Guibas, J. Tremblay, S. Khamis, et al. Efficient geometry-awane 3D generative adversarial networks. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16123-16133, 2022. 5
[19] E. R. Chan, M. Monteiro, P. Kellnhofer, J. Wu, and G. Wetzstein. pi-GAN: Periodic Implicit
Generative Adversarial Networks for 3D-Aware Image Synthesis. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5799-5809, 2021.
[20] C. Chen,, Deng, and Ahn. ROOTS: Object-Centric Representation and Rendering of Scenes. J. Mach. Learn. Res., 1-25939-- 2021. 25 28

---

[21] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A Simple Framework for Contrastive Learning of Visual Representations. In ICML, 2020. [
[22] P. Dayan, G. E. Hinton, R. M. Neal, and R. S. Zemel. The Helmholtz Machine. Neural computation, 7(5):889-93) 904, 1995. [
[23] A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum Likelihood from Incomplete Data via the EM Algorithm. Journal of the Royal Statistical Society, Series B, 39(1):1-38, 1977. 6
26
[24] Y. Deng, J. Yang, J. Xiang, and X. Tong. GRAM: Generative Radiance Manifolds for 3D- Aware Image Generation, 2021. 3
[25] T. Devries, M. A. Bautista, N. Srivastava, G. W. Taylor, and J. M. Susskind. Unconstrained Scene Generation with Locally Conditioned Radiance Fields. 2021 IEEF/CVF International
Conference on Computer Vision (ICCV), pages 14284-14293, 2021. 3
[26] T. v. Dijk and G. d. Croon. How do neural networks see depth in single images? In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2183-2191, 2019. 3
[27] K. Doya, S. Ishii, A. Pouget, and R. P. Rao. Bayesian brain: Probabilistic approaches to neural coding. MIT press, 2007. S
[28] D. Driess, Huang, Li, R. Tedrake, and M. Toussaint. Learning multi-object dynamics
with compositional neural radiance fields. CoRR, abs/2202.11855. 2022.
[29] Y. Du, S. Li, Y. Sharma, B. J. Tenenbaum, and I. Mordatch. Unsupervised learning of compositional energy concepts. In Advances in Neural Information Processing Systems, 2021.
[30] S. Ehrhardt, O. Groth, A. Monszpart, M. Engelcke, I. Posner, N. J. Mitra, and A. Vedaldi. RELATE: physically plausible multi-object scene synthesis using structured latent spaces, In Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. 28
[31] P. Emami, P. He, S. Ranka, and A. Rangarajan. Efficient iterative amortized inference for learning symmetric and disentangled multi-object representations. In International Conference Machine Learning, pages 2970-2981 PMLR, 2021. 3
[32] M. Engelcke, O. Jones, and I. Posner. GENESIS-VE V2: inferring unordered object representa- tions without iterative refinement. In M. Ranzato, A. Beygelzimer, Y. N. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 14, 2021, virtual, pages 8085-8094, 2021.
3 28
[33] M. Engelcke, A. R. Kosiorek, O. P. Jones, and Posner. GENESIS: generative scene inference and sampling with object-centric latent representations. In 8th International Conference on Learning Reprexentations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, 2020. I 2
3 6
26 28
[34] S. Eslami, N. Heess, T. Weber, Y. Tassa, D. Szepesvari, G. E. Hinton, et al. Attend, infer, repeat: Fast scene understanding with generative models. Advances in Neural Information
Processing Systems, 29:3225-322 3233, 2016.
3 28
[35] S. A. Eslami, D. J. Rezende, Besse, Viola, S. MoMcos, M. Garnelo, A. Ruderman, A.
Rusu, I. Danihelka, K. Gregor, et al. Neural scene representation and rendering. Science,
360(6394): 204-1210. 2018
26
28
[36] R. Geirhos, J. Jacobsen, C. Michaelis, R. Zemel, W. Brendel, M. Bethge, and F. A. Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11))665-
673, 2020. 2
19
[37] R. Geirhos, P. Rubisch, C. Michaelis, M. Bethge, F. A. Wichmann, and W. Brendel. ImageNet- trained CNNs biased towards texture; increasing shape bias improves accuracy and robust- ness. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans,
LA. USA, May 6-9, 2019, 2019. 3
[38] A. Goyal, A. Lamb, P. Gampa, P. Beaudoin, C. Blundell, S. Levine, Y. Bengio, and M. C. Mozer. Factorizing declarative and procedural knowledge structured, dynamical environments. In International Conference on Learning Representations, 2021.
12

---

[39] A. Goyal, A. Lamb, J. Hoffmann, S. Sodhani, S. Levine, Y. Bengio, and B. Schölkopf. Recurrent independent mechanisms. In International Conference on Learning Representations.
2021. 3
[40] K. Greff, R. L. Kaufman, R. Kabra, N. Watters, C. Burgess, D. Zoran, L. Matthey, M. Botvinick, and A. Lerehner. Multi-object representation learning with iterative variational inference. In International Conference on Machine Learning, pages 2424-2222222 2424-2433. PMLR, 2019. 8 28
[41] U. Grenander. Lectures in pattern theory-volume 1: Pattern synthesis. Applied Mathematical Sciences, 1976.
[42] I. Gulrajani and D. Lopez-Paz. In search of lost domain generalization. In International Conference on Learning Representations, 2021. 3
[43] M. Guo, A. Fathi, Wu, and T. Funkhouser. Object-centric neural scene rendering. CoRR, abs/2012.08503, 2020. B
[44] W. K. Hastings. Monte Carlo Sampling Methods using Markov Chains and their Applications.
Biometrika, 57(1):97-109 Apr. 1970. 6
[45] P. Henderson and C. H. Lampert. Unsupervised object-centric video generation and decompo-
sition in 3D. In Advances in Neural Information Processing Systems (NeurlPS) 33, 2020. 6
25
26 [28]
[46]
Hendrycks, S. Basart, N. Mu, Kadavath, F. Wang. Dorundo, Desai, T. Zhu, S. Parajuli, M. Guo, et al. The many faces of robustness: A critical analysis of out-of- distribution generalization.
Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8340 1440-8349,40 2021.
[47] D. Hendrycks and T. Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. In International Conference on Learning Representations, 2019.
[48] D. Hendrycks, K. Zhao, S. Basart, Steinhard, and D. Song. Natural adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15262- 2021.
[49] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. GANs Trained by Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. In Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30, 2017. 7
[50] I. Higgins, L. Matthey, A. Pal, C. P. Burgess, X. Glorot, M. M. Botvinick, S. Mohamed, and A. Lerchner. beta- VAE: Learning Basic Visual Concepts with a Constrained Variational Framework. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017.
8 25
[51] G. E. Hinton. How represent phole hierarchies in neural network. CORR, abs/2102.12666 2021.
19
[52] H. Izadinia, Q. Shan, and S. M. Seitz. Im2cad. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5134-5143, 2017. 3
[53] V. Jampani, S. Nowozin, M. Loper, and P. Gehler. The informed sampler: discriminative approach to Bayesian inference in generative computer vision models. Computer Vision and Image Understanding, 136:32 44, 2015. 3
[54] E. Jang, S. Gu, and B. Poole. Categorical reparameterizztion with gumbel-softmax. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017. 5
[55] W. Jang and L.. Agapito. CodeNeRF: Disentangled Neural Radiance Fields for Object Cat- egories. In 2021 IEFF/CVF International Conference Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 12929-12938. IEEE, 2021. 3
[56] D. Janzing and B. Schölkopf. Causal inference using the algorithmic Markov condition. IEEE Trans. on Information Theory, 2010. 2
[57] J. Jiang and S. Ahn. Generative neurosymbolic machines. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS
2020, December 6-12, 2020, virtual, 2020. 3
10
27 28
13

---

[58] J. Jiang, S. Janghorbani. G. de Melo, and S. Ahn. SCALOR: generative world models with scalable object representations. In 8th International Conference on Learning Representations. ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. 28
[59] J. Johnson, B. Hariharan, L. van der Maaten, L. Fei-Fei. C. L. Zitmick, and R. Girshick. CLEVR: a diagnostic dataset for compositional language and elementary visual reasoning. In CVPR, 2017. 7 27
[60] R. Kabra, D. Zoran, G. Erdogan, L.. Matthey, A. Creswell, M. M. Botvinick, A. Lerchner, and C. P. Burgess. Simone: View-invariant, comporally-abstractee object representations via unsupervised video decomposition. In M. Ranzato, A. Beygelzimer, Y. N. Dauphin, P. Liang, and W. Vaughan, editors, Advances Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14 14, 2021, virtual, pages 20146-20159 2021.
3 28
[61] L. Karazija, S. Choudhury, I. Laina, C. Rupprecht, and A. Vedaldi. Unsupervised multi-object segmentation by predicting probable motion patterns. In Advances in Neural Information Processing Systems 35, 2022. 3
[62] L. Karazija, I. Laina, and C. Rupprecht. ClevrTex: A texture-rich benchmark for unsupervised multi-object segmentation. In J. Vanschoren and S. Yeung, editors, Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurlPS Datasets and Benchmarks 2021, December 2021, virtual, 2021. 3
[63] D. Kersten, P. Mamassian, and A. Yuille. Object perception as Bayesian inference. Annu. Rev. Psychol., 55:271 2004. 3
[64] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In Y. Bengio and Y. LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015, 5 25
[65] D. P. Kingma and M. Welling. Auto-encoding variational bayes. Y. Bengio and Y. LeCun, editors, 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014. 5
[66] T. Kipf, G. F. Elsayed, A. Mahendran, A. Stone, S. Sabour, G. Heigold, R. Jonschkowski, A. Dosovitskiy, and K. Greff. Conditional Object-Centric Learning from Video. In Interna- tional Conference on Learning Representations (ICLR), 2022. 3
[67] Kobayashi, M. Suzuki, and Y. Matsuo. Learning global spatial information for multi-view
object-centric models, 2022. 3
[68] P. W. Koh, S. Sagawa, H. Marklund, S. M. Xie, M. Zhang, A. Balsubramani, W. Hu, M. Ya- sunaga, R. L., Phillips, I. Gao, et al. Wilds: A benchmark of in-the-wild distribution shifts. In
International Conference on Machine Learning, pages 5637-5664. PMLR, 2021. 3
[69] D. Koller and N. Friedman. Probabilistic graphical models: principles and techniques. MIT press, 2009. 6
[70] A. Kortylewski, He, Q. Liu, C. Cosgrove, C. Yang, and A. L. Yuille. Compositional generative networks and robustness to perceptible image changes. In 202/ 55th Annual Conference on Information Sciences and Systems (CISS), pages 1-8. IEEE, 2021. |
[71] Kosiorek, S. Sabour, Y. W. Teh, and G. E. Hinton. Stacked capsule autoencoders. Advances in neural information processing systems, 32, 2019.
19
[72] A. R. Kosiorek, H. Strathmann, D. Zoran, P. Moreno, R. Schneider, S. Mokrá, and D. J. Rezende. NeEF-VAE: A geometry aware 3d scene generative model. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 5742-5752. PMLR, 2021.
27 28
[73] A. Krizhevsky, I. Sutskever, and G. E. Hinton. ImageNet classification with deep convolutional neural networks. In F. Pereira, C. Burges, L. Bottou, and K. Weinberger, editors, Advances in Neural Information Processing Systems, volume 25, 2012. [
[74] D. Krueger, F. Caballero, J. Jacobsen, A. Zhang, J. Binas, D. Zhang, R. Le Priol, and Courville.
generalization via risk extrapolation (rex). In International
Conference on Machine Learning, pages 5815-5826. PMLR, 2021. 3
14

---

[75] T. D. Kulkarni, P. Kohli, J. B. Tenenbaum, and V. Mansinghka. Picture: A probabilistic programming language for scene perception. In 2015 JEEE Conference on Computer Vision and Puttern Recognition (CVPR), pages 4390443999 4399, 2015. 3
[76] B. M. Lake, T. D. Ullman. J. B. Tenenbaum, and S. J. Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40, 2017.
[77] J. Li, Z. Feng, Q. She, H. Ding, C. Wang, and G. H. Lee. Mine: Towards continuous depth mpi with nerf for novel view synthesis. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 12558-12568, 2021. 3
[78] Z. Lin, Y. Wu, S. V. Peri, W. Sun, G. Singh, F. Deng, J. Jiang, and S. Ahn. SPACE: unsu- pervised object-oriented scene representation via spatial attention and decomposition. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia,
April 26-30, 2020, 2020. 5 28
[79] L. Liu, J. Gu, K. Zaw Lin, T.-S, Chua, and C. Theobalt. Neural sparse voxel fields. Advances
in Neural Information Processing Systems, 33:15651-15666 2020. 5
[80] N. Liu, S. Li, Y. Du, J. Tenenbaum, and A. Torralba. Learning compose visual relations.
Advances in Neural Information Processing Systems, 34, 2021.
3
[81] F. Locatello, S. Bauer, M. Lucic, G. Raetsch, S. Gelly, B. Schölkopf, and O. Bachem. Chal- lenging common assumptions in the unsupervised learning of disentangled representations. In international conference on machine learning. pages 41 14 4124. PMLR, 2019. 3
[82] F. Locatello, D. Weissenbom,, T. Unterthiner, A. Mahendran, G. Heigold, J. Uszkoreit, A. Doso- vitskiy, and T. Kipf. Object-centric learning with slot attention. Advances in Neural Information Processing Systems, 33:1 1525- 2020. 3
7
9
19
27
[83] M. M. Loper and M. J. Black, Opendr: An approximate differentiable renderer. In European Conference on Computer Vision, pages 154-169. Springer, 2014.
[84] C. J. Maddison, A. Mnih, and Y. W. Teh. The concrete distribution: A continuous relaxation of discrete random variables. In International Conference on Learning Representations, 2017.5 5
[85] R. Martin- Martin-Brualla, N. Radwan, M. S. Sajjadi, T. Barron, A. Dosovitskiy, and Duckworth. Nerf in the wild: Neural radiance fields for unconstrained photo collections. In Proceedings of the IEFF/CVE Conference on Computer Vision and Pattern Recognition, pages 7210-7219, 2021.
3
[86] N. Max. Optical models for direct volume rendering. IEEE Trans. on Visualization and Computer Graphics, 1995.
[87] T. Milbich, K. Roth, S. Sinha, L. Schmidt, M. Ghassemi, and B. Ommer. Characterizing generalization under out-of-distribution shifts in deep metric learning. Advances in Neural
Information Processing Systems, 34, 2021. 3
[88] Mildenhall, P. Srinivasan, M. Tancik, T. Barron, R. Ramamoorthi, and R. Ng. Nerf: representing scenes as neural radiance fields for view synthesis. Commun. ACM, 65(1)999-16, 2022.
24 25 26
[89] M. L. Montero, J. S. Bowers, R. P. Costa, C. J. H. Ludwig, and G. Malhotra. Lost in latent space: Disentangled models and the challenge of combinatorial generalisation. arXiv preprint urXiv:2204.02220 2022.
[90] P. Moreno Comellas. Vision inverse graphics for detailed scene understanding. 2019.
[91] K. P. Murphy. Probabilistic Machine Learning: Advanced Topics. MIT Press, 2023.
9
[92] V. Nagarajan, A. Andreassen, and B. Neyshabur. Understanding the failure modes of out- of-distribution generalization. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021. 3
[93] L. Nanbo, C. Eastwood, and R. B. Fisher. Learning object-centric representations of multi- object scenes from multiple views. In 34th Conference on Neural Information Processing Systems, 2020.
3
27 28
[94] T. Sguyen-Phuucc, L. I. Theis, C. Richardt, and Y.-L. Yang. HoloGAN: Unsupervised learn- ing of 3D representations from natural images. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 7588-7597, 2019. 2
15

---

[95] T. (guyen-Phuooe C. Richardt, L. Mai, Y.-L. Yang. and N. Mitra. BlockGAN: Learning
3D Object-aware Scene Representations from Unlabelled Images. In Advances in Neural Information Processing Systems 33, Nov 2020. 3
[96] M. Niemeyer, J. T. Barron, B. Mildenhall, M. M. Sajjadi, A. Geiger, and N. Radwan. RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputs. In
Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2022. 3
[97] M. Niemeyer and A. Geiger. GIRAFFE: Representing Scenes as Compositional Generative
Neural Feature Fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and
Puttern Recognition, pages 1453-1 1464, 2021.
2
24
25
28
[98] M. Niemeyer, L. Mescheder, M. Oechsle, and A. Geiger. Differentiable volumetric rendering:
Learning implicit 3D representations without 3D supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3504-3515, 2020. E
[99] J. Ost, F. Mannan, N. Thuerey, J. Knodt, and Heide. Neural scene graphs for dynamic scenes. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2856-2865, 2021.
[100] G. Parascandolo, A. Neitz, A. Orvieto, L. Gresele, and B. Schölkopf. Learning explanations that are hard to vary. In International Conference on Learning Representations, 2021. 3
[101] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang. J. Bai, and S. Chintala. PyTorch: An Imperative Style, Deep Learning Library. In Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc., 2019. 25
High-Performane
[102] J. Pearl. Causality. Cambridge University Press, 2 edition, 2009,
2 6
[103] S. Peng, M. Niemeyer, L. Mescheder, M. Pollefeys, and A. Geiger. Convolutional occupancy
networks, In European Conference on Computer Vision, pages 523-540. Springer, 2020. 3
[104] J. Peters, P. Bühlmann, and N. Meinshausen. Causal inference by using invariant prediction: identification and confidence intervals. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 78(5):947-1012 2016.
[105] J. Peters, D. Janzing, and B. Schölkopf. Elements of causal inference: foundations and learning algorithms. The MIT Press, 2017.
[106] M. Pezeshki, O. Kaba, Y. Bengio, A. C. Courville, D. Precup, and G. Lajoie. Gradient starva- tion: A learning proclivity in neural networks, Advances in Neural Information Processing Systems,
2021. [9]
[107] R. Ranganath, S. Gerrish, and D. M. Blei. Black box variational inference. In Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics, AISTATS 2014, Reykjavik, Iceland, April 22-25, 2014, volume 33 of JMLR Workshop and Conference
Proceedings, pages 814-822. JMLR.org, 2014. 6
[108) J. Ren, P. J. Liu, F. Fertig, J. Snoek, R. Poplin, M. Depristo, J. Dillon, and B. Lakshmi- narayanan. Likelihood ratios for out-of-distribution detection. Advances in neural information
processing systems, 32, 2019. 6
[109] D. J. Rezende, S. Mohamed, and D. Wierstra. Stochastic backpropagation and approximate inference in deep generative models. International conference on machine learning, pages 1278-1286. PMLR, 2014. 5
[110] M. T. Ribeiro. S. Singh, and C. Guestrin. "Why should trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference knowledge discovery and data mining, pages 1135-1144, 2016
33
[111] L. Romaszko, C. K. Williams, P. Moreno, and P. Kohli. Vision-as- Inweas-IIvvveeeeeeee Graahie-: Ob- taining a Rich 3D Explanation of a Scene from a Single Image. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 851-859, 2017.
[112] L. Romaszko, C. K. Williams, and J. Winn. Learning direct optimization for scene understand- ing. Pattern Recognition, 105:10 2020. 3
[113] B. Schölkopf, D. Janzing, J. Peters, F. Sgouritsa, K. Zhang, and J. M. Mooij. On causal and anticausal learning. In Proceedings of the 29th International Conference on Machine Learning, ICML 2012, Edinburgh, Scotland, UK, June 26 - July 1, 2012. icmlec / Omnipress, 2012. 2
16

---

[114] L. Schott, J. V. Kügelgen, F. Träuble, P. V. Gehler, C. Russell, M. Bethge, B. Schölkopf, F. Looatello,, and W. Brendel. Visual representation learning does not generalize strongly within the same domain. In International Conference on Learning Representations, 2022.
[115] K. Schwarz, Y. Liao, M. Niemeyer, and A. Geiger. GRAF; Generative Radiance Fields for 3D-Aware Image Synthesis. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems 33: Anmual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
[116] B. Schölkopf, F. Locatello, S. Bauer, N. R. Ke, N. Kalchbrenner, A. Goyal, and Y. Bengio. Toward causal representation learning. Proceedings of the IEEE, 109(5)6612-6344 2021. D
[117] H. Shah, K. Tamuly, A. Raghunathan, P. Jain, and P. Netrapalli. The pitfalls of simplicity bias in neural networks. Advances Neural Information Processing Systems. 33:9573 958595555228 2020.
9
[118] S. Shahtalebi, J.-C. Gagnon-Audet, T. Laleh, M. Faramarzi, K. Ahuja, and I. Rish. SAND- mask: An Enhanced Gradient Masking Strategy for the Discovery of Invariances in Domain Generalization. In ICML UDL Workshop, 2021.
[119] Y. Shi, J. Seely, P. Torr, S. N, A. Hannun, N. Usumier, and G. Synnaeve. Gradient matching for domain generalization. In International Conference on Learning Representations, 2022. B
[120] V. Sitzmann, J. Martel, A. Bergman, D. Lindell, and G. Wetzstein. Implicit neural representa- tions with periodic activation functions. Advances in Neural Information Processing Systems,
33, 2020. 3
[121] V. Sitzmann, M. Zollhöfer, and G. Wetzstein. Scene representation networks: Continuous 3d- structure-aware neural scene representations. In H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché-Buc, E. B. Fox, and R. Gamett, editors, Advances in Neural Information Processing Systems 32: Annuul Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 1 19-1130, 2019. 3
[122] K. Stelzner, K. Kersting, and A. R. Kosiorek. Decomposing 3d scenes into objects via unsupervised volume segmentation. CoRR, abs/2104.01144, 148, 2021.
25
28
[123] M. Tatarchenko, S. R. Richter, R. Ranftl, Z. Li, V. Koltun, and T. Brox. What do single-view
3d reconstruction networks learn? In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 3405-3414, 2019. 3
[124] A. Tewari, O. Fried, Thies, V. Sitzmann, S. Lombardi, K. Sunkavalli, R. Martin-Brualle
T. Simon, J. Saragih, M. Nießner, et al. State of the art on neural rendering. In Computer
Graphies Forum, volume 39, pages 701-727. Wiley Online Library, 2020. 3
[125] A. Trevithick and B. Yang. GRF: Learning General Radiance Field for 3D Scene Representa- tion and Rendering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15182-15192, 2021. 3
[126] S. van Steenkiste, K. Kurach, J. Schmidhuber, and S. Gelly. Investigating object composition- ality in generative adversarial networks. Neural Networks, 130:309 325, 2020. B
[127] V. Vapnik. Principles of risk minimization for learning theory. Advances in neural information processing systems, 4, 1991.
[128] J. von Kügelgen, I. Ustyuzhaninov, P. V. Gehler, M. Bethge, and B. Schölkopf. Towards causal generative scene models via competition of experts. CORR, abs/2004. 12906, 2020.
[129] M. Welling. Do we still need models or just more data and compute?, Apr 2019, Retrieved from https://staff. fnwi uva http///wwllinn/ww....o contentt/ uploads/Modeel versus-
AI- pdf I March 2022.
[130] M. Welling and Y. W. Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th International Conference International Conference Machine Learning, ICML'1 11, 1, page 681-688, Madison, WI, USA, 2011. Omnipress. 6
[131] O. Wiles, S. Gowal, Stimberg, S.-A. Rebuffi, Ktena, K. D. Dvijotham, and A. T. Cemgil. A fine-grainee analysis on distribution shift. In International Conference on Learning Repre- sentations, 2022. S
[132] S. Wizadwongsa, P. Phongthawee, J. Yenphraphai, and S. Suwajanakorn. Nex: Real-time view synthesis with neural basis expansion. In IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2021. 3
17

---

[133] K. Y. Xiao, L. Engstrom, A. Ilyas, and A. Madry. Noise or signal: The role of image
backgrounds in object recognition. In International Conference on Learning Representations.
2021. 9
[134] Y. Xue, Y. Li, K. K. Singh, and Y. J. Lee. GIRAFFE HD: A High-Resolution 3D-aware Generative Model. In CVPR, 2022.
[135] L. Yariv, Y. Kasten, D. Moran, M. Galun, M. Atzmon, R. Basri, and Y. Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance,
Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
[136] N. Ye, K. Li, H. Bai, R. Yu, I.. Hong, F. Zhou, 7. Li, and J. Zhu. Ood-bench: Quantifying and understanding two dimensions out-of-distributiio generalization. CVPR, 2022.
[137] L. Yen- Yen-Chen. Nerf-pytorch. http:s///gthub..coo//aaaaellll/p
2020.
24
[138] A. Yu, V. Ye, M. Tancik, and A. Kanazawa. pixelnerf: Neural radiance fields from one or few images. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 4578 4778,, 2021.
[139] H.-X Yu, L. Guibas, and J. Wu. Unsupervised discovery of object radiance fields. In
International Conference on Learning Representationss 2022. 5 25
27 28
[140] A. Yuille and D. Kersten. Vision as Bayesian inference: analysis by synthesis? Trends in cognitive sciences, 10(7):301-308, 2006. 3
[141] A. L.. Yuille and C. Liu. Deep nets: What have they ever done for vision? International Journal of Computer Vision, 129(3).781-80) 2021.
[142] J. R. Zech, M. A. Badgeley, M. Liu, A. B. Costa, J. J. Titano, and E. K. Oermann. Variable generalization performance of deep learning model to detect pneumonia in chest radiographs: a cross-sectional study. PLoS medicine, 15(1 l):e 1002683, 2018. 3
18


Reviews:
[{"review_id": "n8zQG3DKluh", "reviewer": "Reviewer_uNLg", "summary": "This paper proposes a generative model for unsupervised object-centric 3D scene understanding. The proposed model is claimed as \"causal\" as it is baked in with a multi-object Neural Radiance Field with explicit latent variables for color, shape, and object positions. This model is trained with ELBO in Variational Bayes. To demonstrate that the introduction of the \"causal\" structure can help generalize the inference of latent variables to OOD scenes, i.e. scenes not covered in the support set of the training data, the authors further propose an MCMC sampling method for test time only. The proposed model shows superior performance in GQN dataset where objects are iid and ARROW datasets where there is a specific correlation between objects.  ", "questions": "Apart from the questions I raised in the weakness part above, I would appreciate it if the authors would like to answer the following:\n\n1. Since the way of inference in OOD images is different in the proposed model and existing models (MCMC vs Variational Encoder), it is unclear which component between the explicit generative model and the MCMC inference plays a more critical role in the success of OOD generalization. Maybe the authors would like to apply their MCMC samplers to the trained decoder of Slot Attention, uORF, and NeRF-VAE? I believe this ablation can be beneficial for readers who are interested in OOD generalization. \n2. If time allows, authors may also want to compare these models in datasets where there are more authentic textures, for example, the CLEVR-TEX dataset? \n3. Authors claimed to use uniform distribution to replace the learned prior when generalizing to OOD images, how is that implemented and what is the rationale here? More specifically, what is the range of the support space, and why so? \n\nI will be happy to raise my rating if authors would like to resolve some of my concerns. \n\n--------------------\n\nPost discussion: Thanks authors for providing results on experiments I requested, I have raised my rating. ", "limitations": "Apart from the limitation authors listed in Sec 4.5, I believe the authors may also want to talk about how statistical assumptions in the proposal distribution in MH sampling may affect the OOD inference, And how can the assumption of knowing there is indeed a distribution shift can be fulfilled in reality. ", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\\+ This paper is very well written. I liked the narrative in which the authors first describe the representation and the generative models with probabilistic language, then introduce the inductive bias of NeRF, and then move on to introduce training and inference. This makes the method section very easy to follow. \n\n\\+ The introduction of the scene variable is fairly interesting. Existing methods for object disentanglement mostly assume iid prior over objects. The hierarchical design of the generative model echoes the \"conditional independence\" in causality, making the assumption in models more generic. \n\n\\+ Though couples of existing works use NeRF as a scene generator, the explicit modeling of the latent variable and the MCMC sampling in OOD scenes are novel contributions. In particular, the observation that the variational encoder from Variational Bayes training is the critical bottleneck for OOD generalization is very sharp. \n\n\\+ As far as I know, generalizing to novel viewpoints is something not covered by the existing literature on multi-object representation learning. \n\n\n\n\\- Though I liked the idea of using MCMC sampling for inference in OOD images, I have some concerns about its concrete realization. First, to account for the distribution shift in the latent causal factors, the proposed method needs to manually replace the learned prior of latent variables with uninformative distributions. Then is it fair to compare with other existing models since here the model is actually informed of which causal factor is intervened in the distribution shift? Second, the combination of Langevin dynamics and Metropolis-Hastings is interesting, but why the proposal of each object can be \"considered\" independently? \n\n\\- The comparison with the baselines is not completely fair. First, Slot Attention model does not use either the viewpoint information or multiple views of the same scene. Maybe authors should try a variant of it imitating how GQN extends VAE? I am also curious why the trained Slot Attention model does not generalize to objects with OOD number of objects since it should be successful according to the original paper. Second, NeRF-VAE is not encouraged to have structured latent representation, so we kind of know a priori it will fail the composition test, maybe the authors should try some disentangled VAEs such as beta-VAE? Third, the correlation in the ARROW dataset that the arrow object is always pointing to a special object is known a priori and the proposed model seems to perform well in this dataset by incorporating this knowledge into the model. In reality, we don't really know what kind of correlation there are among objects, how would the proposed model still be generic? ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ig5EYClfIUv", "reviewer": "Reviewer_KD5e", "summary": "The authors proposed a VAE-based NERF-like object-centric framework to model 3D-scenes. The model represents objects as separate NERFs. Moreover, authors make architectural choices that reflect causal independent mechanisms, so this enables OOD generalization. Plus, a novel MCMC inference scheme is proposed to infer out-of-distribution scenes. The model outperforms other models in OOD setting and performs competitively in In-distribution set-up.", "questions": "Ablation study questions \n\nMain questions\n1) Can you explicitly state all architectural choices you made to enable independent mechanisms and perform an ablation study on these choices? \n2) Do you need a two-step generation? The general latent layout of the scene is generated firstly and then latents of objects are generated based on this layout. Can you ablate it? \n3) Did you apply a new MCMC procedure to your baselines (slot-attn, uORF, NERF-VAE) for OOD setup? If not -- why? \n4) Can you ablate your MCMC procedure for OOD setup and show what scores the model obtains when one uses an encoder to obtain latents?\n\nSmall issues\n1) I did not understand figure 1, can you maybe elaborate on this picture a little bit more.\n2) I suggest to the author to include a graphical model in the main text to increase readability. \n", "limitations": "-", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- the idea and necessity of a novel MCMC inference scheme are intuitive and well-explained\n- the model outperforms baselines in OOD setting and performs competitively in In-distribution set-up.\n- the model is OC and generative (that is novel as far as I understand) compare to its OC counterpart (uORF), so I believe it’s easier to edit a scene object-wise.\n- authors make architectural choices to reflect the causal independent mechanisms.\n\nWeaknesses\n\n- there is no ablation study on model design -- this is my main concern\n- the architectural choices that make the model more causal are not explicitly stated in one place in the main text, so it’s hard to follow what exactly these choices are. And there is no ablation study on these choices either. \n\nOverall, in my opinion, the authors implemented the right priors to make OOD generalization possible and the paper seems novel to me. \n\nI am willing to increase my score if a proper ablation study will be performed.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "VJCcC86eb02", "reviewer": "Reviewer_tfzG", "summary": "This paper presents a slotted generative model with a NeRF rendering decoder. Each slot parameterizes a scene function and the decoder mixes the scene functions while rendering (summing densities and averaging colors). The model is trained as a 2-stage VAE - first training the slotted model VAE and then a \"global\" VAE which models the marginal posterior distribution of the first model. The slot latents are factorized into \"appearance\" and \"position\" and the latter is one-hot encoded (relaxed with Gumbel softmax) and used as a convolution kernel to \"shift\" the scene function in scene space while rendering.\nThe model receives one or more input views of a scene and is trained as a VAE, much like NeRF-VAE but with slots.\nAn inference procedure is also proposed to work in the case of OOD inputs where the amortized encoder is not expected to work.\nThe method is demonstrated to work nicely on two synthetic datasets - GQN and ARROW.\nThere are some experiments that show that the model is able to learn factorized representation and these can be manipulated.", "questions": "* How is the conditional model p(z_s|z_g) structured? I looked at the supplementary material and it wasn't clear - how is the mapping for a single latent to multiple slots done? is it just an MLP which scales with the number of slots?\n\n* Follow-up to the above: can you change the number of objects without affecting the number parameters in the model? Consequently - can you instantiate a model with a different number of slots than used in training?\n\n* What prevents the object location to be learned in a non-canonical position and compensate with the appropriate shift through the inferred position? There's nothing constraining the object to be learned around the origin.\n\n* Have you tried the actual learned encoder on OOD data instead of using MCMC inference? I know this is claimed not to work in the paper but I didn't see an experiment showing this.\n* Follow-up to the above: have you tried amortized iterative inference for evaluation of OOD? (IODINE works on unseen combinations of color and shape, for example)", "limitations": "I think the authors addressed the model limitations well in the paper - the above question directly refer to some of the missing points I feel.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality:\n  While mostly an amalgamation of existing methods this is an interesting model and the fact that it works, even on synthetic data, is non-trivial. The one-hot encoding of position is a nice touch and I suspect makes the model work better. \n  I am, however, not convinced by the \"Causality\" claim of the model - I am not a causality person, but I feel this is a stretch - this model is not more \"causal\" than IODINE, for example - it does, maybe, reflect the underlying generative process better (through cameras for example) but I feel this is a bit of stretch (see more below)\n\nQuality:\n  The paper is quite thorough, and all in all quite good. I feel the choice of comparing only to NeRF based baselines is a weakness - much of the rendering process, especially in simple scenes like these, can be learned well by 2D models (up to NVS) and these should have been compared to (for example, GQN and IODINE). I would want to see analysis of the global latent, and how it affects things (the authors do mention this under \"limitations\")\n\nClarity:\n  The paper is written nicely and quite clearly, but I feel several elements are not described well - especially the role and structure of the global latent, see below. \n\nSignificance:\n  All in all I feel this is an important contribution to the community with some non-trivial advances.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "KoUY902ctYs", "reviewer": "Reviewer_gUkP", "summary": "This paper proposes a generative model of images that factors appearance of the image into background and multiple objects — each object is associated with its own NERF based appearance model drawn from a global prior and together can be used to forward render an associated image in a differentiable way.  Latent object appearance models are inferred using a VAE-like framework with neural-net encoders.\n\nFinally by replacing certain learned priors by uniform priors, the authors show that their models are able to work in OOD settings quite accurately.  Results in this paper are all on synthetic baselines.\n\n", "questions": "* I cannot figure out how are shape and color separated in this model — it’s possible that I am missing something, but shape and color variables seem to be treated symmetrically throughout, so what forces one variable to map to shape and the other to map to color?\n* Related to this question, it would be useful to provide further factorization in Equation 2 to explain the joint distribution of shape, color and position of a single given object.\n* I wonder if there is a principled way to decide when it is okay to replace a learned prior with a uniform distribution.  In other words, when is it okay to extrapolate from the dataset?  Clearly the answer cannot be “all the time” — could the authors shed some light on how to make this decision appropriately?\n", "limitations": "yes.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nA full instance-level generative model of image appearance is arguably one of the “holy grails” of computer vision — and the fact that this paper takes a step in this very ambitious direction and achieves reasonably good results (on synthetic data) is a strength.  \n\nThere are two possible families of comparison — NERF based models which generally do not have a probabilistic object based decomposition of the scene, and 2d models of image composition which do capture object based decompositions, but are not able to learn from multi-view data in the same way. Compared to other works along the same lines (e.g. slot attention), the authors also show strong performance in their model’s ability to generalize to out-of-distribution data.  \n\nAnother strong point of the paper is the attention paid to reliable optimization.  For example, the authors find that modeling positions using a categorical variable helps (compared, e.g, to using naive spatial transformer models which would model pose using a continuous variable).  The idea is similar to the way that Faster R-CNN works for object detection --- use a discrete grid to predict at an \"anchor grid\" of positions, then use ROIAlign (which is basically an axis aligned STN) to crop features.\n\n\nWeaknesses:\n\nThere are some major things that are not modeled in this work but would be important in the long run — these include orientations (or a more general space of transformations beyond point-to-point translation) and semantics (there is not a way that the concept of “chair” can be learned by training on many images).  Also, even though images can be “out-of-distribution” in the sense of having different numbers of objects, being placed in different locations or having novel compositions, I don’t believe that generalizing to novel types of objects is handled well in this approach.  \n\nAnother weakness of the approach (mostly having to do with the fact that it requires MCMC) is the running time, which the authors acknowledge to be a limitation.  I would recommend mentioning more specifics about time requirements.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept