LAVAE: Disentangling Location and Appearance
Authors: Andrea Dittadi
Ole Winther
Venue:  (2020)
Keywords: structured scene representations
compositional representations
generative models
unsupervised learning

Abstract: We propose a probabilistic generative model for unsupervised learning of structured, interpretable, object-based representations of visual scenes. We use amortized variational inference to train the generative model end-to-end. The learned representations of object location and appearance are fully disentangled, and objects are represented independently of each other in the latent space. Unlike previous approaches that disentangle location and appearance, ours generalizes seamlessly to scenes with many more objects than encountered in the training regime. We evaluate the proposed model on multi-MNIST and multi-dSprites data sets.

Under review as a conference paper at ICLR 2020
LAVAE: DISENTANGLING LOCATION AND APPEARANCE
Anonymous authors Paper under double-blind review
ABSTRACT
We propose a probabilistic generative model for unsupervised learning of struc- tured, interpretable, object-based representations of visual scenes. We use amor- tized variational inference to train the generative model end-to-end. The learned representations of object location and appearance are fully disentangled, and ob- jects are represented independently of each other in the latent space. Unlike previ- ous approaches that disentangle location and appearance, ours generalizes seam- lessly to scenes with many more objects than encountered in the training regime. We evaluate the proposed model on multi-MNIST and multi-dSprites data sets.
INTRODUCTION
Many hallmarks of human intelligence rely on the capability to perceive the world as a layout of distinct physical objects that endure through time-a skill that infants acquire in early child- hood (Spelke) 1990; ( and Kinzler (007). Learning compositional, object-based repre- sentations of visual scenes, however, is still regarded as open challenge for artificial systems (Ben-
gio et al. 2013 Garnelo and Shanahan 2019).
Recently, there has been a growing interest in unsupervised learning of disentangled representa- tions (Locatello et al. (2018). which should separate the distinct, informative factors of variations in the data, and contain all the information on the data in a compact, interpretable structure (Bengio itt al. 2013).. This notion is highly relevant in the context of visual scene representation learning, where distinct objects should arguably be represented recent breakthroughs (Chen et al., 2016;; Higgins et al. [217]] Kim and Mnih 2018). multi-objecc scenarios are rarely considered (Eslami et al. 2016 Steenkiste et al. 2018 Burgess et al., 2019).
disentangled fashion. However, despite
We propose the Location-Appearanee Variational AutoEncoder (LAVAE), probabilistic generative model that, without supervision, learns structured, compositional, object-based representations of visual scenes. We explicitly model an object's location and appearance with distinct latent vari- ables, unlike in most previous works, thus providing a highly beneficial inductive bias. Following the framework of variational autoencoders (VAEs) (Kingma and Welling 2013 Rezende et al. 2014), we parameterize the approximate variational posterior of the latent variables with inference networks that trained end-to-end with the generative model. Our model learns to correctly count objects and compute a compositional, object-wise, interpretable representation of the scene. Objects represented independently of each other, and each object's location and appearance tangled. Unlike previous approaches that disentangle location and appearance, LAVAE generalizes seamlessly to scenes with many more objects than the training regime. We demonstrate these capabilities on multi-MNIST and multi-dSprites data sets similar to those by Eslami et al. (2016) and Greff et al. 2019).
disen-
METHOD
Generative model. We propose latent variable model for images in which the latent space is
factored into location and appearance of a variable number of objects. For each image x with D pixels, the number of objects is modeled by latent variable their locations by n latent variables (1) (z 11
(8) and their appearance by [ We assume the number of objects in every image

---

Under review as a conference paper at ICLR 2020
be bounded by M. The joint distribution of the observed and latent variables for each data point is:
p(x, Bloe, Rapp: r2) = p(x[ Zloc; Zapp (hap))((Zuaae Zioc
12)pp(Zapp (2)p(n)
=p(x Zloce Zapp)
II
p loc
( loc
jes pl (opsppp) p(n),
(1)
where we use the shorthand 7app
(1) Rapp J i=1 and similarly for Mloc-
The generative process can be described follows. First, the number of objects n sampled from categorical distribution
p(n) = Cat(M + a)) n1 E fO.....
(2)
where a is a learned probability vector of size M + 1. The n location variables are sequentially sampled without replacement from categorical distribution with D classes:
p Jac 1o0 j<i) yea) Cat(D,b())
1 b D D-i+1 (ID 1
(
loc
where: i 1, ...... n, and 1p is vector of ones of length D. To each inc which is one-hot repre- sentation of an object's location, corresponds a continuous appearance vector Zapp NO 1. It) of size that describes the object.
The likelihood function p(x Zloc: Zapp) is parameterized in compositional manner. For each im- age, the visual representation, or sprite, of the ith object generated from zapp by a shared function Saprite. Each sprite is then convolved with 2-dimensional Kronecker delta that is the one-hot rep- resentation of the object's location. Finally, the resulting n tensors are added together to give pixel-wise parameters À (A1,...,Ap) of the distribution:
=
Σ (Zupp) * Zooc
(2ppp)
(8)
(4)
where * denotes 2-dimensional discrete convolution.
Inference model. The approximate posterior for a data point x has the following form:
(Zppp Zloe, n n|x x) ((Zapp Zlocy nI, x) (Zoo |,, x)q(n) x)
n
II
q app
Ton
x)q
inc
(4)
( loe
J
q(n |x)
Since each data point is assumed independent, the lower bound (ELBO) the marginal log likeli- hood is the sum of the ELBO for each data point, which is:
log p() 2 E.y [log p(x Zlocs Zapp)] + E. log P(Zioc; Zapp, n)
(6)
(Z Zapp: xx)
where the first term is the likelihood and the second is the negative Kullback-Leibler (KL) divergence between (Ziue, Zupp: x) and p(Ziocy Zapp: n.). The different terms of the KL divergence can be
derived as Appendix A and estimated by Monte Carlo sampling.
Two inference networks compute appearance and location feature maps, happ and hjoc- both having size D like the input. The inference model for the number of objects n is a categorical distribution parameterized by function of the location features Scount (hioc). Object locations follow cate- gorical distributions without replacement parameterized by logits hlce The vector at location Zooc in the feature map happ represents the appearance parameters for object i.e. the mean and log variance of (Zapp Zoo x). The overall inference process can be summarized as follows:
q(12) x) = Cat(n; M + 1, Seoone(hhac)))
(7)
(6) 200
10c
{2
(0) $200 Cat(z D, (0)
1 n
(8)
2

---

Under review as a conference paper at ICLR 2020
1 app o (6) (i)
happ [ood
spp
q app loc x)
(Zzope upp diag(o app (P))
(10)
where by v(ex] denote the kth element of (with eg standard basis vector), and the probability vector for location sampling at each step is computed iteratively:
(E) b
softmax(hiod) (ID E
Loo
(J)
b by
(11)
The expectations in the variational bound handled as follows: For sampling. This gives a biased gradient estimator, but in practice this has not affected inference on n. We use the Gunbel-soffmax relaxation (Jang et al. 2016: Maddison al. 2016) for Zinc and the Gaussian reparameterization trick for Zapp-
discrete categorical
3 RESULTS
In all experiments we use the same architecture for LAVAF. See Appendix B for all implementation details. Briefly, the sprite decoder in the generative model p(x Zluc, Zupp/ consists of a fully nected layer and a convolutional layer, followed by 3 residual convolutional blocks. The appearance and location inference networks are fully convolutional, thus the feature maps happ and o have the same spatial size as x.
As baseline to compare the proposed model against used VAE implemented fully convolu- tional network: the absence of fully connected layers makes it easier to preserve spatial information, thereby allowing the model achieve a higher likelihood and generalize more naturally. Morcover, this choice of baseline is closer in spirit to our model than VAE with fully connected layers.
We evaluate LAVAE on Multi-MNIST and multi-dSprites data sets consisting of 200k images with to objects in each image. 190k images are used for training, whereas the remaining 10k are left for evaluation. We generated an additional test set 10k images with 7 objects each. Examples with more than 3 objects are never used for training.
3.1 MULTI-MNIST
Each image in the multi-MNIST data set consists a number of statically binarized MNIST digits scattered at random (avoiding overlaps)
black canvas of size 64 x 64. The digits first rescaled from their original size (28 x 28) to 15 15 by bilinear interpolation, and finally binarized by rounding. When generating images, digits are picked from a pool of either 1k or 10k MNIST digits. We call the two resulting data sets multi-MNNST-IJ and multi- Multi-MNNSTT-O6 depending on the number of MNIST digits used for generation. We independently model each pixel as Bernoulli random variable parameterized by the decoder:
p(x Zine, Zapp)
II d=1
p( Blocs Zapp)
Bernoulli(zaa Aa) d=1
II
(12)
where the parameter vector À defined in Section is the pixel-wise mean of the Bernoulli distribu- tion.
Figure [ qualitatively shows the performance of LAVAE on Multi-MNIST-IIN 10k test images. The inferred location all objects an image are summarized by a black canvas where white pixels indicate object presence. For each of those locations, the model infers a corresponding appearance latent variable. Each digit on the right is generated by faprite from one of these appearance variables.
Samples from the prior shown in Figure 2 note that the number of generated objects is consistent with the training set, since the prior p(n) is learned. Quantitative evaluation results are shown in Table I and compared with VAE baseline. The inferred object count correct almost all images of all test sets-even with a held-out number objects and across tested random seeds.
A fundamental characteristic of disentangled representations is that change in single representa- tional factor should correspond to a change in a single underlying factor of variation (Bengio et al.] Locatello et al. 2018). By demonstrating that the appearance and location of single objects
2013
3

---

Under review as a conference paper at ICLR 2020
7
8
8
4
7
7 3
3
3
7
Figure 1: Inference and reconstruction on test images. For each image, from left to right: in- put, reconstruction, summary of inferred locations, sprites generated from each appearance latent variable.
//
7
Figure 2:
Generated samples.
Left: images generated by LAVAE from its prior
P(Zioe [Zppp n)p(n), where p(n) is learned from data. Right: images generated by the baseline from its prior p(z).
can be independently manipulated in the latent space, the qualitative disentanglement experiments in Figure prove that objects are disentangled from another, and the appearance and location of each object.
3.2 MULTI-DSPRITES
The multi-dSprites data set is generated similarly to the multi-MNIST ones, by scattering a number of sprites black canvas of size 64 x 64. Here, the sprites simple shapes in different col- ors, sizes, and orientations, as in the dSprites data set (Higgins et al., 2017]] Matthey et al.) (017). The shape of each sprite is randomly chosen among square, ellipse, and triangle. The maximum sprite size is 19 x 19, and each sprite's 's scale is randomly chosen among 6 linearly spaced values in [0.6, 1.0]. The orientation angle of each sprite is uniformly chosen among 40 linearly spaced values in 2T). The color is uniformly chosen among the 7 colors such that the RGB values are saturated (either or 255), and least one of them is not O. This means that each color component of each pixel is binary random variable and can be modelled independently as in the multi-MNIST case (leading to 3/D terms the likelihood, instead of D)).
Figure shows images generated by sampling from the prior in LAVAE and in the VAE baseline, where we can appreciate how our model accurately captures the number and diversity of objects in the data set. Figure 5 shows instead example of inference on test images, explained in Section[.1] Finally, as we did for multi-MNIST, we performed disentanglement experiments where, starting from the latent representation of test image, manipulate the latent variables. In Figure we change the order of the location variables to swap objects, whereas Figure 7 shows the effect of separately altering the location and appearance latent variables of single object.

---

Under review as a conference paper at ICLR 2020
Table 1: Quantitative results multi-MNIST and multi-dSpriits test sets. The log likelihood lower bound is estimated with 100 importance samples. Note that the MNIST-10K dataset is more com- plex task than MNIST-IK because the model has capture larger variation in appearance. The object count accuracy is measured as the percentage of images for which the inferred number of objects matches the ground truth (which is not available during training).
multi-MNIST-Ik
multi-MNIST-IOK
multi-dSprites
-logp(x)
n acc.
logg(xx) S
acc.
logp(x) S
7t acc.
LAVAE
42.2 + 1.1
99.9%
60.4 + 2.8
99.7%
43.9 + 1.0
99.9%
baseline
51.6 + 0.6
64.6 0.8
45.8 0.7
9
5
9
0
6
9
9
9
3
3
3
3
Figure 3: Disentanglement experiments test images. Objects are represented independently of each other, and their location and appearance are disentangled by design. Left: Latent traversal on one of the location variables. Top right: Reordering the sequence {z or equivalently of {zapp}; }. leads to objects being swapped (top row: original reconstruction; bottom row: swapped objects). Bottom right: In each row, latent traversal on one of the appearance variables along one dimension.
3.3 GENERALIZING TO MORE OBJECTS
As mentioned above, LAVAE correctly infers the number of objects in images from the 7-object versions our data sets, despite the fact that it was only trained on images with up to objects. Furthermore, by representing each object independently and disentangling location and appearance of each object, it accurately decomposes 7-object scenes and allows intervention as easily as in images with fewer objects. Figure demonstrates this the 7-object version of the multi-MNIST- 10k data set. Finally, in Figure 9 we show images generated by LAVAE after modifying the prior p(n) be uniform in 54.
RELATED WORK
Our work builds on recent advances probabilistic generative modelling, in particular variational autoencoders (VAEs) [Kingma and Welling 2013 Rezende al. 2014). One of the methods clos- est to our work in spirit is Attend Infer Repeat (AIR) (Eslami et al. 2016). which performs explicit object-wise inference through recurrent network that iteratively attends to object at time. limitation of this approach however is that has not been shown to generalize well to a larger number of objects. Closely related to work is also the multi-entity VAE (Nash al. 2017).
5

---

Under review as a conference paper at ICLR 2020
Figure 4:
Generated samples.
Left: images generated by LAVAE from its prior
piZioe (Zapp n)p(n), where p(n) learned from data. Right: images generated by the baseline from its prior p(z).
Figure 5: Example of inference and reconstruction on multi-dSprites test images. From left to right: input, reconstruction, summary of inferred locations, sprites generated from the inferred ap- pearance latent variables.
in which multiple objects independently modelled by different latent variables. The inference process does not include an explicit attention mechanism, and uses instead a spatial KL map as proxy for object presence. Each object's latent decoded into full image, and these aggregated by an element-wise operation, thus the representation of each object's location and appearance are entangled. In the same spirit, the recently proposed generative models MONet (Burgess al. 2019)
Figure 6: Object swap. Changing the order of location (or equivalently appearance) latent variables leads to objects being swapped. Top row: original reconstruction of test image; bottom row: objects are swapped by manipulating the latent variables.

---

Under review as a conference paper at ICLR 2020
Figure 7: Left: Location traversal. Latent traversal on location variables in test image with objects. Right: Appearance traversal. Latent traversal on appearance variables: changing Zapp for some i along one latent dimension corresponds changing appearance attributes of one specific object. The appearance latent space is only partially disentangled. Here we show examples where change in latent dimension leads to a change of a single factor of variation (rows 2, 4, 6). However, in row there is a change both in color and shape, and in row 3 both in color and scale.
A
2
4
2
2
2
D
D
6
5
6
4
5
5
4
5
I
g
8
4
12
'
g
2
2
2
2
2
6
S
o
2
4
4
2
4²
4
8
0
0
0
C
4
6
3
7
7
7
f
7
2
2
2
2
2
2
2
0
6
-
0
0
0
3
5
3
5
3
5
3
5
8
3
5
5
t
1
t
Figure 8: Disentanglement with more objects. Latent traversal on location (top) and appearance (bottom) variables, multi-MNISTT-100 test images containing objects. LAVAE still correctly infer the scene's structure and reconstruct it, allowing intervention on location or appearance of single objects.
and IODINE (Greff et al. 2019) learn without supervision to segment the scene into independent and interpretable object-based parts. Although these are more flexible than AIR and can model more complex scenes, the representations they learn of object location and appearance are not disentan- gled. All methods cited here are likelihood based so they can and should be compared in terms of test log likelihood. We leave this for future work.
Other unsupervised approaches to visual scene decomposition include Neural Expectation Maxi- mization (Greff et al. 2017] Van Steenkiste et al. 2018), which amortizes the classic EM for spa- tial mixture model, and Generative Query Networks (Eslami et al. 2018), that learn representations of rich 3D scenes but do not factor them into objects and need point-of-view information during training. Methods following the iison-as--nverss, -nverse-rrappic paradigm (Poggio et al.] 1985 Yuille
and Kersten 2006) learn structured, object-centered representations by making strong assumptions on the latent codes or by exploiting the true generative model (Kulkarni et al. 2015) Wu et al. 2017
Tian et al. 2019). Non-probabillistic approaches scene understanding include adversarially trained

---

Under review as a conference paper at ICLR 2020
[7
7
5
2
t
4
8
7
2
3
V
8
3
4
7
2
7
Figure 9; Generation with fixed number of objects. Images generated by LAVAE from modified prior which n takes value 5 or 6 with probability 1/2.
generative models (Pathak et al. 2016) and self-supervised methods (Doersch et al. 2015 Vondrick ct al. 2018). These, however, do not explicitly tackle representation learning, and often have rely on heuristics such as region masks. Finally, examples of supervised approaches are semantic and instance segmentation (Ronneberger et al. [2133] He et 2017 [égou et al. [017]] [iu et al. 2018), where acquiring labels for training is typically expensive, and the focus not on learning structured representations.
5 CONCLUSION
We presented LAVAE, probabilistic generative model for unsupervised learning of structured, compositional, object- based representations of visual scenes. We follow the amortized stochastic variational inference framework, and approximate the latent posteriors by inference networks that are trained end-to-end with the generative model. On multi-MNIST and multi-dSprites data sets, Thanks
LAVAE learns without supervision to correctly count and locate all objects in to the structure of the generative model, objects are represented independently of each other, and the location and appearance of each object are completely disentangled. We demonstrate this in qualitative experiments, where we manipulate location or appearance of single objects independently the latent space. Our model naturally generalizes visual scenes with many more objects than encountered during training.
These properties make LAVAE robust to scene complexity, opening up possibilities for leveraging the learned representations for downstream tasks and reinforcement learning agents. However, in order to smoothly transfer to scenes with semantically different components, the appearance latent space should be disentangled. Since in this work focused robust model based disentanglement of location and appearance, more work should be done to fully assess and improve disentanglement properties in the appearance model. Another natural extension this work is apply it complex natural images and 3d scenes.
REFERENCES
Y. Bengio, A. Courville, and P. Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 3588):1798-1822 2013.
C. P. Burgess, L. Matthey, N. Watters, R. Kabra, I. Higgins, M. Botvinick, and A. Lerehner. Monet: Unsupervised scene decomposition and representation. arXiv preprint arXiv: 1901./1399 2019.
8

---

Under review as a conference paper at ICLR 2020
X. Chen, Y. Duan, R. Houthooft, J. Schulman, I. Sutskever, and P. Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in neural information processing systems, pages 2172-2180. 2016.
C. Doersch, A. Gupta, and A. A. Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE International Conference on Computer Vision, pages 1422-1430, 2015.
A. Eslami, N. Heess, T. Weber, Y. Tassa, D. Szepesvari, G. Hinton, et al. Attend, infer, repeat: Fast scene understanding with generative models. In Advances in Neural Information Processing Systems, pages 3225-3233, 2016.
S. A. Eslami, D. J. Rezende, F. Besse, F. Viola, A. S. Morcos, M. Garnelo, A. Ruderman, A. A. Rusu, I. Danihelka, K. Gregor, al. Neural scene representation and rendering. Science, 360 104-111,, 2018.
M. Garnelo and M. Shanahan. Reconciling deep learning with symbolic artificial intelligence: rep- resenting objects and relations. Current Opinion in Behavioral Sciences, 29:17-23, 2019.
K. Greff, S. van Steenkiste, and J. Schmidhuber. Neural expectation maximization. Advances in Neural Information Processing Systems, pages 6691-6701. 2017.
K. Greff, L. Kaufmann, Kabra, N. Watters, Burgess, Zoran, L. Matthey, M. Botvinick, and A. Lerchner. Multi-object representation learning with iterative variational inference. arXiv preprint arXiv:/903.000
2019.
K. He, G. Gkioxari, P. Dollár, and Girshick. Mask r-cnn. Proceedings of the IEEE international conference on computer vision, pages 2961-2969, 2017.
I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. ICLR, 2(5)66, 2017.
E. Jang, S. Gu, and Poole. Categorical reparameterization with gumbel-softuax. arXiv preprint arXiv:1611.01144 2016.
S. Jégou, M. Drozdzal, D. Vazquez, A. Romero, and Y. Bengio. The hundred layers tiramisu: Fully convolutional densenets for semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 11-19, 2017.
H. Kim and A. Mnih. Disentangling by factorising. arXiv preprint
2018.
D. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv: 1312.6114, 2013.
T. D. Kulkarni, W. F. Whitney, P. Kohli, and J. Tenenbaum. Deep convolutional inverse graphics network. In Advances in neural information processing systems, pages 2539-2547, 2015.
S. Liu, L. Qi, H. Qin, J. Shi, and J. Jia. Path aggregation network for instance segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8759- 8768, 2018
F. Locatello, S. Bauer, M. Lucic, S. Gelly, B. Schölkopf, and O. Bachem. Challenging com- mon assumptions in the unsupervised learning of disentangled representations. arXiv preprint arXiv.1811.123599 2018
C. Maddison, A. Mnih, and Y. W. Teh. The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint arXiv:1611.007/1 2016.
I. Matthey, I. Higgins, D. Hassabis, and A. Lerchner. dsprites: Disentanglement testing sprites dataset.
2017.
C. Nash, A. Eslami, C. Burgess, I. Higgins, D. Zoran, T. Weber, and P. Battaglia. The multi-entity variational autoencoder. NIPS Workshops, 2017.
9

---

Under review as a conference paper at ICLR 2020
D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, and A. A. Efros. Context encoders: Feature learning by inpainting. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2536-2544, 2016.
T. Poggio, Torre, and C. Koch. Computational vision and regularization theory. Nature, 317 (6035):314-31,, 1985.
D. J. Rezende, S. Mohamed, and D. Wierstra. Stochastic backpropagation and approximate infer- ence deep generative models. arXiv preprint arXiv:1401.4082 2014.
O. Ronneberger, Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pages 234-24 Springer, 2015.
E. S. Spelke. Principles of object perception. Cognitive science, 14(1):29-56, 1990.
E. S. Spelke. Where perceiving ends and thinking begins: The apprehension of objects in infancy. In Perceptual development in infancy, pages 209-246. Psychology Press, 2013.
E. S. Spelke and K. D. Kinzler. Core knowledge. Developmental science, 10(1):89-96, 2007.
Y. Tian, A. Luo, X. Sun, K. Ellis, W. T. Freeman, J. B. Tenenbaum, and J. Wu. Learning to infer and execute 3d shape programs. arXiv preprint arXiv: 2019.
Van Steenkiste, M. Chang, K. Greff, and Schmidhuber. Relational neural expectation maximiza- tion: Unsupervised discovery of objects and their interactions. arXiv preprint arXiv:1802 10353, 2018.
C. Vondrick, A. Shrivastava, A. Fathi, Guadarrama, and K. Murphy, Tracking emerges by col- orizing videos. In Proceedings of the European Conference Computer Vision (ECCV), pages 391-408, 2018.
J. Wu, J. B. Tenenbaum, and Kohli. Neural scene de-rendering, In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 699-707, 2017.
A. Yuille and D. Kersten. Vision as bayesian inference: analysis by synthesis? Trends in Cognitive Sciences, 10(7):301-308, 2006.
10

---

Under review as a conference paper at ICLR 2020
A KL DIVERGENCE
The KL Eq. equation can be expanded as follows:
ZaZppp Zioe: n, x) () Zioo n2 x) x)
DEL (g(Z)oe zapp: n x) || p(Zioos Zapp: 11)) = Byy log
P(Zapp (2)p(( (Zioo 12)p(1)
(Zapp Zloos n2, x)
log a(1 x)
Eyg Muppi
log
+ Botn [xx)
p(Zapp 12)
p(n)
q (Zo x)
+ 1i nt x)
log
p(Z 12)
= B2 [,x)g(n x)
Σ DEL. (u(2lip
(2 Mloe x) p(asst)))
+ DKL. Prl,(n( [((n/x) p(n))
+ Eg(n x) nx,cc)
log
{ ( ge
log p(
20or I 2 (5)
(1)
Staocseee y22))
where all expectations can be estimated by Monte Carlo sampling.
B IMPLEMENTATION DETAILS
The input image x is fed into a residual network with 4 blocks having 2 convolutional layers cach. Every convolution followed by a Leaky ReLU and batch normalization. A final convolutional layer outputs the feature map happ with 2L channels. The output of an identical but independent residual network is fed into a 3-layer convolutional network with one output channel that represents the location logits hloc. The logits are multiplied by constant factor 7 = 4 ~ 1/2 log D. See below for more details about the rescaling of logits.
The number of objects is then inferred by a deterministic function fcount of the location logits. This function filters out points in hioc that are not local maxima, then counts the number of points above the threshold 4y, If ń is the inferred number of objects in the scene, fcouut outputs the corresponding one-hot vector, and the distribution q(n x) = 5(n - n) deterministically takes the value n. We fix the maximum number of objects per image to M 10.
in The locations (2 ii= are iteratively sampled from a categorical distribution with logits hioc- where all the logits of the previously sampled location are set to low value to prevent them from being sampled again. At the ith step, when the ith object's location sampled, the corresponding feature vector in happ is interpreted as the means and log variances of the L components of the ith appearance variable. When sampling from the categorical distribution, we use Gumbel-Softne single-sample gradient estimation. The temperature parameter 7 is exponentially annealed from 0.5 to 0.02 in 100k steps. The latent space for each appearance variable has L = 32 dimensions.
for
The sprite-generating function /sprite is convolutional network that takes as input a sampled ap- (8) pearance vector Zapp- The first part of the network consists of fully connected layer, convolu- tional layer, and a bilinear interpolation. The vector zapp is then expanded and concatenated the resulting tensor along the channel dimension. The second part of the network consists of 3 residual blocks with 2 convolutional layers each, followed by final convolutional layer with sigmoid ac- tivation. Leaky ReL.U and batch normalization are used after each layer, except in the last residual block. The size of a generated sprite for the multi-MNIST data set is 17 17, for multi-dSprites 21 x 21. The mean of the Bernoulli output is then computed from these sprites explained above.
We optimized the model with stochastic gradient descent, using Adamax with batch size 64. The initial learning rate was le-4 or 5e-4 for the location inference network (for the multi-MNIST and multi-dSprites data sets, respectively) and le-3 for the rest of the model. In the location inference net, the learning rate was exponentially decayed by factor of 100 in 100k steps. The model param- eters are about 500k, split almost evenly among location inference, appearance inference, and sprite generation.
11

---

Under review as a conference paper at ICLR 2020
Training warm-up. In practice found it beneficial to include warm-up period at the be- ginning of training, in which an auxiliary loss is added to the negative ELBO. We train a fully- convolutional VAË in parallel with model, and take the location-wise KL in the latent space as a rough proxy for object location, as suggested by Nash et al.] (017)). 'he additional loss is the squared distance between hioc and the KL. map. The network interring object location is therefore initially encouraged to mimic a function that is a (rather rough) approximation of the location, and then it is fine-tuned. Intuitively, we are biasing sampling of Zioe favor of information-rich loca- tions, which significantly speeds up and stabilizes training. To stabilize training, we also found it beneficial to randomly force steps) and then linearly decays to 0.1 in the following 30k steps.
during training with probability that during warm-up (30k
The auxiliary VAF is implemented as fully convolutional network, in which the encoder consists of 3 residual blocks with downsampling between blocks, and final convolutional layer. The decoder loosely mirrors the encoder. The latent variables are arranged as 3D tensor with size 11 x x 8. The auxiliary loss is added to the original training loss for warm-up phase of 30k steps. In the subsequent 30k steps, the contribution of this term to the overall loss is linearly annealed to O.
Rescaling location logits. Assume have &o objects and the logits of the inferred location distribution are either O or h. The probability of one of the "hot" locations after softmax is e²/(D) - + ke*)
where t is constant that should not depend on D. and should be close to 1/k. Solving for h we get / = log + log(U) - k) log(1 - kt). If D is large enough and k small enough, we have log(D - k) = log D, and the constant log is relatively small in mag- nitude. Because of our assumptions on the logits and on f, we can write log(1 kt) x O, and therefore the high logits It should be approximately proportional to log D. Thus, using the same fully convolutional network architecture for multiple image sizes, the logits should be scaled by factor proportional to log D. The threshold for counting objects should likewise follow this rule of thumb.
Baseline implementation. As baseline we use a fully convolutional (-VAE, where the latent vari- ables are organized as 3D tensor. Being closer in spirit to our method, this leads to fairer compar- ison. Furthermore, this inductive bias makes it easier to preserve spatial information. Indeed, this empirically leads better likelihood and generated samples than a VAE with fully connected layers (and a 1D structure for latent variables), and allows to model more naturally varying number of objects in scene. The encoder consists of convolutional layer with stride 2, followed by resid- ual blocks with 2 convolutional layers each. Between the first two pairs of blocks, a convolutional layer with stride 2 reduces dimensionality. The resulting tensor has size 64 x x 8, and a final convolutional layer brings the number of channels to 2dvAE/8 where dvAF is the number of latent variables. The decoder architecture takes as input a sample of size dvAE/8² /82 x 8 x 8 and outputs the pixel-wise Bernoulli means. Its architecture loosely mirrors the encoder's. 's, with convolutions being replaced by transposed convolutions, except for the last upsampling operation which consists of bilinear interpolation and ordinary convolution. All convolutions and transposed convolutions, except for the ones between two residual blocks, are followed by Leaky ReLU and batch normal- ization. The last convolutional layer is only followed by a sigmoid nonlinearity. In our experiments we used dyAE= = 1024 and we linearly annealed /3 from 0 to I in 100k steps. The number of model parameters about 1M.
C RESULTS ON MULTI-MNIST I
Here we present additional visual results on the multi-MNIST-- 1k data set, similar those discussed in the main text.
12

---

Under review as a conference paper at ICLR 2020
Figure 10:
Prior samples. Left: images generated by sampling from LAVAE's prior
PiZ [Z)ppZapp n)p(n), where p(n) is learned from data. Right: images generated by sampling from the baseline's prior p(z).
7
C7
0
2
7
7
7
5
5
6
7
7
;
8
7
Figure 11: Disentanglement experiments on test images with more objects than in the training regime. Objects are represented independently of each other, and their location and appearance are disentangled by design. Top left: Reordering the sequence {zioc] J equivalently of (zapp } leads objects being swapped (top row: original reconstruction; bottom row: swapped objects). Top right: Latent traversal on one of the 7 location variables. Bottom: Latent traversal on one of the appearance variables (along 3 different dimensions).
13


Reviews:
[{"review_id": "SygdvztCKB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nAfter reading reviews and comments I have decided to confirm the initial rating.\n\n===================\n\nThe work presents an approach to encode latent representations of objects such that there are separate and disentangled representations for location and appearance of objects in a scene.  The work presents impressive qualitative results which shows the practical use of the proposal on multi-mnist and multi-dSprites. \n\nWhile the use of inference networks proposing positions for the network as a means of improving the disentanglement is clever and seems novel, though not unlike inference sub-networks which are well-known in conditional generation, the evaluation is not up to a standard I can endorse, resulting in a recommendation to reject. \n\nDespite the interesting qualitative results, I will have to quote the work in saying, “All methods cited here are likelihood based so they can and should be compared in terms of test log likelihood. We leave this for future work.”.  Indeed the cited works should have been evaluated against, especially Greff et al. 2019, Nash et al, 2017, and Eslami et al, 2016, which are all very similar.  As written it’s impossible to tell whether this work actually improves over the state of the art, we only have the constructed baseline (which as a community we all know clearly would not have worked). \n\nA figure showing the relevant submodules of the network architecture and what they do in relation to the overall method would be helpful to understand the pipeline and how the inference network relates to the whole.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "H1xyP11TYH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a probabilistic generative model for identifying the location, type and colour of images placed within a grid. The paper is in general well written and presents a large number of visual images demonstrating that the approach works. The main concerns with the paper are as follows:\n\n1) The implementation details for the work are relegated to an appendix. As this is a core concept for the work one would expect this to be presented in the main body of the work.\n\n2) Although there are a large number of visual images there is little in the way of analytical analysis of the work. As a particular concern the authors claim that figure 3 ‘prove’ that objects are disentangled. From Maths 101 I remember it being drummed into us that ‘proof by example is not a proof’.\n\n3) The are parts of the process which are not explained for example, why is the following conducted? ‘The digits are first rescaled from their original size (28 × 28) to 15 × 15 by bilinear interpolation, and finally binarized by rounding.’\n\n4) I would also like to know why this is called disentangling as the whole process prevents the ‘icons’ from overlapping. Disentangle normally refers to things which are at least overlapping. There are examples of works in this area.\n\n5) The example cases are simple. One would expect at least one real-world example.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1xFs1anFH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a compositional generative model of images, where the image is described by a variable number of latent variables. Moreover, the latent variables are disentangled, in the sense that they represent different parts of the scene, and where appearance and location are described separately. While the generative model is very similar to AIR [1], the central claim of the paper is that the proposed inference scheme can generalize to a much higher number of objects than seen during training, which is demonstrated empirically, and with which AIR struggles. Better generalization is achieved by removing the recurrent core of AIR and replacing it with a fully-convolutional inference network, which predicts appearance vectors at every location in the feature map. The appearance vectors are then stochastically sampled without replacement according to a location distribution. Unlike AIR, this approach does not model the object scale.\n\nI recommend REJECTing this paper. While the improved generalization performance is a useful property, it has been achieved previously and in a very similar fashion by SPAIR [2], which follows a similar model design. SPAIR still uses an RNN, while the proposed approach does not, but this is only a small simplification and does not warrant publication at a top tier conference. There are no other contributions in this paper. Additionally, on the one hand, the experimental evaluation is insufficient: the proposed approach is compared only against a fully-convolutional VAE, while very similar models like AIR [1], SPAIR [2], SuPAIR [3] are not considered. On the other hand, the experimental section focuses on the disentanglement of representations, which is a) evident from the model construction and b) achieved in all previous models. \n\nThe paper is clearly written, and the presented generative model is interesting. Having said that, the problem that this paper addresses is mostly solved in [2] and [4]; also both [2] and [4] scale the general approach introduced in [1] to much higher number of objects (in the hundreds) and more general settings (real images, atari games).\n\n[1] Eslami et. al., “Attend, Infer, Repeat:...”, NIPS 2016.\n[2] Crawford and Pineau, “Spatially Invariant Unsupervised Object Detection with Convolutional Neural Networks”, AAAI 2019.\n[3] Stelzner et. al., “Faster Attend-Infer-Repeat with Tractable Probabilistic Models”, ICML 2019.\n[4] Jiang et. al., “Scalable Object-Oriented Sequential Generative Models”, arXiv 2019.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a VAE approach where the model learns representation while disentangling the location and appearance information. The reviewers found issues with the experimental evaluation of the paper, and have given many useful feedback. None of the reviewers were willing to change their score during the discussion period. with the current score, the paper does not make the cut for ICLR, and I recommend to reject this paper. 
Decision: Reject