Variational Neural Cellular Automata
Authors: Rasmus Berg Palm
Miguel González Duque
Shyam Sudhakaran
Sebastian Risi
Venue: ICLR 2022 Poster (2022)
Keywords: Neural Cellular Automata
Cellular Automata
Self-Organization
Generative Models

Abstract: In nature, the process of cellular growth and differentiation has lead to an amazing diversity of organisms --- algae, starfish, giant sequoia, tardigrades, and orcas are all created by the same generative process.
Inspired by the incredible diversity of this biological generative process, we propose a generative model, the Variational Neural Cellular Automata (VNCA), which is loosely inspired by the biological processes of cellular growth and differentiation. Unlike previous related works, the VNCA is a proper probabilistic generative model, and we evaluate it according to best practices. We find that the VNCA learns to reconstruct samples well and that despite its relatively few parameters and simple local-only communication, the VNCA can learn to generate a large variety of output from information encoded in a common vector format. While there is a significant gap to the current state-of-the-art in terms of generative modeling performance, we show that the VNCA can learn a purely self-organizing generative process of data. Additionally, the self-organizing nature bestows the VNCA with some inherent robustness against perturbations in the early stages of growth.

Published as a conference paper at ICLR 2022
VARIATIONAL NEURAL CELLULAR AUTOMATA
Rasmus Berg Palm', Miguel González-Duque" SSham Sudhakaran & Sebastian Risi Creative AI Lab IT University of Copenhagen Copenhagen. Denmark raramm,, Pmigd, ssebr feitu.de dk, #ahyamsnair@@proooomall.
ABSTRACT
In nature, the process of cellular growth and differentiation has lead to an amazing diversity of organisms algae, starfish, giant sequoia, tardigrades, and orcas are all created by the same generative process. Inspired by the incredible diversity of this biological generative process, we propose generative model, the Variational Neural Cellular Automata (VNCA), which is loosely inspired by the biological processes of cellular growth and differentiation. Unlike previous related works, the VNCA is proper probabilistic generative model, and we evaluate it accord- ing to best practices. We find that VNCA learns to reconstruct samples well and that despite its relatively few parameters and simple local-only communica- tion, the VNCA can learn to generate a large variety of output from information encoded in common vector format. While there is significant gap the cur- rent state-of-the-ard the VNCA can learn a purely self-organizing generative process of data. Addi- tionally, we show that the VNCA can learn a distribution of stable attractors that can recover from significant damage.
terms of generative modeling performance, we show that
INTRODUCTION
The process of cellular growth and differentiation capable of creating an astonishing breadth of form and function, filling every conceivable niche in our ecosystem. Organisms range from complex multi-cellular organisms like trees, birds, and humans to tiny microorganisms living near hydrothermal vents in hot, toxic water at immense pressure. All these incredibly diverse organisms are the result of the same generative process cellular growth and differentiation.
Cellular Automata (CA) are computational systems inspired by this process of cellular growth and differentiation, where "cells" iteratively update their state based on the state of their neighbor cells
9
9
9
9
9
9
9
7
9
9
9
9
9
9
9
9
9
9
9 9
9 9
9
9
9
9
9
9
Figure I: VNCA self-organized growth and damage recovery. Left: The VNCA has learned
self-organising generative process that generates faces, from initial random seed of cells from (o 1). Time goes top-down and left-to-right. Every eight steps all the cells duplicate, which can be seen the diagonal. Right: A damage recovery sequence for MNIST digit.

---

Published as a conference paper at ICLR 2022
2g p())
Z 2 X
21 (2-())
Z x 44
x 2
(
doubling
Figure 2: VNCA overview. Left: Generative process. 20 is sampled from p(=) and acts initial x 2 seed of cells, which grows through series of K doubling NCA steps. |z| denotes the size of the latent space, which identical the size of the cell states. Finally, the last cell hidden conditions the parameters of the likelihood distribution p( p(z|z). Middle: Each doubling NCA step consists of number of NCA steps followed by a doubling operation. At each NCA step, the cells can only communicate with their immediate neighbors. Right: The doubling operator doubles the cell grid as depicted, where the color indicates the state vector of each cell.
and the rules of the CA. Even with simple rules, CA can exhibit complex and surprising behavior with just four simple rules, Conway's Game of Life Turing complete. Neural Cellular Automata (NCA) CA where the cell states vectors, and the rules of the CA are parameterized and learned by neural networks (Mordvintsev et al., 2020 Nichele et al. 2017 Staaney & Miikkulanenn 2003 Wulff & Hertz 1992). NCAs have been shown learn generate images. 3D structures, and even functional artifacts that are capable of regenerating when damaged (Mordvintsev et al. 20200 Sudhakaran et al. 2021). While these results are impressive, the NCAs can only generate (and regenerate) the single artifact it is trained on, lacking the diverse generative properties of current probabilistic generative models. In this paper, we introduce the VNCA, an NCA based architecture that addresses these limitation while only relying on local communication and self-organization.
Generative modeling is fundamental task in machine learning, and several classes of methods have been proposed. Probabilistic generative models are attractive class of models that directly define distribution over the data p(z). which enable sampling and computing (at least approximately) like- lihoods of observed data. The average likelihood on test data can then be compared between models, which enables fair comparison between generative models. The Variational
(VAE) is seminal probabilistic generative model, which models the data using latent variable model, such Pa (=<x)))p(() where p(z) a fixed prior and p=(x]]> is a learned decoder (Kingma Welling 2013 Rezende et al., 2014). The parameters of the model is learned by maximizing
that p(x)
the Evidence Lower Bound (ELBO), lower bound on log p(x). which can be efficiently computed
using amortized variational inference (Kingma Welling 2013 Rezende al. 2014).
On a high level, our proposed generative model combines NCAs with VAES, by using a NCA as the decoder in a VAE. We perform experiments with a standard NCA decoder and novel NCA architecture, which duplicates all cells every M steps, inspired by cell mitosis. This results in better generative modeling performance and has the computational benefit that the VNCA only computes cell updates for the living cells at each step, which are relatively few early in growth. While our model inspired by the biological process of cellular growth and differentiation. to be an accurate model of this process and is naturally constrained by what can efficiently compute.
not intended
Other works have explored similar directions. The VAE-NCA proposed in Chen & Wang (2020), is actually, despite the name, not a VAE since does not define a generative model or use variational inference. Rather is auto-encoder that decodes set of weights, which parameterize NCA that reconstructs images. NCAM (Ruiz et al. 2020) similarly uses an auto-encoue antooen to decode weights for NCA, which then generates images. In contrast both of these, method generative
2

---

Published as a conference paper at ICLR 2022
model, uses single learned NCA, and samples the initial state of the seed cells from a prior p(z). StampCA Frans 2021) also learns an auto-encoder but similarly encodes the initial hidden NCA state of a central cell. None of these generative models, in the sense that it's not possible to sample data from them, nor do they assign likelihoods to samples. While previous works all show impressive reconstructions, it is impossible to evaluate how good generative models they really are. In contrast, we aim to offer a fair and honest evaluation of the VNCA as a generative model.
Graph Neural Networks (GNNs) can be seen as a generalization of the local-only communication in NCA, where the grid neighborhood is relaxed into an arbitrary graph neighborhood (Grattarola et al.) 2021]. GNNs compute vector-valued messages along the edges of the graph that are accumulated at the nodes, using a shared function among all nodes and edges. GNNs have been successfully used to model molecular properties (Gilmer et al. 2017); reasoning about objects and their interactions Santoro et al., 2017), and learning to control self-assembling morphologies (Pathak et al. 2019).
Powerful generative image models have been proposed based on the variational auto-encoder frame- work. A common distinction is whether the decoder is auto-regressive not, i.e., whether the de- coder conditions its outputs on previous outputs, usually those above and to the left of the current pixel (Van den Oord et al. 2016; [Salimans et al. 2017). Auto-regressive decoders are so effec- tive at modelling p(æ) directly that they have the tendency to ignore the latent codes thus failing to reconstruct images well (Alemi et al.] 2018). Additionally, they expensive to sample from since sampling must be done one pixel at a time. Our decoder is not auto-regressive, thus cheaper to sample from, and more similar e.g., BIVA (Maaløe et al.] (019) NVAE (Vahdat & Kautz) 2020). Contrary to BIVA, NVAE and other similar deep convolution-basee VAES, our VNCA de- fines a relatively simple decoder function that only consider the immediate neighborhood of a cell (pixel) at each step and is applied iteratively over a number of steps. The VNCA thus aims to learns a self-organising generative process.
NCAs and oelf-organizing systems in general. represent an exciting research direction. The re- peated application of simple function that only relies on local communication lends itself well to a distributed leaderless computation paradigm, useful in, for instance, swarm robotics (Rubenstein al. (2012). Additionally, such self-organizing systems are often inherently robust to perturbations [Mordvintsev et al. 2020; Najarro & Risi 2020 Tang & Ha 2021 Sudhakaran et al. (021I). In the VNCA, we show that this robustness allows us to set a large fraction of the cells' states to zero and recover almost perfectly by additional iterations of growth (Fig. D right). By introducing an NCA that is a proper generative probabilistic model, we hope to further spur the development of methods at the intersection of artificial life and generative models.
VARIATIONAL NEURAL CELLULAR AUTOMATA
The VNCA defines a generative latent variable model with p(2) and pe(x122, where p(z) = [21/ = 0, Σ = 7) is a Gaussian with zero mean and diagonal co-variance and po (x)2) is learned decoder, with parameters 0. The model is trained by maximizing the evidence lower bound (ELBO),
logp(x) Dxa ll p<(|))
p(x)) + Ez=gs(=ta) log po (a|=),
where, qo is a learned encoder, with parameters Φ, which learns to perform amortized varia- tional inference. Since DRt. 2 O. the right hand side is a lower bound on log p(z) and increasing it either increases log p(x) or decreases DKL (2)((z))) Il ((z(z)), which is how closely the amortized variational inference matches the true unknown posterior p(2 (z). By using the reparameterizztion trick, the sampling, z ~ (2(x))) is differentiable, and both Φ and 0 can be learned efficiently with stochastic gradient ascent (Kingma & Welling 2013). The DKI. term on the right hand side can be weighted with a parameter /3 to control whether the VAE should prioritize better reconstructions (3 < 1), or better samples (3 1) (Higgins et al. 2016). For 3 > this is still (looser) lower bound on log p(z).
The decoder of the generative process, Po is based on NCA (Fig. 2 The NCA defines re- current computation over rector-valued "cells" in grid. At step of the NCA, the cells compute an additive update to their state, based only on the state of their neighbor cells in a 3 3 neighborhood,
Zi,t Zi.t-1 ua (1
(I)
3

---

Published as a conference paper at ICLR 2022
8
7
3
2
R
2
2
8
7
8
2
g
7
8
0
6
9
5
5
7
8
5
Figure 3: Left: Test set reconstructions. Right: Unconditional samples from the prior. The VNCA 84.23 nats evaluated with 128 importance weighted samples.
achieves p(r) 2
7
9
3
4
3
3
3
3
3 3
3
3
4
4 4
3
3
3
3
3
Figure 4: Self-organising MNIST, with the seed cell states sampled from the prior. Each 32 32 sub- image shows the digit after step of NCA or doubling. Time flows top-to-bottom and left-to-right. The doubling steps can be seen on the diagonal. Note: this shows sample averages for clarity.
where vg is a learned update function, index denote a single cell and (11) denotes the indexes of the 3 x 3 neighborhood of cell i, including All cells updated in parallel. In practice, uo is efficiently implemented using Convolutional Neural Network (CNN) (Gilpin) 2019 with a single 3 x 3 filter followed by number of 1 filters and coneriicasssies. The initial grid 20 consists of single sample from the latent space repeated in 2 x 2 grid.
traditional NCA defines whether cell is alive or dead based on the cell state and its neighbors' cell states (Mordvintsev et al.) (2020). This allows the NCA to "grow" from a small initial seed of alive cells. Our NCA uses different approach inspired by cell mitosis. Every M steps of regular NCA updates all the cells duplicate by creating new cells initialized to their own current state to the right and below themselves, "pushing" the existing cells out of the way as they do so. This efficiently implemented using a "nearest" type image rescaling operation (Fig. 2 right). After K doubling and M(K + 1) steps of NCA updates, the final cell states ZT condition the parameters of the likelihood distribution. For RGB images, (Salimans et al., 2017), such that the 10L first channels are the parameters of the discrete logistic mixture"anne and for binary images Bernoulli likelihood with the first channel of each cell being log
discrete logistic likelihood with L mixtures
The update function is single 3 x 3 convolution, followed by four residual blocks (He et al. 2016). Each block consists of 1 convolution, Exponential Linear Unit (ELU) non-linearity (Clevert et al. 2015) and another 1 x convolution. Finally, a single 1 convolution maps to the cell state size. This last convolution layers' weights and biases are initialized zero prevent the recurrent computation from overflowing. The number of channels for all convolutional layers, the size of the latent space, and the cell state all 256. The total number of parameters in the NCA decoder is approximately 1.2M.
Since we are primarily interested in examining the properties of the NCA decoder, the encoder (=]2)) is a fairly standard deep convolutional network. It consists of single layer of 5 x 5 con-
We use the implementation from http

---

Published as a conference paper at ICLR 2022
1
7
8
8
O
O
2
2
2
2
7
7
7
7
4
4
9
8
8
8
8
8
5
5
5
3 (
3 3
3
3
(b)
(c)
Figure 5: Exploring the latent space of VNCA. Fig- 5a shows several linear interpolations between
random digits in the latent space of a VNCA trained on binarized MNIST. Figs. [5] and |c] show the result reducing the dimensionality of 5000 digits chosen at random for the VNCA and for the deep convolutional baseline respectively. Notice how the latent space of VNCA has more t-SNE structure and cleaner separation of digit encodings. Note: this shows sample averages for clarity.
volution with 32 channels followed by four layers of 5 x 5 convolution with stride 2, where each of these layers has twice the number of channels as the previous layer. Finally, the down-scaled image is flattened and passed through a dense layer which maps to twice the latent space dimensionality. The first half encodes the mean and the second half the log-variance of a Gaussian. All convolutional layers are followed by ELU non- oon-linearities. For detailed model definition, see the appendix.
3 EXPERIMENTS
Except where otherwise noted, we use a batch size of 32, Adam optimizer (Kingma & Ba 2014) 10-4learning rate, L = 1 logistic mixture component, clip the gradient norm to 10 (Pascanu al. 2013) and train the VNCA for 100.000 gradient updates. Architectures and hyper-parameters were heuristically explored and chosen based on their performance on the validation sets and the memory limits of our GPUs. We use a single sample to compute the ELBO when training and measure final log-likelihoods on the test set using 128 importance weighted samples, which gives a tighter lower bound than the ELBO (Burda et al.) 2015). We report log likelihoods in nats summed over all the dimensions for MNIST and bits per dimension (bpd) for CelebA per convention for these datasets. The conversion between the two is bpd natss/ WWW'C' C' In 2), where HWC is height, width and channels respectively. When visualizing the output of the generative models we show x ~ p(x z), Le. samples from the generative model or the average, when the noise of the individual samples detracts from the analysis. When we show averages we note this in the figure captions.
3.1 MNIST
For our first experiment we chose MNIST, since is widely used in the generative modeling litera- ture and a relatively easy dataset. Since the VNCA generates images that have height and width that are powers of 2, we pad the MNIST images to 32 32 with zeros and use K doublings. We use the statically binarized MNIST from Larochelle & Murray (201T).
The VNCA learns a generative model of MNIST digits and also capable of accurately reconstruct-
ing digits given a latent code by the encoder (Fig. It achieves log p(z) 2 -84.23 nats evaluated with 128 importance weighted samples. While not ttate- which is around 777 nats (Sinha & Dieng 2021 Maaløe et al. (2019), it is decent generative model. In comparison, the complex auto-regressive PixeICNN decoder achieves -8.3 (Van den Oord et al. 2016).
Fig. 4 shows the self-organization and growth of the digits. Initially, they are just uniform squares, but after doubling twice to size of 8 x 8, it appears that the process is iterating towards a pattern. After doubling, the NCA steps shows signs of "enhancing" the image, moving towards smoother, more natural-looking digit. The VNCA has learned a self-organizing process that converges to diverse naturally looking hand-written digits.
Latent space analysis We visualize in Fig. 55 the latent space of our VNCA by selecting 5000 random images from the binarized MNIST dataset, encoding them, and reducing their dimensional-
5

---

Published as a conference paper at ICLR 2022
ity using I-Stochastic Neighbour Embedding (t-SNE) (van der Maaten & Hinton 2008 We find that our dimensionally-reduee latent space cleanly separates digits, especially when compared to a deep convolutional baseline in Fig. 5c This baseline is composed of the same encoder the VNCA, fol- lowed by symmetric deconvolutional decoder (see the appendix for details). Further, Fig. 1a..5assooss four interpolations between the encodings of digits selected at random from the dataset.
3.2 CELEBA
The CelebA dataset contains 202,599 images celebrity faces (Liu et al. 2015 and has been extensively used in the generative modeling literature. It is significantly harder dataset to model than MNIST and allows us to fairly compare the VNCA against state-of of-the-art generative methods. We the version with images rescaled 64 64.
Figure 6: CelebA results. Left: Test reconstructions. Right: Unconditional samples from the prior. The VNCA achieves 4.42 bits per dimension on the test set evaluated with 128 importance weighted samples.
The VNCA learns to reconstruct the images well, but the unconditional samples do not look like faces, although there are some faint face-like structures in some of the samples (Fig. 6 The VNCA achieves 4.42 bits per dimension evaluated with 128 importance weighted samples. This far from state-of-the-art which is around 2 bits per dimension, using diffusion based models and deep hierarchical VAEs (Kim et al. 2021 Vahdat & Kautz 2020).
Figure 7: VNCA CelebA results when trained with 100. Left: Test set reconstructions. Right: Unconditional samples from the prior. Note: this shows sample averages for clarity. This version achieves much worse bits per dimension 2 5.40, but visually the samples are much better. Note: this shows sample averages for clarity.
Sceing that the reconstructions are good, but the samples are poor, we train VNCA with 100, which strongly encourages the VNCA to produce better samples at the cost of the reconstructions. See fig. for the results. This VNCA technically much worse generative model with bits per dimension 2 5.40, but visually the samples are much better. Individually the samples are noisy due to the large variance of the logistic distribution, so we visualize the average sample. For individual samples from the likelihood see the appendix.
With the improved visual samples, we can visualize the growth (Fig. Here we see that the VNCA has learned self-organizing generative process that generates faces from random initial seeds. Similar to the MNIST results, the images go through alternating steps of doubling and NCA steps that refine the pixellated images after a doubling.

---

Published as a conference paper at ICLR 2022
Figure 8: Growth of CelebA samples from the prior from a VNCA trained with B 100. Note: this shows sample averages for clarity.
Resilience to damage during growth Due to the iterative self-organising nature of the compu- tation, the original NCA architectures were shown to be resilient in the face of damage, especially trained for it (Mordvintsev al.) 2020). To examine the VNCAs inherent resilience to damage during growth, we perform an experiment in which we set the right half of all cell states to zero, at varying stages of growth. We compared the VNCA resilience to damage against an baseline using an identical encoder and a deep convolutional VAE with 5 up-convolutions as the decoder. To our surprise the baseline VAE decoder recover approximately similarly well from this damage, at similar stages in the decoding process. See Fig. IT in the appendix for the results.
3.3 OPTIMIZING FOR RESILIENCE TO DAMAGE
Inspired by the damage resilience shown in [Mordvintsev et al. 2020 we investigate how resilient can make the VNCA to damage by explicitly optimizing for it. In order to recover from damage we need to grow until the damage is recovered. However, due to the doubling layers, this would quickly result in an image larger than the original data. As such we modify the VNCA architecture by removing the doubling layers. Since this non-doubling VNCA variant does not changes the resolution we can run for any number of steps. The NCA decoder of this variant almost identical to the original NCA architecture proposed in (Mordvintsev et al. 2020). The only differences are that (1) we consider all cells alive at all time, since we found that the aliveness concept proposed in Mordvintsev et al. (2020) made training unstable, (2) we seed the entire H x W with instead of just the center and (3) we update all cells all the time. The only difference to the doubling VNCA
is that remove the doubling layers, and seed the entire grid with 20, eq. 1 still describes the update rule. See the appendix for a figure of this non-doubling variant.
Pool and damage recovery training. In order optimize for stability and damage recovery adapt the pool and damage recovery training procedure proposed in Mordvintsev et al. (2020) to our generative modelling setting. During training half of every training batch sampled from pool of previous samples and their corresponding cell states after running the NCA, 27. Half of these pool states 2T are subjected to damage by setting the cell state to zero in random H/2 x W/2 square. After the NCA has run for T steps, we put all the new 27 states in the pool, shuffle the pool, and retain the first P samples, where corresponds to the fixed size our pool. We use P = 1024 for our experiments. See appendix A.3 for pseudo-code. The effect is that half of every training batch normal, and half corresponds to running the NCA for an additional T steps on a previous sample, of which half are damaged. For the half that corresponds running the NCA for additional T steps, the gradient computation is truncated, corresponding to truncated backpropagation through time. As such, are longer maximizing the ELBBO exactly, but in practice it still seems to work.
7

---

Published as a conference paper at ICLR 2022
MNIST. We train non-doubling VNCA with pool and damage training MNIST. It uses very simple update network consisting of a single 3 x convolution, ELU non con-linearity, and then a single 1 convolution. For each batch sample T uniformly from 32 to 64 steps. It is trained with a batch and latent size of 128. It attains worse log-likelihoods than the doubling VNCA with log p(x) 2 -90.97 nats, however, the VNCA learns distribution of attractors stable over hundreds of steps which recover almost perfectly from damage (Fig. 9 While this looks like in-painting or
3 4
2
2
4
original
9
7
8
8
8
8
9
4
2
9
9
8
8
8
8
8
8
damaged
8
8
8
8
8
8 8
8
3
4 2
4
3
4
3
3
8
8
3
recovered
Figure Damage recovery on MNIST samples. Left: From top to bottom: samples from the model, samples with damage applied, recovered samples after 7 NCA steps. Right: details of the damage recovery process showing sampled uniformly from 32 to 64). Note: this shows sample averages for clarity.
36 steps of NCA growth on the leftmost damaged sample (T' is
data-impuus inttaimputation,, it is not, and we should be careful not compare to such methods. When doing in-painting the method only has access to the pixels of the damaged image, whereas here, each pixel corresponds to vector valued cell state, that potentially encode the contents of the entire image. Indeed this seems to be the case, since the VNCA almost perfectly recover the original input.
CelebA. We also train a non-doubling VNCA with pool and damage recovery on CelebA. The hyper-parameters are identical the doubling variant except the data is rescaled 32 x 32, run for 64 to 128 steps sampled uniformly, and use 128 hidden NCA states instead of 256, due memory limitations. Similarly MNIST, the non-doubling CelebA variant model, achieving 5.03 bits per dimension, on the easier 32 x 32 version of CelebA, but learns to recover from damage and learns attractors that are stable for hundreds of NCA steps (Fig. I0).
worse generative
original
damaged
recovered
Figure 10: CelebA Damage Recovery. Left: From top to bottom: samples from the model, samples with damage applied, recovered samples after T NCA steps. Right: details of the damage recovery process, cropped for brevity, showing the first 24 and the last steps on the lefumost damaged sample. For all 121 steps, the appendix. Note: this shows sample averages for clarity.
4 DISCUSSION
Inspired by the biological generative process of cellular division and differentiation, proposed the VNCA, a generative model based on organiignn & iterative computation, relying only on local cellular communication. We showed that, while far from state-of-the-art, is capable of learning
8

---

Published as a conference paper at ICLR 2022
oega-ongan generative model of data. Further, showed that it can learn distribution of stable attractors of data that can recover from damage.
We proposed two variants of the VNCA. One based on doubling of cells at fixed intervals, inspired by cell mitosis, and a non-doubling variant. The non-doubling variant is better suited for damage recovery and long-term stability but it is worse generative model. We hypothesize that (1) the doubling operation is a good inductive bias on image data, and that (2) it allows the cells to agree on global structure early on. The communication in the early growth steps effectively enables the doubling VNCA to communicate over, what becomes, large distances, while the non-doubling NCA needs many steps to achieve something similar. The doubling VNCA also has the benefit that it is computationally cheaper since it is only in the last stage of growth that the computations are performed on the entire image. Compared to the non-doubling VNCA this results in a large speedup and lowers the memory requirements dramatically. How to combine the damage resilience of the non-doubling variant with the better generative modelling of the mitosis inspired variant an exciting avenue for future research.
One challenging aspect the VNCA is that all the parameters of the decoder are the update func- tion, which is iteratively applied M + 1) times (K doublings and M NCA steps per doubling). This means that adding more parameters, and thus more computation, to the decoder is M(K + 1) times more computationally costly than in standard non-iterative decoder. In other words, could have a decoder for CelebA with M(K + 1) = 48 times more parameters at the same compu- tational cost. For this reason, the update function
use has few parameters, just 1. 1.2M, compared to state-of-the-ar decoders which routinely have hundreds of millions, allowing much more powerful decoders (Maaløe al. 2019 Salimans et al. 2017 Vahdat & Kautz, 2020).
Another set of generative models that generates and reconstructs data in an iterative process are 2015), which have recently achieved state-of-the-art results
diffusion models DSohl-Dickstein in datasets like CelebA (Ho et al. 2020; Kim et al., 2021). While the VNCA decoding processes involves applying a function repeatedly, the decoding processes diffusion models involves iter- atively sampling from Markov chain of approximate distribution. In the future, the connection to diffusion models should be explored in more detail.
ACKNOWLEDGEMENT
This work was funded by DDF-Researde Projectl grant (9131- 00042B), the Danish Ministry of Education and Science, Digital Pilot Hub and Skylab Digital.
REPRODUCIBILITY STATEMENT
The code to reproduce every experiment is available at github. wwtp//ssusserrrgramn/aaoooa used to run all experiment, including all syper-parameters. are recorded in the immutable git revision history. The revisions corresponding to the experimental results in the paper are highlighted in the repository README.
The code
REFERENCES
Alexander Alemi, Ben Poole, Ian Fischer, Joshua Dillon, Rif A Saurous, and Kevin Murphy. Fixing broken elbo. International Conference on Machine Learning, pp. 159-168. PMLR, 2018.
Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov, Importance weighted autoencoders. arXiv preprint arXiv:1509 3099000519 2015.
Mingxiang Chen and Zhecheng Wang. Image generation with neural cellular automatas. arXiv preprint 040:02001000949, 2020.
Djórk-Aré Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint 151.v:151..07255 2015.
Kevin Frans.
Stampca:
Growing emoji
with URL
conditional http: httt::/kvfrans... com/
neural cellu-
lar automata.
kofrans.com, 2021.
st.ampca condeccc--ondie inc.nniiiioaloo
luallllarree auularrautomata/

---

Published as a conference paper at ICLR 2022
Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine learning, pp- 1263-1272. PMLR, 2017.
William Gilpin. Cellular automata convolutional neural networks. Physical Review E, 100(3): 032402, 2019.
Daniele Grattarola, Lorenzo Livi, and Cesare Alippi. Learning graph cellular automata. Advances in Neural Information Processing Systems, 34, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. 2016.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilissic models. arXiv preprint arxiv:2006. 1/239, 2020.
Dongjun Kim, Seungjae Shin, Kyungwoo Song, Wanmo Kang, and Il-Chul Moon. Score matching model for unbounded data score. arXiv preprint arXiv:2/06.05527, 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint 1rXiv:1312.6114, 2013.
Hugo Larochelle and lain Murray. The neural autoregressive distribution estimator. In Proceed- ings of the fourteenth international conference on artificial intelligence and statistics, pp. 29-37. JMLR Workshop and Conference Proceedings, 2011.
Ziwei Liu, Ping Luo. Xiaogang Wang. and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Lars Maaløe, Marco Fraccaro, Valentin Liévin, and Ole Winther. Biva: very deep hierarchy of latent variables for generative modeling. arXiv preprint
2019.
Alexander Mordvintsev, Ettore Randazzo, Eyvind Niklasson, and Michael Levin. Growing neural
cellular automata. Distill, 2020. doi: 10.23915/distie 5/di0ttll.00003.. https://issill.ppbb2000//
Elias Najarro and Sebastian Risi. Meta-Leamiig through Hebbian Plasticity in Random Networks. In Advances in Neural Information Processing Systems, 2020. URL https: /tttpp://aaxiv.... org/
abs lbbs/207..o 02686
Stefano Nichele, Mathias Berild Ose, Sebastian Risi, and Gunnar Tufte. Ca-neat: evolved compo- sitional pattern producing networks for cellular automata morphogenesis and replication. IEEE Transactions on Cognitive and Developmental Systems, 10(3))687-700, 2017.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International conference on machine learning, pp. 1318. PMLR, 2013.
Deepak Pathak, Chris Lu, Trevor Darrell, Phillip Isola, and Alexci A Efros. Learning to trol self-assembling morphologies: study of generalization via modularity. arXiv preprint 190.055600. 2019.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and ap- proximate inference in deep generative models. In International conference on machine learning, pp. 1278-1286. PMLR, 2014.
Michael Rubenstein, Christian Ahler, and Radhika Nagpal. Kilobot: A low cost scalable robot sys- tem for collective behaviors. In 2012 IEEE international conference on robotics and automation, pp. 3293-3299 IEEE, 2012.
10

---

Published as a conference paper at ICLR 2022
Alejandro Hernandez Ruiz, Armand Vilalta, and Francesc Moreno-Noguer. Neural cellular automata manifold. arXiv preprint 11X1:22006.12155, 2020.
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma. Pixelcn+++: Improving the pixelenn with discretized logistic mixture likelihood and other modifications. arXiv preprint arXiv:1701.05517 2017.
Adam Santoro, David Raposo, David GT Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. A simple neural network module for relational reasoning. arXiv preprint arXiv: 0e1.11766..1222 2017.
Samarth Sinha and Adji B Dieng. Consistency regularization for variational auto-encoders. arXiv preprint arXiv:2 101.1055..44541 /4859, 2021.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Francis Bach and David Blei (eds.). Pro- ceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 2256 2265, Lille, France, 07-09 Jul 2015. PMLR. URL https::/prrceediing
pilrspres. v37 htt3///ehh-d 1tt......ssiiiiimmmmmm...m.m.m..
Kenneth O Stanley and Risto Mükkullanen. A taxonomy for artificial embryogeny. Artificial life, (2):93-130, 2003.
Shyam Sudhakaran, Djordje Grbic, Siyan Li, Adam Katona, Elias Najarro, Claire Glanois, and Sehastian Risi. Growing 3d artefacts and functional machines with neural cellular automata. arXiv preprint arX1v:2103.08737, 2021.
Yujin Tang and David Ha. The sensory neuron as transformer: fermutation-inaarinn neural net- works for reinforcement learning. arXiv preprint arXiv:2109.02869, 2021.
Arash Vahdat and Jan Kautz. Nvae: A deep hierarchical variational autoencoder. arXiv preprint arXiv:2007.03898, 2020.
Aaron Van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In International Conference on Machine Learning, pp. 1747-1756. PMLR, 2016.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of Ma-
chine Learning Research, 9:2579-2600 2008. UtL/http://www. jml r. org/papers/v9/
wardermaatenn8an ht.l]
N Wulff and J A Hertz. Learning cellular automaton dynamics with neural networks. Advances in Neural Information Processing Systems, 5:631-633 1992.

---

Published as a conference paper at ICLR 2022
A APPENDIX
A.1 DAMAGE DURING GROWTH
(a) VNCA
(h) Deep convolutional decoder baseline
Figure 11: Unconditional samples, after being exposed damage varying stages of growth. The leftmost image is not exposed to any damage. The next six images represent damage applied at steps t [0, 8, 16, 22, 40) corresponding to the initial state, and the steps just before each doubling. (b) shows the equivalent process in a deep convolutional baseline: the leftmost image is left intact, and the following columns present the result of damaging the hidden state before each up-convolution. The second and third columns are the same, since both represent the equivalent to damaging the before the first duplication. Note: both these figures show sample averages for clarity.
compare against resilience to damage the decoding/doubling process (see Fig, M.. we trained a deep convolutional VAE on the same dataset, fitting the same distribution.
CelebA baseline
This is the specification of the model as a PyTorch module:
VAB
(encoder): Sequential
(0): Conv2d 32, ternel_sizer- (5, 5), stride= (1, 1), padding= (2, 2)
(1): ELU (alphap==1100)
(2): C32v 64, kernel_sizo=055 5), stride= (2, 2), padding= (2, 2)
(3): (LU(alpha=1.0)
(4): C64v2d(64, 128, kernel_siza=(5, 5), stride= (2, 2), padding= (2, 2)
(5): (aU(alpha-1.0)
(6) : (1nv2d 256, kernel_size- (5, 5), stride- (2, 2), padding- (2, 2))
(7): (ld((alpa=11.0)
(8): Conv2d(256, 512, size= 5), stride= 2), padding= (2,
(9): ELUtalpha=1.0)
(10): Flatten(start_ end dim=-1)
(11): Linear (in_features-8112) out_features-512, bias-True)
(decoder_linear)) (
(0): Linear (i___eeturte,,,
cut_features=4096, bias-True)
(1): ELU (alpha-111)
(decoder): Sequential
(0): convTranspose22 (1024, 512, kernel_size=(5, (5, 5), stride= (2, 2) padding= (2, 2), output__padingggttt 1))
(1): ELUtalpha=1.0)
(2) ConvTranspose2d 256, kernel_size-(1" (5, 5), stride- (2, 2) padding (2, 2), output_padding--o (1, 1))
12

---

Published as a conference paper at ICLR 2022
(3): ELU (alpha=1.0)
(4): ConvTranspose2d (256, 123, kernel_size-(5, 5), stride= 2),
padding= (2, 2), (udputt_eeddingg11, 1))
(5): (alphaa110)))
(6): ConvTranspose2a122 (128, 64, kernel_size= (5, 5), stride- 2), padding= 2), cutput_padding=1l, 1))
(7): EIU(alpha-1.0)
(8) : ConvTranspose2d (64, 32, kernel_size=($5 (5, 5), stride= (2, 2),
(2 2), output_padding (1, 11)
(9): ELU(aipha=1.0)
(10): Conviransposeed 10, kernel_size- (5, 5), stride- 1)
padding- (2, )
During training, we used the ELBO loss with 3 = 100 since we faced similar issues when trying to get sensible unconditioned samples. worth remarking that this baseline doubles the image via up-convolutions the same number of times the VNCA used for this experiment.
A.2 NON-DOUBLING VNCA VARIANT
Z 1x1
20 p(2)
4
X
W
2o repeat (20)
Z
H W
NCA(2o)
H X
/ x W
NCA(2T
7t++1
X
2T)
Figure 12: Non-doubling VNCA variant architecture
13

---

Published as a conference paper at ICLR 2022
A.3 POOL AND DAMAGE TRAINING
import random
2
3
def pool_ [oo___dgg_trainpp___ data, batch_size, pool_size, optim):
4
pool =
5
while not converged
x random, (data, batch_size)
7
n_ batch_size /// 2
8
?_poc1 None
9
if (pool) n_pool sampless Sample from the pool
x pool, 2 pool :o samples]
haa_pool_samppp
x x_pool
12
2_pool damage_half (2_pool
13
e_z_0, 2_0 encode
if z_pool:
z samples:] z_pool
P __x_given_a_f, z_t (z_0) Run the NCA
17
= (p__ggvp__i L.backwards
_z_0, p_2_0)
18
optimstep())
20
Add new states to pool peol append (x, ?_T)) random, shuffll (pool) pool pool [:pool_size]
21
22
23
14

---

Published as a conference paper at ICLR 2022
A.4 NOTO EMOJI
The Noto Emoji font contains 2656 vector graphic emojis This dataset allows for comparison with previous NCA auto-encoder work, which uses it (Frans 2021 Chen Wang 2020 Ruiz et al. 2020 Mordvintsev et al.] 2020). Also, the emoji are diverse, which tests the ability of the VNCA to generate a large variety of outputs encoded in a single latent vector format. However, due to the small amount of samples and the very large variation, it is not a great dataset for evaluating generative model. We train a VNCA on 64 x 64 emoji examples using K doublings and using 20% (531) of the images as test set.
The VNCA learns to reconstruct the inputs well similarly to the auto- auto-encoder NCAs (Frans) 2021 Chen & Wang 2020 Ruiz et al., 2020) as can be seen on the left but fails to generate realistie-looking samples, as can seen on the right
(Fig. 13). It does particularly well on the faces and human emojis, probably because they make up a large part of the dataset and are the most regular. While reconstructs most images well, it fails on some, in particular the signs and symbols that are unique, not resembling any other emojis in the dataset, e.g.. a highway, a safety pin, and a microscope. While the VNCA here fails to learn a good impressed with its ability generate such diverse outputs from single
generative model, common vector encoding, using a simple orga--igg process.
NG
Figure 13: Noto Emoji results. Left: Test reconstructions. Right: Unconditional samples from the prior. The VNCA achieves 3.09 bits per dimension on the test set evaluated with 128 importance weighted samples.
15

---

Published as a conference paper at ICLR 2022
A.5
MODEL SPECIFICATION
The detailed model specification can be seen below in the standard PyTorch Module format.
VNCA
(encoder): Sequentiall
(0): (onv2d 32, kernel_size=15, 5), stride= (1, 1), padding- (2, 2))
(1) EL0(alpha=1.0)
(2) : (32v2d 64, kernel_size- 5), stride= (2, 2), padding= (2, 2)
(3) (alphaa-1110)
(4): Conv2d 128, kernel_size=55, 5), stride= (2, 2), padding- (2, 2))
(5): ELU(alpha=1.01
(6): Coav2d(128, 256, (ezeel_ 5), stride= (2, 2), padding= (2, 21)
(7): ELUtalpha=1.0)
(8) (256, 512, kernel_size- (5, 5), stride- (2, 2), padding (2, 2))
(9): (Iu(alpaa11.0)
(10): <latten(start_ddim)) end_dim=-1)
(11): Linear (infeattinaat out fut_features--12, bias-True)
(nca): : DataParallel
(module) NCA
(updatc_net): Sequential
(0): (256 256, kerncl_sizo-(3, (3, 3), stride= 1), padding- (1, 1))
(1): Residual( (delegate): (
(0): Conv2d 256, kernel_size-11, (1, 1), stride- 1))
(1): (1D(a1)pa-1.00)
(2): Conv 256, sernel_size-re 1), stride= 11)
(2): Residual
(delegate): Sequential
(0): C256 256, kernel_size-(1) (1, 1), stride- 1))
(1) ELU(alpha=1.0)
(2) (256 256, (1 1), stride= 11)
(3): Residual(
(delegate): Sequential
(0): (256, 256, kernel_size- (1, 1), stride- 1))
(1): (alphapa-11
(2): (256, 256, kernel_size-(1 (1, 1), stride-(1, 1))
(4): Residual (delegate): Sequential
(0): (256, 256, kernel_size=1, 1), stride-(1, 1))
(1): ELU(alpha=1.0)
(2): (256 256, size,l_ 1), stride= 1))
(5): (256 256, (izeel, 1), stride=(1, 1))
A.6 DEEP CONVOLUTIONAL BASELINES
MNIST baseline Fig. we compared our VNCA to a deep convolutional VAE baseline. Here is the specification of that model using PyTorch's module format:
VAR
(encoder): Sequentiall
16

---

Published as a conference paper at ICLR 2022
(0): Conv2d 32, Bernel_size-fe 5), stride=(1, (1, 1), padding= (4,
(1) EL0(alpha=1.0)
(2) : C32v2d 64, kernel_size=15, 5), stride= (2, 2), padding- (2, 2))
(3) (alphaa11000)
(4): (64,2d 128, kernel_size= 5), stride= (2, 2), padding- (2, 2))
( (a1
(6) (128, 256, kernel_size-15, (5, 5), stride- (2, 2), padding- (2, 2)
(7): ELU(alpha=1.0)
(8): Conv2d(256, 512, kernel_size=(5 5), stride= 2), padding= (2, 21
(9) ELUialpha=1.0)
(10): Flatten
end_dim=-1]
(11): Linear Linear (ia__feattrres-2006 out out_featuues-256 bias-True)
(decoder (inoderr_inner): Sequential
(0): (f fet_feetture=2249, bias-True)
(decoder): Sequential
(0): ConvTransposc2a55. (512, 256, kernel_sizc=(55 (5, 5), (2, 2),
padding= (2, 2), output_padding-11, 1))
(1): (ad(haph--1.0)
(2) ConvTranspose2d (256, 128, kernel_size- (5, 5), stride- (2, 2), padding- 2), output_padding-11, 11]
(3): ELU(alpha=1.0)
(4): CenvTranspose2a(133, 64, pernel_size=re 5), stride= 2),
(2, 2), output putput_paddingg-t 11)
:5): (alpha-1100)
(6) ConvTransposee2d (64, 32, kernel_size- (5, 5), stride- 2),
(2dding- 2), output_paddiigg11, 1))
(7) : (aU(alph==1.0)
(8) ConvTranspose2211,, 1, kernel seanelssize(($ (5, 5), stride= (1, 1), padding= (4, 4))
We believe this is a fair baseline, since contains 9.5M trainable parameters, in comparison to the VNCA used for this experiment, which contains around 5M.
A.7 CELEBA UNCONDITIONAL SAMPLES
A.8 ADDITIONAL EXPERIMENTS
We tried several variations of the VNCA that, perhaps surprisingly, were not better. We report them here so that future research can take them into account.
Size We tried deeper (up to residual layers) and wider (512) update function networks, with no effect. A bigger latent space of 512 didn't 'ttt seem to matter. bigger NCA hidden state of 512 also didn't 't seem to matter. We tried T, (1,4) logistic mixtures, and I. = 1 was better.
Non-strict NCA architectures. We tried replacing the x convolution layers with means the NCA is no longer relying only on local communication, but this also wasn't 't much better. We also tried a version that had separate update network per scale, i.e., networks. Surprisingly this was only slightly better.
which
Batch size Batch size did, surprisingly, seem matter. Bigger batches consistently trained a lot better. With batch size 4 it was impossible to train. Batch size 32 seemed sufficient and was chosen as a trade-off between speed and quality.
We tried I) A GRU update function, 2) _ + (b))((z)), where was learned scalar and a is the sigmoid and 3) version which held half of the cell hidden state constant throughout all updates. None were better.
Update function:
A.9 CELEBA RECOVERY
A. NON-DOUBLING GROWTHI
17

---

Published as a conference paper at ICLR 2022
Figure 14: CelebA unconditional individual samples from VNCA trained with 8 100.
18

---

Published as a conference paper at ICLR 2022
Figure 15: CelebA growth of unconditional sample from VNCA trained with 3 100.
19

---

Published as a conference paper at ICLR 2022
Figure 16: CelebA recovery of damage showing T' = 121 NCA steps. Note: this shows sample averages for clarity.
20

---

Published as a conference paper at ICLR 2022
9
9
3
8
3
3
3
3
3
3
3
3
3
3
3
8
8
3
3
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
3
Figure 17: Growth CelebA and MNIST digits for the non-doubling VNCA variant.
21


Reviews:
[{"review_id": "gq8DIwLbdY", "reviewer": "Reviewer_kwgv", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n\n- I enjoyed the paper and, although I have some comments, I think the work can be interesting for the ICLR community. There is growing interest in (neural) cellular automata, and this work is a valuable addition to the topic. \n\n- The paper addresses a limitation of typical morphogenesis approaches with NCA, which is that they are usually trained to generate a single image/state. This work is a first step towards learning NCA that generalize to different target states. \n\n- The experimental results are interesting despite having some limitations. The VNCA can reconstruct the images well when starting from a latent sample produced by the encoder. The limitations only arise when using the model for generation starting from a random sample from the prior, but the authors also show how to partially address these issues (using the $\\beta$-VAE approach). \n\n- The paper is well written and all concepts are explained adequately.\n\n**Weaknesses**\n\nI have a few suggestions and questions that I hope can help the authors improve the paper. \n\n1. Have the authors tested their model without the upsampling part? \n    \n    The base NCA of Mordvintsev et al. is a sufficiently powerful architecture for morphogenesis (starting from a single nonzero pixel) and it is not immediately clear why the upsampling step is necessary for the VNCA. An explanation of what led the authors to this choice (besides the link with mitosis) would be interesting.\n\n2. As noted by the authors, the upsampling step forces the output to have a given dimension, multiple of $2^M$. \n\n    Related to this, have the authors investigated some settings in which the images are exactly as big as the output of the VNCA, without zero padding? This would give us interesting insights into the behaviour of the VNCA at the boundaries. \n\n3. I have a few questions regarding the architecture:\n\n    - What is the activation function of the last layer of the NCA? \n    The authors only mention ELU but the last activation should ensure that the output of the model is a meaningful image (so a sigmoid or rescaled tanh).\n    - How was the NCA architecture chosen? Why four residual blocks? Why two 1x1 convolutions in each block? \n    - Why is there no activation after the last convolution of each residual block? \n    - How was the architecture of the encoder chosen?\n    - How were the number of channels, learning rate, batch size, gradient clipping, $K$, $L$, and $\\beta$ determined? \n      I'm not asking the authors to perform an in-depth tuning, but if the hyperparameters were chosen arbitrarily it is important that the readers know it. \n    - How do 1.2M parameters compare to Mordvintsev at al.'s NCA? \n\n5. In Figure 4, the final generated digits are significantly smoother than those in Figure 3, right panel. Were the examples of Figure 4 cherry-picked somehow? \n\n6. There are no details about the training procedure. For example, Mordvintsev et al. use a replay memory to ensure the stability of the generated pattern. Do the authors have a comparable training procedure in place? \n\n    NCA are not well known in the community (yet), so papers on the subject must a good job of explaining the base concepts until they are more widely adopted. \n\n7. Two references could/should be added in Section 1: \n\n    - Wulff and Hertz, \"Learning cellular automaton dynamics with neural networks\", Neural Information Processing Systems (1992). \n       This is one of the first papers to model CA transition rules with neural networks (they even use CNNs!) and a good reference to cite for completeness. \n\n    - Grattarola et al., \"Learning Graph Cellular Automata\", Neural Information Processing Systems (2021). \n       This is a very recent paper (published after the ICLR submission date) that explores the use of graph neural networks to learn cellular automata on graphs (they also do morphogenesis). It seems like a good match for the last paragraph of Page 2, which has no references. \n\n**Other comments**\n\n- Some readers may not be familiar with the \"bits per pixel\" metric. An explanation of what the metric represents could be useful. \n\n- Regarding this claim: \"It is not immediately clear what a generative model that takes damage and super-resolution into account would look like\". \nI don't think that NCA are that different from any typical neural network. Couldn't we train the VNCA as any other super-resolution or de-noising autoencoder?\n\n    For example, the de-noising could be done as they do in the NCA paper, and the super-resolution by fixing two different generations at which the loss is computed, using the target image and a downscaled version of it. \n\n- Have the authors performed a latent space analysis to see if different classes of images get clustered in some particular way?\n\n- There is no mention of code anywhere in the submission. I encourage the authors to make the code available if the paper gets accepted (ideally it should have been included with the submission, but I understand that it is not mandatory and I have not taken this into consideration when evaluating the paper). \n\n----------\n\n**After rebuttal**: The authors have addressed my comments. I have raised my score from \"6: marginally above the acceptance threshold\" to \"8: accept, good paper\".", "summary_of_the_paper": "This paper introduces a variational generative model based on Neural Cellular Automata (NCA). The model is called the Variational NCA (VNCA).\n\nThe VNCA is designed for images: \n\n- The encoder is a typical convolutional neural network. \n- The decoder is an NCA that iteratively refines the image, alternating convolutions and upsampling up to the desired image size. The upsampling process is loosely inspired by mitosis in living cells. \n\nThe authors perform morphogenesis experiments on three datasets: MNIST, Noto Emoji, and CelebA.\nThe results are good on MNIST, less so on the other two datasets, although there is clear evidence that the model can learn to generate meaningful images.\n\nThe authors also perform an experiment to see if the VNCA is robust to perturbations (occlusions) and show that the model has a reasonable degree of robustness even without ever seeing any perturbations at training time.", "summary_of_the_review": "- The authors propose a variational neural cellular automaton, which learns to generate images by iterating the transition rule.\n- The paper is interesting, with good results, and a good fit for ICLR.\n- The paper solves an interesting problem on the topic of neural cellular automata. \n- There are some doubts/limitations that I have asked the authors to address (mainly concerning the architecture of the model).\n- There are some missing references and details that would help the readers to get a better sense of the subject.\n\n-----\n**After rebuttal**: The authors have addressed my comments. I have raised my score from \"6: marginally above the acceptance threshold\" to \"8: accept, good paper\".", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "XCoqle2rd_u", "reviewer": "Reviewer_AnwX", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "As the authors rightly point out, cellular automata and self-organizing systems are interesting directions of research. As I explain below, the main insight that can be derived from this paper (although not explicitly mentioned) is that a class of deep convolutional networks (networks that have repeated blocks and satisfy some other requirements) can be interpreted as neural cellular automata. With this alternative perspective, much of what the paper shows is not surprising and the questions that do arise are not addressed.\n\nHere are a number of questions that the paper does not address: \n\n- How does the decoder architecture proposed here differ from other decoder architectures proposed in the past? \nThe full architecture of the decoder network (including all the NCA steps) can be rolled into a single convolutional network with upsampling. From this perspective, the decoder architecture has repeating blocks (which allows it to be interpreted as an NCA). As a deep convolutional network, it is no surprise that the proposed decoder can function as a part of a VAE. Furthermore, I believe that many deep convolutional networks with repeating blocks (say deep resents) can also be interpreted as NCAs. Therefore, what seems to be novel in this paper (in the eyes of this reviewer) is the NCA *interpretation* and not the architecture itself.\n\n- What does one gain by the NCA interpretation of such deep conv net? \nAs mentioned above, the NCA decoder can equivalently be viewed as a deep net of a specific shape. From that perspective, the analyses of the paper are not novel. An important question for the paper is then: what does one gain by this NCA interpretation? How does this NCA type structure change the behavior of the conv net if at all?\n\n- Is there a way to train this network while remaining true to the NCA principles?\nThe way the entire system is trained is using backprop through time. This goes against the ideals of self-organizing systems to some extent. Is there any way to train this system such that the updates to the weights of each automaton are also local?", "summary_of_the_paper": "The submission proposes a VAE whose decoder is implemented via neural cellular automata (NCA). The authors show that this construction performs well when looking at the likelihood of the evidence (given samples) and show that the architecture has some robustness properties against damage during generation. ", "summary_of_the_review": "Unfortunately, almost all the technical and experimental demonstrations of the paper can be thought of as rather straight-forward consequences of the fact that the decoder is in the end a deep convolutional network. With this perspective, there is not a lot of novelty in the paper and the consequences of the fact that this deep convolutional network can in fact be implemented as cellular automata are not discussed.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "KN_tl_1RGAd", "reviewer": "Reviewer_rDYM", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n1. The paper is very well-written, I could clearly follow the ideas and the experimental setting (modulo a few minor points mentioned at the end of this review). It should be reproducible.\n2. It is interesting to see how good NCAs are in terms of likelihoods. Even the knowledge that they are not so good is valuable for the community (although I would prefer to see a few independent attempts like this to draw such a conclusion).\n3. This is a very honest paper, it reveals all the shortcomings upfront and doesn't hide anything between the lines.\n\n##########################################################################\n\nCons:\n\n1. The experimental results are underwhelming. There is little evidence that the method actually works, except on MNIST.\n2. The motivation of this work is not fully clear to me. What problem of prior work is this paper solving? I see that it introduces additional architectural limitations, and that's probably why the quality is much lower than SOTA. But what could we potentially gain?\n\n##########################################################################\n\nQuestions during rebuttal period: \n\nSignificantly improving experimental results would make me reconsider the rating - not sure if this is feasible during rebuttal though.\n\n#########################################################################\n\nMinor suggestions and typos: \n\n(1) \"and several methods have been proposed.\" - maybe \"several classes of methods\", generative modeling is quite a vast field.\n\n(2) \"The Variational Auto-Encoder (VAE) is a seminal probabilistic generative model\" - I would suggest citing both Kingma & Welling and Rezende et al., both papers appeared roughly at the same time. Up to you though.\n\n(3) Van Oord -> van den Oord\n\n(4) convolution based -> convolution-based\n\n(5) \"It achieves log p(x) >= -84.23\" - is it in bits or nats? why not report bpd?\n\n(6) \"state-of-the-art, which is around -78.6\" - assuming bits, it's not sota. See here: https://paperswithcode.com/sota/image-generation-on-mnist\n\n(7) \"Note: this shows the likelihood probabilities instead of samples from the likelihood for clarity.\" - likelihood probabilities sounds strange, why not just \"probabilities\"?\n\n(8) \"This is far from state-of-the-art which is around 2 bits per dimension\" - what are the dimensions? Looking at https://paperswithcode.com/sota/image-generation-on-celeba-64x64, I assume it's 64x64?", "summary_of_the_paper": "This paper proposes variational extension of Neural Cellular Automata for image generation. It performs experiments on MNIST, Noto Emoji and CelebA. The likelihood results are shown to be significantly behind SOTA on CelebA and also behind on other datasets. The paper provides qualitative analysis of the self-organized generation process and shows robustness to early-stage perturbations of latents.", "summary_of_the_review": "I am slightly leaning towards rejecting this paper. While the writing is high-quality and the analysis is interesting, the practical results are underwhelming, both in sample quality (see Figure 7) and in likelihoods.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "04xuNIBCoDI", "reviewer": "Reviewer_cp9d", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n- Figure 2 is extremely clear, and I understand pretty much how the model works.\n\n- I appreciate that the authors looked at a number of different datasets to compare their model against. \n\n- I think the \"damage\" experiments in Figure 10 were very interesting and well done, and could have been more of a showcase of the paper.\n\nCons:\n\n- I think this work could be improved by showing more tasks this approach would work very well for. For example, in Figure 10, what does neuron/latent variable ablation look like with a normal variational autoencoder with upsampling? Can this be quantified with log-likelihood? This would be a great metric to compare against.\n\n- For the decoding process figures (Figure 1 and 4), I don't see how this approach necessarily differs from a regular variational autoencoder with some sort of upsampling. \n\n- With respect to inpainting of images, there are many other approaches that directly use the log-likelihood that could be compared against.\n\n- I don't think the claims of \"super-resolution\" are well founded, especially if those claims are only using MNIST as the input dataset and the reconstructions have artifacts. This could be sufficiently tested by training the model either blurry or downsampled images and comparing the \"super-resolved\" images to the true high-resolution images.\n\nNeutral:\n\n- The individual samples reported in the appendix are a bit surprising (their low quality). I think this should have been reported more in the main part of the paper.\n", "summary_of_the_paper": "The authors incorporate Neural Cellular Automata (NCA) into the decoding process of a variaitional autoencoder. They then train this model on a number of toy datasets, test out beta-VAE to make better reconstructions, and show a few reconstruction/decoding tasks that highlight the contributions of the described approach.", "summary_of_the_review": "Generally, I think this model is an incremental improvement over neural cellular automata. While I do think that this approach is interesting and the results are sound, I think there could be more careful analysis of where the VNCA shines compared to existing approaches.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Meta Review for Variational Neural Cellular Automata

This paper proposes a generative model, a VAE whose decoder is implemented via neural cellular automata (NCA). The authors show that this model performs well for reconstruction, but they also show that the architecture has some robustness properties against damage during generation. 

Experiments were conducted on 3 datasets: MNIST, Noto Emoji, and CelebA, and while experimental results were great on MNIST, the method was less performant so on the other two datasets, although there is clear evidence that the model can learn to generate meaningful images. For the robustness experiments, the authors show that VNCA is robust to perturbations (occlusions) and show that the model has a reasonable degree of robustness even without ever seeing any perturbations at training time.

All authors agree that this model is an improvement over neural cellular automata, and that the approach is interesting and the results are sound (and even useful). Initially, there were concerns that NCA's were simply convolutional neural networks (the connection is already known, and not the point of the paper), and issues with comparison with baselines for damage reconstruction tasks, but these were addressed by the authors (which the reviewers have acknowledged, and have improved their scores). The authors have also responded to the concerns of reviewer cp9d, and due to the lack of response from cp9d, I assessed the authors' response myself and find that they do address the concerns (in particular, they removed claims of super-resolution, and improved the clarity of the work). With that in mind, the score of 5 is viewed as a score of 6 from my perspective (giving this work effectively an average score of 6).

After my assessment of the paper and reviews, I agree with reviewer kwgv, as they have summarized the work in their original review:
- The authors propose a variational neural cellular automaton, which learns to generate images by iterating the transition rule.
- The paper is interesting, with good results, and a good fit for ICLR.
- The paper solves an interesting problem on the topic of neural cellular automata.
- There are some doubts/limitations that I have asked the authors to address (mainly concerning the architecture of the model).
- There are some missing references and details that would help the readers to get a better sense of the subject.

Crucially, kwgv have acknowledged that the *authors have improved the paper significantly after the reviews, and they have addressed all questions and comments that [they] raised* (especially with regards to the last 2 points), and kwgv has subsequently championed the work with a score of 8. With the increased scores from kwgv and AnwX in mind, and also with what I view as an increased score of 6 from cp9d (in the lack of response from the reviewer, the authors have addressed the concerns in my judgement), my conclusion is that this is a nice work that bridges NCAs with generative models, and I think the work will be a useful addition to the growing literature in this space. I will recommend it for acceptance at ICLR 2022 as a poster.
Decision: Accept (Poster)