Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes variational extension of Neural Cellular Automata for image generation. It performs experiments on MNIST, Noto Emoji and CelebA. The likelihood results are shown to be significantly behind SOTA on CelebA and also behind on other datasets. The paper provides qualitative analysis of the self-organized generation process and shows robustness to early-stage perturbations of latents.
main_review: Pros:

1. The paper is very well-written, I could clearly follow the ideas and the experimental setting (modulo a few minor points mentioned at the end of this review). It should be reproducible.
2. It is interesting to see how good NCAs are in terms of likelihoods. Even the knowledge that they are not so good is valuable for the community (although I would prefer to see a few independent attempts like this to draw such a conclusion).
3. This is a very honest paper, it reveals all the shortcomings upfront and doesn't hide anything between the lines.

##########################################################################

Cons:

1. The experimental results are underwhelming. There is little evidence that the method actually works, except on MNIST.
2. The motivation of this work is not fully clear to me. What problem of prior work is this paper solving? I see that it introduces additional architectural limitations, and that's probably why the quality is much lower than SOTA. But what could we potentially gain?

##########################################################################

Questions during rebuttal period: 

Significantly improving experimental results would make me reconsider the rating - not sure if this is feasible during rebuttal though.

#########################################################################

Minor suggestions and typos: 

(1) "and several methods have been proposed." - maybe "several classes of methods", generative modeling is quite a vast field.

(2) "The Variational Auto-Encoder (VAE) is a seminal probabilistic generative model" - I would suggest citing both Kingma & Welling and Rezende et al., both papers appeared roughly at the same time. Up to you though.

(3) Van Oord -> van den Oord

(4) convolution based -> convolution-based

(5) "It achieves log p(x) >= -84.23" - is it in bits or nats? why not report bpd?

(6) "state-of-the-art, which is around -78.6" - assuming bits, it's not sota. See here: https://paperswithcode.com/sota/image-generation-on-mnist

(7) "Note: this shows the likelihood probabilities instead of samples from the likelihood for clarity." - likelihood probabilities sounds strange, why not just "probabilities"?

(8) "This is far from state-of-the-art which is around 2 bits per dimension" - what are the dimensions? Looking at https://paperswithcode.com/sota/image-generation-on-celeba-64x64, I assume it's 64x64?
summary_of_the_review: I am slightly leaning towards rejecting this paper. While the writing is high-quality and the analysis is interesting, the practical results are underwhelming, both in sample quality (see Figure 7) and in likelihoods.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper introduces a variational generative model based on Neural Cellular Automata (NCA). The model is called the Variational NCA (VNCA).

The VNCA is designed for images: 

- The encoder is a typical convolutional neural network. 
- The decoder is an NCA that iteratively refines the image, alternating convolutions and upsampling up to the desired image size. The upsampling process is loosely inspired by mitosis in living cells. 

The authors perform morphogenesis experiments on three datasets: MNIST, Noto Emoji, and CelebA.
The results are good on MNIST, less so on the other two datasets, although there is clear evidence that the model can learn to generate meaningful images.

The authors also perform an experiment to see if the VNCA is robust to perturbations (occlusions) and show that the model has a reasonable degree of robustness even without ever seeing any perturbations at training time.
main_review: **Strengths**

- I enjoyed the paper and, although I have some comments, I think the work can be interesting for the ICLR community. There is growing interest in (neural) cellular automata, and this work is a valuable addition to the topic. 

- The paper addresses a limitation of typical morphogenesis approaches with NCA, which is that they are usually trained to generate a single image/state. This work is a first step towards learning NCA that generalize to different target states. 

- The experimental results are interesting despite having some limitations. The VNCA can reconstruct the images well when starting from a latent sample produced by the encoder. The limitations only arise when using the model for generation starting from a random sample from the prior, but the authors also show how to partially address these issues (using the $\beta$-VAE approach). 

- The paper is well written and all concepts are explained adequately.

**Weaknesses**

I have a few suggestions and questions that I hope can help the authors improve the paper. 

1. Have the authors tested their model without the upsampling part? 
    
    The base NCA of Mordvintsev et al. is a sufficiently powerful architecture for morphogenesis (starting from a single nonzero pixel) and it is not immediately clear why the upsampling step is necessary for the VNCA. An explanation of what led the authors to this choice (besides the link with mitosis) would be interesting.

2. As noted by the authors, the upsampling step forces the output to have a given dimension, multiple of $2^M$. 

    Related to this, have the authors investigated some settings in which the images are exactly as big as the output of the VNCA, without zero padding? This would give us interesting insights into the behaviour of the VNCA at the boundaries. 

3. I have a few questions regarding the architecture:

    - What is the activation function of the last layer of the NCA? 
    The authors only mention ELU but the last activation should ensure that the output of the model is a meaningful image (so a sigmoid or rescaled tanh).
    - How was the NCA architecture chosen? Why four residual blocks? Why two 1x1 convolutions in each block? 
    - Why is there no activation after the last convolution of each residual block? 
    - How was the architecture of the encoder chosen?
    - How were the number of channels, learning rate, batch size, gradient clipping, $K$, $L$, and $\beta$ determined? 
      I'm not asking the authors to perform an in-depth tuning, but if the hyperparameters were chosen arbitrarily it is important that the readers know it. 
    - How do 1.2M parameters compare to Mordvintsev at al.'s NCA? 

5. In Figure 4, the final generated digits are significantly smoother than those in Figure 3, right panel. Were the examples of Figure 4 cherry-picked somehow? 

6. There are no details about the training procedure. For example, Mordvintsev et al. use a replay memory to ensure the stability of the generated pattern. Do the authors have a comparable training procedure in place? 

    NCA are not well known in the community (yet), so papers on the subject must a good job of explaining the base concepts until they are more widely adopted. 

7. Two references could/should be added in Section 1: 

    - Wulff and Hertz, "Learning cellular automaton dynamics with neural networks", Neural Information Processing Systems (1992). 
       This is one of the first papers to model CA transition rules with neural networks (they even use CNNs!) and a good reference to cite for completeness. 

    - Grattarola et al., "Learning Graph Cellular Automata", Neural Information Processing Systems (2021). 
       This is a very recent paper (published after the ICLR submission date) that explores the use of graph neural networks to learn cellular automata on graphs (they also do morphogenesis). It seems like a good match for the last paragraph of Page 2, which has no references. 

**Other comments**

- Some readers may not be familiar with the "bits per pixel" metric. An explanation of what the metric represents could be useful. 

- Regarding this claim: "It is not immediately clear what a generative model that takes damage and super-resolution into account would look like". 
I don't think that NCA are that different from any typical neural network. Couldn't we train the VNCA as any other super-resolution or de-noising autoencoder?

    For example, the de-noising could be done as they do in the NCA paper, and the super-resolution by fixing two different generations at which the loss is computed, using the target image and a downscaled version of it. 

- Have the authors performed a latent space analysis to see if different classes of images get clustered in some particular way?

- There is no mention of code anywhere in the submission. I encourage the authors to make the code available if the paper gets accepted (ideally it should have been included with the submission, but I understand that it is not mandatory and I have not taken this into consideration when evaluating the paper). 

----------

**After rebuttal**: The authors have addressed my comments. I have raised my score from "6: marginally above the acceptance threshold" to "8: accept, good paper".
summary_of_the_review: - The authors propose a variational neural cellular automaton, which learns to generate images by iterating the transition rule.
- The paper is interesting, with good results, and a good fit for ICLR.
- The paper solves an interesting problem on the topic of neural cellular automata. 
- There are some doubts/limitations that I have asked the authors to address (mainly concerning the architecture of the model).
- There are some missing references and details that would help the readers to get a better sense of the subject.

-----
**After rebuttal**: The authors have addressed my comments. I have raised my score from "6: marginally above the acceptance threshold" to "8: accept, good paper".
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The submission proposes a VAE whose decoder is implemented via neural cellular automata (NCA). The authors show that this construction performs well when looking at the likelihood of the evidence (given samples) and show that the architecture has some robustness properties against damage during generation. 
main_review: As the authors rightly point out, cellular automata and self-organizing systems are interesting directions of research. As I explain below, the main insight that can be derived from this paper (although not explicitly mentioned) is that a class of deep convolutional networks (networks that have repeated blocks and satisfy some other requirements) can be interpreted as neural cellular automata. With this alternative perspective, much of what the paper shows is not surprising and the questions that do arise are not addressed.

Here are a number of questions that the paper does not address: 

- How does the decoder architecture proposed here differ from other decoder architectures proposed in the past? 
The full architecture of the decoder network (including all the NCA steps) can be rolled into a single convolutional network with upsampling. From this perspective, the decoder architecture has repeating blocks (which allows it to be interpreted as an NCA). As a deep convolutional network, it is no surprise that the proposed decoder can function as a part of a VAE. Furthermore, I believe that many deep convolutional networks with repeating blocks (say deep resents) can also be interpreted as NCAs. Therefore, what seems to be novel in this paper (in the eyes of this reviewer) is the NCA *interpretation* and not the architecture itself.

- What does one gain by the NCA interpretation of such deep conv net? 
As mentioned above, the NCA decoder can equivalently be viewed as a deep net of a specific shape. From that perspective, the analyses of the paper are not novel. An important question for the paper is then: what does one gain by this NCA interpretation? How does this NCA type structure change the behavior of the conv net if at all?

- Is there a way to train this network while remaining true to the NCA principles?
The way the entire system is trained is using backprop through time. This goes against the ideals of self-organizing systems to some extent. Is there any way to train this system such that the updates to the weights of each automaton are also local?
summary_of_the_review: Unfortunately, almost all the technical and experimental demonstrations of the paper can be thought of as rather straight-forward consequences of the fact that the decoder is in the end a deep convolutional network. With this perspective, there is not a lot of novelty in the paper and the consequences of the fact that this deep convolutional network can in fact be implemented as cellular automata are not discussed.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors incorporate Neural Cellular Automata (NCA) into the decoding process of a variaitional autoencoder. They then train this model on a number of toy datasets, test out beta-VAE to make better reconstructions, and show a few reconstruction/decoding tasks that highlight the contributions of the described approach.
main_review: Pros:

- Figure 2 is extremely clear, and I understand pretty much how the model works.

- I appreciate that the authors looked at a number of different datasets to compare their model against. 

- I think the "damage" experiments in Figure 10 were very interesting and well done, and could have been more of a showcase of the paper.

Cons:

- I think this work could be improved by showing more tasks this approach would work very well for. For example, in Figure 10, what does neuron/latent variable ablation look like with a normal variational autoencoder with upsampling? Can this be quantified with log-likelihood? This would be a great metric to compare against.

- For the decoding process figures (Figure 1 and 4), I don't see how this approach necessarily differs from a regular variational autoencoder with some sort of upsampling. 

- With respect to inpainting of images, there are many other approaches that directly use the log-likelihood that could be compared against.

- I don't think the claims of "super-resolution" are well founded, especially if those claims are only using MNIST as the input dataset and the reconstructions have artifacts. This could be sufficiently tested by training the model either blurry or downsampled images and comparing the "super-resolved" images to the true high-resolution images.

Neutral:

- The individual samples reported in the appendix are a bit surprising (their low quality). I think this should have been reported more in the main part of the paper.

summary_of_the_review: Generally, I think this model is an incremental improvement over neural cellular automata. While I do think that this approach is interesting and the results are sound, I think there could be more careful analysis of where the VNCA shines compared to existing approaches.
review_title: null

Meta-review: Meta Review for Variational Neural Cellular Automata

This paper proposes a generative model, a VAE whose decoder is implemented via neural cellular automata (NCA). The authors show that this model performs well for reconstruction, but they also show that the architecture has some robustness properties against damage during generation. 

Experiments were conducted on 3 datasets: MNIST, Noto Emoji, and CelebA, and while experimental results were great on MNIST, the method was less performant so on the other two datasets, although there is clear evidence that the model can learn to generate meaningful images. For the robustness experiments, the authors show that VNCA is robust to perturbations (occlusions) and show that the model has a reasonable degree of robustness even without ever seeing any perturbations at training time.

All authors agree that this model is an improvement over neural cellular automata, and that the approach is interesting and the results are sound (and even useful). Initially, there were concerns that NCA's were simply convolutional neural networks (the connection is already known, and not the point of the paper), and issues with comparison with baselines for damage reconstruction tasks, but these were addressed by the authors (which the reviewers have acknowledged, and have improved their scores). The authors have also responded to the concerns of reviewer cp9d, and due to the lack of response from cp9d, I assessed the authors' response myself and find that they do address the concerns (in particular, they removed claims of super-resolution, and improved the clarity of the work). With that in mind, the score of 5 is viewed as a score of 6 from my perspective (giving this work effectively an average score of 6).

After my assessment of the paper and reviews, I agree with reviewer kwgv, as they have summarized the work in their original review:
- The authors propose a variational neural cellular automaton, which learns to generate images by iterating the transition rule.
- The paper is interesting, with good results, and a good fit for ICLR.
- The paper solves an interesting problem on the topic of neural cellular automata.
- There are some doubts/limitations that I have asked the authors to address (mainly concerning the architecture of the model).
- There are some missing references and details that would help the readers to get a better sense of the subject.

Crucially, kwgv have acknowledged that the *authors have improved the paper significantly after the reviews, and they have addressed all questions and comments that [they] raised* (especially with regards to the last 2 points), and kwgv has subsequently championed the work with a score of 8. With the increased scores from kwgv and AnwX in mind, and also with what I view as an increased score of 6 from cp9d (in the lack of response from the reviewer, the authors have addressed the concerns in my judgement), my conclusion is that this is a nice work that bridges NCAs with generative models, and I think the work will be a useful addition to the growing literature in this space. I will recommend it for acceptance at ICLR 2022 as a poster.
Decision: Accept (Poster)