Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors combine the Vision Transformer architecture with Neural Cellula Automata (NCA, which combines artificial neural networks with cellula automata). They demonstrate that introducing self-attention and positional encodings to NCAs significantly improves performance. One of the main contributions of this work is the authors adaptation of to the global self-attention mechanism to respect the localisation condition for NCAs. The receptive field of individual cells is implicitly grown due to cell updates during CA iterations such that there is a global propagation of information from localised self-attention. This self-organisation is shown to give performance advantages when compared to other architectures. 



strengths_and_weaknesses: This is a welcome extension of NCAs to include self-attention and, as such, is highly original. As I understand it, one of the main challenges of achieving this has been how to combine the local computation of NCAs with global updates in ViT, without having highly inefficient learning process. The main contribution from this paper has been an attempt to do exactly this. 

The authors have provided a thorough analysis into the ViTCA, across six benchmark datasets and multiple baseline architectures including ViT, U-Nets and UNetCA. They find that ViTCA can outperform other approaches across on most benchmark datasets. Particularly interesting is the discussion of cell states, as compared to UNetCA, where the authors find that ViTCA is maintains cell stability even when damaged whereas UNetCA diverges. 

In general, I found the manuscript clear and full of information. Some of the figures had almost too much information and it was difficult to relate the caption to the figure. For example, in Figure 2 it took me some time to understand what ‘middle' and 'left,right' exactly referred to. Overall, I thought it was well structured and presented.

questions: One of the main contributions here is the propagation of local information through CA. However, with large CA systems, I can imagine that there is a separation of timescales between local information propagation and the information being implicitly available to make global changes. Have the authors considered (empirically or theoretically) the rate of information propagation through the system and whether this scales with system size?
limitations: As the authors note, there are memory constraints with using recurrent models such as CA, which they offer potential future solutions to.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a novel combination of vision transformers and Neural Cellular Automata (NCAs), and uses them to create denoising autoencoders. An NCA is essentially a cellular automata with the node updates being performed by a neural net. The ViTCA proposed here adds to this attention heads that are only focused on neighboring cells, and includes positional encodings. The paper demonstrates superior performance to a ViT on denoising autoencoder tasks, and demonstrates the robustness of the model to various perturbations.

I have read the authors' responses and am happy to update my rating to a 7. Their responses were quite thorough, although (correct me if I'm wrong) they didn't respond to the "one task" critique.



strengths_and_weaknesses: Strengths

1. This is a novel combination of transformers and neural cellular automata.

2. The results look very good compared to a vanilla ViT. E.g., Figure 1 and parts of Figure 2 make a good case for the model.

3. While the baseline ViT used for comparison has slightly fewer parameters than the baseline ViTCA, making for an unfair comparison, smaller versions of the ViTCA (e.g., ViTCA-i) also outperform it with many fewer parameters. This could be better emphasized in the paper. Although this does beg the question - how would an inverted bottleneck ViT with a similar number of parameters perform on this task?

4. The writing is fairly clear, although Figures 2 and 3 are not.

5. I believe, given the results, that this work is significant, but they only demonstrated efficacy on one task, which reduces enthusiasm somewhat.


Weaknesses:

1. It is very hard to understand what is going on in Figures 2 and 3. I can make some sense of Figure 3, but Figure 2 is a mystery. I don’t know what a “splat map” is, and I can’t tell that the cells are focusing on foreground, noise and background, as mentioned in the caption. The authors should try showing these figures to a colleague and finding out what they have to explain to get the idea across, and then simplify the figure to make this clear.

2. This reviewer, at least, would have appreciated a demonstration of this architecture on another problem (e.g., semantic segmentation), rather than a deep dive into the model robustness to ablations and various perturbations (section 4.1.1-4.1.3), which could have been relegated to the supplementary material. 

3. This model is similar to a universal transformer, which also uses essentially depth 1 (since the parameters are used over and over), with iterative processing.


questions: 1. What are the essential differences between your model and a universal transformer with local attention? Universal transformers simply use the same weights over and over, so they are recurrent (Dehghani, et al. 2018). One difference is the asynchronous update rule. Others?

2. On lines 209-210, you mention that ViTCA outperforms the other baselines on 10 out of the 18 datasets, although you are reporting only 6. Are there 12 more datasets you aren’t telling us about??

3. Instead of overflow loss, have you tried just using nonlinear activation functions that stay within the desired range?

Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, Łukasz Kaiser (2018) Universal Transformers. arXiv:1807.03819.
limitations: Yes.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a new family of Neural Cellular Automata (NCA) based on visual Transformer for the first time. The main difference from the plain ViT is the use of a specially localized self-attention mechanism that is still globally organized, thus yielding a parameter-efficient model.  Quantitative and qualitative experiments show that the proposed ViTCA outperforms U-Net, UNetCA, and ViT on various benchmarks. The learned representations are also verified via linear probes evaluation.
strengths_and_weaknesses: Strengths:
+ This paper provides a new perspective on neural cellular automata with a visual Transformer architecture. Although it is a combination of problem and architecture, the authors propose an effective self-attention scheme to improve the model.
+ The experiments are relatively complete and can verify the proposed contributions.
+ The paper is well organized and provides informative figures.
Weaknesses:
- The biggest problem is that the authors do not explain why U-Net is adopted as a baseline model for comparison. This makes experimental comparisons confusing. Did you choose U-net as a CNN structure for comparison?
- It is better to validate the performance of linear probes on larger datasets since experiments are only evaluated on small MNIST and CIFAR 10 datasets. However, I do not mean that this will be the reason for the rejection of the paper.
questions: - Choosing ViT as a baseline to show the superiority of the proposed ViTCA makes sense. But why do you adopt U-Net, the CNN architecture, as the baseline? And why the proposed ViTCA succeeds while U-net fails? The authors do not describe the motivation and the reasons for such an experimental setup. 
- Linear probes have been popular evaluation protocols to show the learning representation via self-supervised learning. I am wondering how does the proposed model perform using fine-tuning? Can the authors provide fine-tuning results or explain the reason why it is not good if fine-tuning performs poorly?
- The proposed model requires iterative training to converge. How can you ensure the stability of the proposed model?
- Performing experiments on MNIST and CIFAR 10 datasets is OK. It is better to validate the performance on larger datasets to show the practicality of the proposed model. The authors can either provide some analysis if the performance is not good.
limitations: The authors have discussed the limitations in Sec 5.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper presents a neural cellular automata (NCA) architecture for image denoising that replaces the typical convolutional layers with a vision transformer.

The experiments show that the proposed architecture achieves marginally better results compared to two baselines on three simple datasets, and the authors show that their model appears to be stable after converging to a target image (NCAs are known to be unstable when evaluated on longer horizons than the one seen at training time).
strengths_and_weaknesses: The idea of the paper is novel, although it doesn't provide substantial contributions besides replacing convolutions with transformers.
The quality of the paper is good, the experiments go fairly in-depth although on just one task of image denoising (which is less interesting than texture generation or other forms of morphogenesis, in the reviewer's personal opinion).

The clarity of the paper is good, almost all details are reported clearly and there are only a few details that require clarification (see below).
The significance of the paper is modest, since it is not even clear that the proposed method provides real advantages compared to the two simple baselines tested by the authors. The paper is unlikely to have an impact on the NCA community, and surely not on the much larger computer vision community. 
questions: Overall, I lean towards a borderline reject which can turn into a borderline accept if the following questions are addressed.

1. What is the meaning of "it is a fairly contractive model capable of fixed-point convergence guarantees"?

2. The attention operation in Equations (1) and (3) is at the patch level, not at the cell level. This means that ViTCA is not comparable 1-to-1 with pixel-level NCAs unless the patch size is set to 1x1. 
Also, it might confuse the reader to refer to the "Moore neighborhood" in line 177 when talking about the patch size, because this term is typically used in CA literature to indicate the neighborhood of a cell (as it is used to compute the update). Here, the effective neighborhood of the cell is 9x9.
Did I misunderstand something?

3. I couldn't find how the positional encodings are computed. IF PEs provide unique coordinates for each pixel, doesn't that defeat the principle of locality of CAs?

4. Why is the ViTCA baseline configured to have a patch size of 1x1, while all the other ViTCA models have 3x3?

5. The number of parameters in Tables 1, 2, and 3 appears to be misreported. 
A few examples: 
    - Table 1, no difference between ViTCA (4 heads) and ViTCA-32 (32 heads)
    - Table 2, no difference between ViTCA with 4-64 heads
    - Table 3, no difference between the variants
    - Table 3, no difference between MLPs with 100 or 1000 hidden units on CIFAR10

    The absence of differences in Tables 1 and 2 is especially puzzling and important to clarify, because ViTCA-32 is the only model that can consistently surpass the baselines (while, e.g., ViTCA-4 cannot).

6. Why use an overflow loss instead of an activation function (sigmoid, tanh) that forces the updates to be in the required ranges?

7. Figure 3 shows some noise sampled from a uniform distribution being injected in the cell update. Where is this described in the text?

8. Since all NCAs are image-to-image maps, why didn't the authors consider other NCA baselines like the original NCA [30] or the Variational NCA [7]? Why not consider other recurrent baselines too? As it is (and considering point 5 above), the comparisons are not very informative.

I'm looking forward to the discussion with the authors. 
limitations: The limitations are not discussed extensively besides what concerns memory costs. 
I have pointed out a few issues in my questions that could be discussed better (e.g., point 3). 

There are no societal concerns with the work.
review_title: null

Meta-review: metareview: As summarized by reviewer GAh5, this paper proposes a novel combination of vision transformers and Neural Cellular Automata (NCAs), and uses them to create denoising autoencoders. An NCA is essentially a cellular automaton with the node updates being performed by a neural net. The ViTCA proposed here adds to this attention heads that are only focused on neighboring cells, and includes positional encodings. The paper demonstrates superior performance to a ViT on denoising autoencoder tasks, and demonstrates the robustness of the model to various perturbations.

All reviewers agree that this is a novel contribution to NCAs. By replacing convolutions with vision transformers, it opens up many possibilities (such as scale). The experiments were conducted with detail and clarity. I do agree with reviewer mu6v, that while the quality of the paper is good, the experiments go fairly in-depth although on just one task of image denoising (which is less interesting than texture generation or other forms of morphogenesis), but even in its current form, it definitely meets the bar for a solid acceptance recommendation from me.

confidence: Certain
Decision: Accept