Introduction
Recent developments at the intersection of two foundational ideas—Artificial Neural Networks (ANNs) and Cellular Automata (CA)—have led to new approaches for constructing Neural Cellular Automata (NCA). These advances have integrated ideas such as variational inference [ 7 ] , U-Nets [ 26 ] , and Graph Neural Networks (GNNs) [ 15 ] with promising results on problems ranging from image synthesis [ 7 , 21 , 20 ] to Reinforcement Learning (RL) [ 6 , 22 ] . Transformers are another significant development in deep learning [ 41 ] , but, until now, have not been examined under an NCA setting.

Vision Transformers (ViTs) [ 13 ] have emerged as a competitive alternative to Convolutional Neural Network (CNN) [ 56 ] architectures for computer vision, such as Residual Networks (ResNets) [ 45 ] . ViTs leverage the self-attention mechanisms of original Transformers [ 41 ] , which have emerged as the dominant approach for sequence modelling in recent years. Our work combines foundational ideas from Transformers and ViTs, leading to a new class of NCAs: Vision Transformer Cellular Automata (ViTCA) .

An effective and ubiquitous Transformer-based learning technique for Natural Language Processing (NLP) pre-training is the unsupervised task of Masked Language Modelling (MLM), popularized by the BERT language model [ 34 ] . The success of MLM-based techniques has similarly inspired recent work re-examining the classical formulation of Denoising Autoencoders (DAEs) [ 51 ] , but for ViTs [ 3 , 13 , 28 ] , introducing tasks such as Masked Image Encoding [ 16 ] and Masked Feature Prediction [ 24 ] for image and video modelling, respectively. This simple yet highly-scalable strategy of masked-based unsupervised pre-training has yielded promising transfer learning results on vision-based downstream tasks such as object detection and segmentation, image classification, and action detection, even outperforming supervised pre-training [ 16 , 24 ] . We examine training methodologies for ViTCA within a DAE setting and perform extensive controlled experiments benchmarking these formulations against modern state of the art architectures, with favourable outcomes, e.g . , Fig. 1 .

Our contributions are as follows: first —to the best of our knowledge—our work is the first to extend NCA methodologies with key Transformer mechanisms, i.e . , self-attention and positional encoding (and embedding), with the beneficial side-effect of circumventing the quadratic complexity of self-attention; second , our ViTCA formulation allows for lower model complexity (by limiting ViT depth) while retaining expressivity through CA iterations on a controlled state—all with the same encoder weights. This yields a demonstrably more parameter-efficient [ 20 ] ViT-based model. Importantly, ViTCA mitigates the problems associated with the explicit tuning of ViT depth originally needed to improve performance ( i.e . , we use a depth of 1). With ViTCA, we simply iterate until cell state convergence. Since ViT (and by extension, ViTCA) employs Layer Normalization (LN) [ 43 ] at each stage of its processing, it is a fairly contractive model capable of fixed-point convergence guarantees [ 32 ] .

In relation to our first contribution, ViTCA respects CA requirements, most importantly that computations remain localized about a cell and its neighbourhood. As such, we modify the global self-attention mechanisms of a ViT to respect this locality requirement (Fig. 2 ). Localized self-attention is not a new idea [ 4 , 19 , 12 , 27 ] ; however, because cells contain state information that depends on its previous state, over CA iterations the effective receptive field of ViTCA’s localized self-attention grows increasingly larger until eventually incorporating information implicitly across all cells. Thus, admitting global propagation of information from spatially localized self-attention. Moreover, due to the self-organizing nature of NCAs, self-organization also manifests itself within the localized self-attention, resulting in a globally agreed-upon arrangement of local self-attention. Thus, circumventing the quadratic complexity of explicit global self-attention (w.r.t. the input size) through a linear amortization over time, and increasing the feasibility of per-pixel dense processing (as we demonstrate). This globally consistent and complex behaviour, which arises from strictly local interactions, is a unique feature of NCAs and confers performance benefits which we observe both qualitatively and quantitatively when comparing ViT and ViTCA for denoising autoencoding.