Denoising autoencoding
We compare between our baseline models and a number of ViTCA variants in the context of denoising autoencoding. We present test set results across six benchmark datasets: a land cover classification dataset intended for representation learning (LandCoverRep) [ 25 ] , MNIST, CelebA [ 47 ] , FashionMNIST, CIFAR10, and Tiny ImageNet (a subset of ImageNet [ 49 ] ). All datasets consist of 32\!\times\!32 resampled images except Tiny ImageNet, which is at 64\!\times\!64 resolution. During testing, we use all masking combinations, chosen in a fixed order, and we update cells using a fixed number of iterations ( T\!=\!64 ). See Tab. 4.1 for quantitative results.

Briefly mentioned in Sec. 3.1 , we employ a masking strategy inspired by Curriculum Learning (CL) [ 23 , 52 ] to ease training. This schedule follows a geometric progression of difficulty—tied to training iterations—maxing out at 10K training iterations. Specifically, masking starts at covering 25% of the input with 1\!\times\!1 patches of noise (dropout for RGB inputs, Gaussian for grayscale), then at each shift in difficulty, new masking configurations are added to the list of available masking configurations in the following order: (2^{0}\!\times\!2^{0},50\%),(2^{0}\!\times\!2^{0},75\%),(2^{1}\!\times\!2^{1},25\%),(2^{1}\!\times\!2^{1},50\%),(2^{1}\!\times\!2^{1},75\%),...,(2^{2}\!\times\!2^{2},75\%) . Masking configurations are randomly chosen from this list.

We initialize weights/parameters using He initialization [ 46 ] , except for the final layer of CA-based models, which are initialized to zero [ 30 ] . Unless otherwise stated, we train for I\!=\!100 K iterations, use a minibatch size b\!=\!32 , AdamW optimizer [ 36 ] , learning rate \eta\!=\!10^{-3} with a cosine annealing schedule [ 40 ] , pool size N_{\mathcal{P}} \!=\!1024 , and cell hidden channel size C_{h}\!=\!32 . In the case of Tiny ImageNet, b\!=\!8 to accommodate training on a single GPU (48GB Quadro RTX 8000). Training typically lasts a day at most, depending on the model. Due to the recurrent iterations required per training step, CA-based models take the longest to train. To alleviate memory limitations for some of our experiments, we use gradient checkpointing [ 44 ] during CA iterations at the cost of backpropagation duration and slight variations in gradients due to its effect on round-off propagation. We also experiment with a cell fusion and mitosis scheme as an alternative. See Appendix A for details on runtime performance, gradient checkpointing, and fusion and mitosis.

Amongst baselines, ViTCA outperforms on most metrics across the majority of datasets used (10 out of 18). Exceptions include LandCoverRep, where UNetCA universally outperforms by a small margin, likely due to the texture-dominant imagery being amenable to convolutions. Notably, ViTCA strongly outperforms on MNIST. Although MNIST is a trivial dataset for common tasks such as classification, our masking/noise strategy turns it into a challenging dataset for denoising autoencoding, e.g . , it is difficult for even a human to classify a 32\!\times\!32 MNIST digit 75% corrupted by 4\!\times\!4 patches of Gaussian noise. We hypothesize that when compared to convolutional models, ViTCA’s weaker inductive biases (owed to attention [ 8 , 5 ] ) immediately outperform these models when there are large regions lacking useful features, e.g . , MNIST digits cover a small space in the canvas. This is not the case with FashionMNIST, where the content is more filled out. Between baselines and ViTCA variants, ViTCA-32 (32 heads) and 32xy (xy-coordinate positional encoding) outperform all models by large margins, demonstrating the benefits of multi-head self-attention. We also experiment with a parameter-reduced (by \sim\!60\% ), inverted bottleneck variant where d\!=\!64 and MLP size is 256, often with a minimal reduction in performance.