AdanCA: Neural Cellular Automata As Adaptors For More Robust Vision Transformer
Authors: Yitao Xu
Tong Zhang
Sabine Susstrunk
Venue: NeurIPS 2024 poster (2024)
Keywords: Neural Cellular Automata
Vision Transformer
Adversarial Robustness
Out-of-distribution generalization
Primary area: deep_learning_architectures
TL;DR: Neural Cellular Automata can be inserted between the middle layers of Vision Transformers (ViTs) to improve ViTs' robustness on image classification.

Abstract: Vision Transformers (ViTs) demonstrate remarkable performance in image classification through visual-token interaction learning, particularly when equipped with local information via region attention or convolutions. Although such architectures improve the feature aggregation from different granularities, they often fail to contribute to the robustness of the networks. Neural Cellular Automata (NCA) enables the modeling of global visual-token representations through local interactions, with its training strategies and architecture design conferring strong generalization ability and robustness against noisy input. In this paper, we propose Adaptor Neural Cellular Automata (AdaNCA) for Vision Transformers that uses NCA as plug-and-play adaptors between ViT layers, thus enhancing ViT's performance and robustness against adversarial samples as well as out-of-distribution inputs. To overcome the large computational overhead of standard NCAs, we propose Dynamic Interaction for more efficient interaction learning. Using our analysis of AdaNCA placement and robustness improvement, we also develop an algorithm for identifying the most effective insertion points for AdaNCA. With less than a 3% increase in parameters, AdaNCA contributes to more than 10% absolute improvement in accuracy under adversarial attacks on the ImageNet1K benchmark. Moreover, we demonstrate with extensive evaluations across eight robustness benchmarks and four ViT architectures that AdaNCA, as a plug-and-play module, consistently improves the robustness of ViTs.

AdaNCA: Neural Cellular Automata as Adaptors for More Robust Vision Transformer
Yitao Xu Tong Zhang Sabine Süsstrunkk Image and Visual Representation Lab École polytechnique fédérale de Lausanne, Lausanne, Switzerland {yitao. xu, tong. zhang, sabine. susstrunk/@epff. ch
Abstract
Vision Transformers (ViTs) demonstrate remarkable performance in image classifi- cation through visual-token interaction learning, particularly when equipped with local information via region attention or convolutions. Although such architectures improve the feature aggregation from different granularities, they often fail to con- tribute to the robustness of the networks. Neural Cellular Automata (NCA) enables the modeling global visual-token representations through local interactions, as its training strategies and architecture design confer strong generalization ability and robustness against noisy input. this paper, we propose Adaptor Neural Cellular Automata (AdaNCA) for Vision Transformers that uses NCA as plug-and-plar adaptors between ViT layers, thus enhancing ViT's performance and robustness against adversarial samples well as out-of-distrbbttion inputs. To overcome the large computational overhead of standard NCAs, we propose Dynamic Interaction for more efficient interaction learning. Using our analysis of AdaNCA placement and robustness improvement, we also develop an algorithm for identifying the most effective insertion points for AdaNCA. With less than 3% increase in parameters, AdaNCA contributes to more than 10% of absolute improvement in accuracy under adversarial attacks on the ImageNetI benchmark. Moreover, we demonstrate with extensive evaluations across eight robustness benchmarks and four ViT ar- chitectures that AdaNCA, as a plug-and-play module, consistently improves the robustness of ViTs.
Introduction
Vision Transformers (ViTs) exhibit impressive performance in image classification, through globally modeling token interactions via self-attenin
mechanisms [1]] 14. 77). Recent works show that
73 or 71 75). further enhances the ViT's capabilities in image classification. Although advanced local structures contribute better captures of local information, the robustness of ViT's has not increased. They remain vulnerable to noisy input such as adversarial
integrating local information into ViTs, e.g., using region attention 26
convolution [8] [O 16, 22 23
37 42, 52
68
samples
13
and out-of-distributioo (OOD) inputs 17
18
67
Recently, Neural Cellular Automata (NCA) was proposed as a lightweight architecture for modeling local cell interactions [44], where cells are represented by ID vectors. To perform downstream tasks, similarly to the idea of token interactions in ViTs, cells in NCA interact with each other by alternating between convolution-
Interaction stage and an MLP-based Update stage 46 The critical difference, however, is that cell interactions in NCA evolve over time by recurrent application of the two stages, whereas VIT computes the token interaction cells dynamically modulate their representations, based on the interactions with their neighbors, and they gradually enlarge their receptive fields. Unlike commonly used convolutional neural networks, NCA maintains resolution during neighborhood expansion. The recurrent update scheme enables the
single step per layer. During this process,
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

cells to explore various states, thus preventing NCA from overfitting and enhancing its generalization ability [4446] NCA training involves various kinds of stochasticity 445, which enables the models to
generalize to input variability and adapt to unpredictable perturbations. It is the modulation of local in-
formation and stochasticity during training that make NCA robust against noisy input
49
48 55
However, the original NCA has substantial com- putational overhead when operating in high- dimensional space, which is common scenario in ViTs. This poses non-trivial challenge when integrating NCA into ViTs. To reduce the dimen- sionality of interaction results, hence to lower the computational cost, we propose Dynamic Interaction to replace the standard Interaction stage. In this stage, tokens dynamically modify the interaction strategy, based on the observa- tion of their environment. This adaptation to the variability of the environment contributes to the model robustness. Our modified Adaptor plug-and-pin module, improves ViTs performances, as illustrated in Figure I Adding AdaNCA to different VIT architectures consistently improves their robust- ness to both adversarial attacks and OOD input. AdaNCA also improves clean accuracy.
Model Group
RVT
RVT-AdaNCA (Ours)
Swin
Adversatal AtLack Acc (%
ConVIT-AdaNGA (Ours)
Swin-AdaNCA (Ours)
RVT-Sese-Plis
ConViT-Bsee"
Swin-Base'
TAPADL-RVT
ConviT-Base
54
50
46
Corruption Eroor (1) on O.O.D input (%)
NCA (AdaNCA),
Figure 1: The accuracy under adversarial attacks (APGD-DLR (9) versus corruption error on out- of-distribution input (ImageNet-C (28)) of various ViT models [16 23 38 42). AdaNCA improves the robustness of different ViTs against both adver- sarial attacks and OOD input. *: the same model architecture but with more layers.
Motivated by empirical observations of the pos- itive relationship between network redundancy and model robustness [28), we develop dy- namic programming algorithm for computing the most effective insert position for AdaNCA within a ViT, based our proposed quantification of network redundancy. Our method results in consistent improvements across cight robustness benchmarks and four different baseline ViT architectures. Critically, we demonstrate that the improvements do not originate from the increase in parameters and FLOPS but are attributed to AdaNCA. Our contributions are as follows:
We propose AdaNCA: It integrates Neural Cellular Automata into ViTs' middle layers as lightweight Adaptors for the robustness enhancement of ViTs against adversarial attacks and OOD inputs in image classification. With less than 3% more parameters, AdaNCA-extended ViTs can, under certain adversarial attacks, achieve 10% higher accuracy.
We introduce Dynamic Interaction to replace the Interaction stage in standard NCA, thus enhancing model robustness and efficiency in terms of parameters and computation.
We propose method for determining the most effective insert positions of AdaNCA, for maximum robustness improvement.
2
Related Works
2.1 Local structure in Vision Transformers
Since the proposal of Vision Transformer (ViT) 14 a series of works have introduced local structures
into ViTs to enhance their performance 8
10 16 22
23
37
42
52
62
68
70
71
73
75| 76).
Here, we mention one of the earliest local structure modifications and those relevant to our work. Stand-Alone Aoff-Attention (SASA), introduced by Ramachandran et al. (54). utilizes sliding window self-attention in ViTs. Following this, Liu et al. (38 develop non-sliding window attention mechanism that partitions feature maps and computes self-attention, both within and between these partitions; it is termed Shifted Window (Swin) attention. Another method for modeling local information convolution. D' Ascoli al. [16] introduce a soft local-inductive bias by using gated positional self-attention, thus fusing self-attention and convolution. Despite these advancements in better modeling of local information, few methods lead to a more robust ViT architecture (2). This leaves the models to behave subpar when encountering slightly noisy inputs distribution shifts.

---

(
Adopter Neuual Celuler Automatt, AdAMCC
Oynami:
Vydoote
Loyar JoyararA
Robusness
Cornelation: 077
interaction
Layerget
Corretation
2 Sareo
Akdundawc
Pai-unse Layer Smilorty
Antiondnncy
Loyer
AdUNCA
Lover indies
Figure 2: Method overview. (a) To improve model performance and robustness, Neural Cellulan Automata (NCA) can be inserted into Vision Transformers (ViTs) as Adaptors, hence termed AdaNCA. The details of AdaNCA are presented in Section [.2] The improvement is maximized when AdaNCA inserted between two layer sets that each consists of similar layers. (b) The robustness improvement brought by AdaNCA is highly correlated with the corresponding network redundancy
quantification the insert position introduced Section 3.3 This supports idea that AdaNCA
should be placed between two sets of redundant layers.
2.2 Robust architecture in Vision Transformers
Researchers have developed various architectural changes for building robust ViT' against adversarial attacks, such as FGSM [6]] or PGD [41] as well as out-of-distribution (OOD) inputs, such image corruption [28 Zhou et al. 78 propose Full Attention Networks to boost the robustness of ViTs against OOD images. Mao et al. 42 first systematically analyze the relationship between different components in VIT, drawing positive relationship between convolutional components and the robustness of ViTs against adversarial samples and OOD data. By extending (42) and
78 Guo et al. (33 propose input-dependent average pooling in order adaptively select different aggregation neighborhoods for different tokens, thus achieving the state-of-the-ee robust ViTs in OOD generalization. Different interpretations of the self-attention operation can also lead more robust architectures 24 158. However, the methods that introduce additional architectures 23
24 58 78
are either implemented on ViTs with limited size or focus on adversarraai robustness. contrary, our method introduces NCA lightweight houg-and-play adaptors into base-level ViTs, thus enhancing their clean accuracy and robustness against both adversarial samples and OOD inputs.
the
2.3 Neural Cellular Automata
Gilpin 20
demonstrates that CA can be represented by convolutional neural networks. By extending Mordvintsev et al. [4 propose NCA order to mimic the biological cell interactions and
20]
model morphogenesis. Following this idea, several works apply NCA in computer vision, including
texture synthesis [43] 46
49
50 image generation [33 47 51 62), and image segmentation
32 57 Randazzo et al. (55 propose applying NCA for modeling collective intelligence on image
classification tasks, but the limitation to binary images restricts its practical application. Tesfaldet
al. 62 first establish the connection between ViT and NCA via recurrent local attention.
leads to more robust model for handling image corruptions in image inpainting tasks. However, their application is limited image impainting on small datasets such as MNIST [1] and CIFARIO [36]. The NCA in (62) attempts to emulate a ViT whereas our approach distinguishes itself by not doing so. We first applies NCA in image classification on ImageNetl K with base-level ViT models. Moreover, we propose the new Dynamic Interaction for efficient cell interaction modeling, reducing the computational overhead, and for enhancing the model performance.
Method
The overview of our method is shown in Figure 2
this section, we first review the NCA model and
ViT architecture. In Section 3.1 establish the connection between NCA and ViT, in terms of token interaction modeling. We then present the design of our AdaNCA Section 3.2 We insert AdaNCA into the middle layers of VIT to improve its robustness. We introduce the relationship between the

---

insert position of AdaNCA and the relative improvements of model robustness in Section 3.3 This relationship leads to the algorithm for deciding the most effective placement for AdaNCA.
3.1 Preliminaries
Vision Transformers Vision Transformers (ViTs) operate on token maps X E R NNC, where the number of tokens is N and each token is represented by a C-dimensional vector. ViTs learn the interaction between these tokens via self-attention [64] and compute the interaction result Xattn, as described in Equation [
QK
Xattn =a
V.
(I)
VD
Q.K, V stand for query. key. and value, respectively. They are deduced from different linear projections of the imput, i.e., Q XWQ. q, ( XWK. V XWv, where WQ. WK. Wy e RCDD, D is the hidden dimensionality in self-attention. is Softmax. After self-attention, tokens are fed into Multilayer Perceptron (MLP) to obtain the updated representations o
X fo ptttt).
(2)
is the MLP and ! stands for its parameters. The self-attention and MLP form single ViT block. and a ViT model can be built via stacking ViT blocks.
Neural Cellular Automata NCA aims modeling cell interactions. In the 2D domain, cells live on a 2D grid with size H x W. Each cell is represented by a vector with dimensionality C. All cells collectively define the cell states S "fxxxx In single step of NCA, to generate the interaction output Sz. cells first interact with their neighborhoods for an information exchange in the Interaction stage 49
the interaction is typically instantiated via depth-wise convolutions [44 46 48
Sz (S C2,.,,,. CM])
C, is the ith convolutional kernel, and M denotes the total number of kernels. "@" denotes depth-wise
convolution. The kernels can either be fixed 49 48 or learnable 44 46). The results of all kernels
are concatenated channel-wise in the @ operation. Sy is then passed to an MLP in the Update stage
40
(4) Sout is then used to update the cell states in residual scheme. f is the MLP, and 0 stands for its parameters. Typically, NCA uses the simplest MLP, with two linear layers and one activation between them. M E " xC, sampled from Bernoullipp), is random binary mask to introduce stochas- ticity in NCA; ensures asynchronicity during the cell updates 45] + is point-wise multiplication.
S fo(ST) 3 M.
NCA learns an underlying dynamic that governs the cell behaviors 50 as depicted by the stochastic differential equation (SDE) Equation 5
aS Ø
(S) M = So (SS @ C2, Cml))el + M.
F represents operations in Interaction as well as Update stages. θ is the containing trainable parameters in the two stages. Discretizing the SDE with At = naturally results in a recurrent residual update scheme:
S S Fo o M.
After t = T' steps, the cell states ST is extracted to accomplish certain downstream tasks. The traditional NCA involves several other specific designs though, in our case, is impractical adapt them. We provide a discussion on this topic in Appendix
3.1.1 Connecting NCA and VIT
Both NCA and ViT learn interactions between set of elements, i.e., tokens in ViT and cells in NCA. Hereafter, we refer to cell in NCA as a token, aligning it with the concept in ViTs. The asynchronicity [45) introduced by the random mask M can be regarded as a cell-wise stochastic depth 31, a more fine-grained version of the sample-wist stochastic depth. In previous NCA works, stochasticity is maintained during testing [6)) Such scheme problematic our case because (1) test-time stochasticity produces obfuscated gradients [3], leading to the circumvention of adversarial

---

St Dilation s
Dynamic Interaction
Multi-scole
Dynamic Interaction
DWConv1.s
DWConvac,s
Conv2D
Update
Weighed Sum
Weights
Token Map S
s
Weighed Sum
St+1
MLP
Point-wise Multiplication
Conv2D
Weights
Figure 3: Overview of AdaNCA architecture. Instead of concatenating the interaction results generated by the depth-wise convolutions, our Dynamic Interaction conducts a point-wise weighted on them improve the efficiency and enhance the performance. The weights obtained based on the token states so that each token can dynamically adjust, according to the inputs, the interaction strategy. The Multi-scale Dynamic Interaction aggregates the results from Dynamic Interaction where the convolutions have different dilation rates. Then, to finish one step of evolution, the output fed into the Update stage.
attacks, and (2) the model can output different results given the same inputs. To this end, adopt the strategy in dropout-like techniques [3], which compensates activation values during training.
Given M Bernoullipp), the evolution of NCA in AdaNCA is defined
Train: st+1 = (S о M; Test S S FF((S)).
We discuss the necessity of such a scheme in Section D.1 in the Appendix, Furthermore, NCA typically outputs the cell states at a random time step T, resulting in random update steps for all cells. Such randomness ensures the stability of NCA across various time steps [44]. Finally, the recurrent steps of NCA during single training epoch enable the exploration of wide range of cell states. In the early stage training, the model not adequately trained hence serves as a source of noise to itself through the recurrence. With all these components, the trained model can effectively handle the variability and unpredictability the input thus be robust against noisy input. Our ablation studies in Section 4.3 demonstrate the effectiveness of these strategies in enhancing the model performance.
3.2 AdaNCA architecture
The architecture of AdaNCA is shown in Figure 3 It shares similar update scheme with the standard NCA, as described in Equation 7 but, more computationally efficient due to the proposed Dynamic Interaction stage. All convolutional kernels for token interaction are trainable. In the following paragraphs, we first present the design of the Dynamic Interaction stage and then introduce way for more efficient token interaction by using multi-scale Dynamic Interaction.
3.2.1 Dynamic Interaction
The Interaction stage in the original NCA performs channel-wise concatenation of the M output tensors from different depth-wise convolutions. Whereas, our Dynamic Interaction computes a weighted sum of those results. Specifically, weight computation network Wz takes the token map I XXCC as input and outputs per-token scalar weights Wim xXX for each of the M kernels. We modify Equation B to Equation
M
C
SpT = (s Wz)(S @
..,, CC,l) = E WT)S =
Σ
Wim
Im C Smc (8)
m=1
5

---

where Smc E IR IRxx depth-wise convolution. We instantiate the weight computation module Wr by using two-layer convolutional network. The first layer transforms the dimensionality from C' to M, and the second layer computes the actual weights, thus producing M scalars for each token. Both layers use 3 x convolutions to factor in information from both the token and its neighbors. To stabilize training, we add a batch normalization between the two convolutions. Our design of the weight computation network coincides with the one in [23] Although, our focus is on extracting various information from the same neighborhoods rather than on aggregating data from different neighborhoods.
SpZ, Sm E RR"xWxC, 'x' denotes the convolution. Recall that "@" is the
3.2.2 Multi-scale Dynamic Interaction
Inspired by 48) which uses multi-scale token Interaction to facilitate long-range token communica- tion, we propose multi-scale Dynamic Interaction. Concretely, all convolutions in Equation g now have more degree of freedom in dilation. Dilation represents the current operating scale being s. and 8 E (1.,...., S}. Hence, the original Dynamic Interaction is a special case where S 1. To increase the feature expressivity, we perform weighted per-token weights WMs are generated by a network Was as described in Equation
the outputs of all scales, where the
SMDZ= = E(s WM = E(s Wa)
Σ
W @ SSIse
(9)
where W Ma DI RHxxxl SMII, SpI. RB The Wz in Equation is shared across all scales. The weight computation network WM mirrors that of Wz.
3.3 Insert positions of AdaNCA
Given a VIT and an AdaNCA, to maximize the robustness improvements, need to determinc where to insert AdaNCA. To this end, we first establish the correlation between the placement of AdaNCA and the robustness enhancement it brings. Motivated by the fact that the network redundancy contributes to the model robustness 28 we bypothesize that the effect of AdaNCA should correlate to the layer redundancy corresponding the insert position. To quantify the redundancy, we propose the Set Cohesion Index k. Given a trained model with L layers and two layer indices i.j E {1, (1,2,... .... L}
where j. x)i,j) is defined in Equation 10
m((,))
1
Σ m.n
Sim(m, n)) -
Lis1
Σ
Sim(m1, n)
(j 1)
1)(j
1)
1,sh
Σ
Sim(m, n1)
(L (L-j)( 1i+1) me(t.jl. Ebj+1.4
(10)
1 stands for the indicator function. Sim(m, n) is the function for quantifying the output similarity between layer m and n. We choose Centered Kernel Alignment (CKA) [35), a common metric for measuring layer similarities inside between neural networks (25). A higher x stands for a more cohesive layer set defined by layers from to j. Inserting AdaNCA after layer would partition the network into two layer sets, and we can compute the sum of of the layers before and after i, i.e., KC(i) corresponds to position i. We assume that AdaNCA will not change the layer similarity structure because it is too small compared to single layer in all ViTs.
+ k(i + 1, L). This serves
quantification of the network redundancy that
In addition the quantification of the network redundancy, the robustness improvement, brought by AdaNCA, is quantified using the relative increase in the attack failure rate of the AdaNCA-inserted models and the corresponding baseline. Specifically, if a model can achieve a clean test accuracy as well as a' accuracy under adversarial attacks, the attack failure rate is 3 a'' The robustness improvement 7 is then defined as 7 -AA In our experiments, we find that y is 0.6938, p < 0.001).
Pbasc
significantly correlated with the network redundancy K. (Pearson correlation We refer readers to Appendix[A]oor for details of the experiments. The results validate our hypothesis and indicate that AdaNCA should be inserted into the position that can maximize network redundancy. We develop a dynamic programming algorithm to find these positions and refer readers Appendix A. for the details.

---

Table 1: The performance of AdaNCA-enhanced ViTs and their corresponding baselines. We report the mCE for IM-C (lower is better) and accuracy for other benchmarks. AdaNCA consistently improves the clean accuracy well the robustness to adversarial and out-of- the baseline models. Note that our models also outperform the larger baselines (* sign), indicating that the performance improvement does not merely originate from the increase in the number of parameters or FLOPS. The TAPADL method (33 can lead to more vulnerable models compared to the baselines (green numbers). Bold indicates the best model. i: the test. is conducted models C.9
inputs of
pre-trained on ImageNe122K [42]. see Appendix C.9
Model
Params (M)
FLOPS
(G)
ImtgeNNt Cleas Acc.
PGDD [1]
CW
APGD DLR ( APGD-CE CE
Adversriaa Inpuis
IM-A L3
CM) 23
OOD inguts
[M-R 20
IM SK 07
RVT-8 23
88.5
17.7
82.7
29.9
21.57
21.91
31,4"
26.5
46.8
48.7
36.0
TAPADI BOADI-RVT
21
89.4
17.
83.1
27.6
10.3
12.7
26.8
32.7
44.7
50.2
18.6
RVT AdaNCa
91.0
19.0
83.3
36.7
30.2
33.2
36.2
31.9
43.2
51.7
39,0
FAN-B [8]
50.4
11.7
83.9
15.0
7.5
10.4
13.1
39.6
46.
53.7
40.8
TAPADI-FAN
50.3
11.8
84.3
15.6
0.2
13.5
16.9
42.3
48.7
54.6
40.7
PAN-B-AdoNVA
51.1
1244
84.1
20.3
10.6
14.1
19.1
42.9
44.7
53.4
41.0
Swin 35
$788
15.4
83.4
21.3
13.4
15.6
23.1
35.8
$4.3
46.5
32.4
Swin
94.1
16.7
83.3
22.8
14.6
15.9
23.8
35.2
53.2
46.9
35.7
Siete B AdaNCA
90.7
16.3
83.7
24.1
20.5
25.1
34.8
36.0
51.5
48.3
35.5
CieVIT- T6
86.5
CoVVIT-
63
19.2
24.1
10.0
89.0
19.0
Table 2: Comparisons of corruption error on each corruption type of ImageNet-C. AdaNCA consistently improves the performance of the baseline model in all categories (Swin-B), and can achieve better results in most categories compared the SOTA method (TAPADL-RYT). Bold indicates the best model. Lower is better.
Model
mCF
Gauss
Noise Shot
Impulse
Defocts
Glaes
Blur
Moxion
Zorm
Snow
Frost
Weather
Fog
Bright
Contrast
Elasse
Digital
Piselate
JPDG
SwissB [33
54.3
43.8
45.7
43.8
61.6
73.2
55.5
66.5
50.0
47.4
47.9
42.1
36.7
68.8
66.4
62.6
Suin-BAAslNNCA
51.5
39.5
40.9
39.8
99.5
69.5
55.0
65.6
49.4
47.1
40.8
4006
37.0
66.3
59.9
61.1
TAPADI -VVT 23
4.7
34.6
36.1
34.0
53.9
62.3
52.2
60.1
41.9
44.9
35.8
374
31.9
57.1
41.1
47.2
RY7-8AA3NNCA
432
33.9
36.1
339
53.7
99.1
47.8
60.5
42.6
39.0
33.9
363
31.7
53.4
40,0
46.2
Experiments
We use four ViT models as the baseline: Swin-base (Swin-B) 33 JAN-Base-Hybrit (FAN-B) 78 RVT-Base-Plus (RVT-B) [42], and ConViT-Base (Con ViT-B) [6]. They include hierarchical model (Swin), a convolution-atteniic share the same structure (RVT and Con ViT). RVT is specifically built to be a robust model, whereas Con ViT is not. All four models are equipped with different kinds of local structures. We use two SOTA models in terms of robustness against out-of-distribution (OOD) data, TAPADL-RVI and TAPADL-FAN 23 for comparison. Note that the SOTA method involves training with an additional loss (ADL) and that, for completeness, we keep the results from the TAPADL models trained with such a loss. However, as our focus is on the effect of architectural changes, we do not incorporate the ADL in training the AdaNCA-enhanced ViTs. We follow the training scheme for each model, respectively, to train the AdaNCA-equipee analysis presented in Section 3.3 to decide the optimal position to insert multiple AdaNCA modules, in which the ImageNetIK tre-trdined weights for analysis are obtained from the Py Torch Image Models library (69. To balance between the computational cost and robustness improvement, limit the number of AdaNCA to two or three, depending on the model architecture. The recurrent time step of all AdaNCA is chosen from ( 21, [2, 4]. 33 5]), and we follow the design principle of ViT; it to put more computation in the middle or high layers [88. We fix the random steps during testing
hybrid model (FAN), and two regular models in which all layers
model from scratch on ImageNet I.. We conduct the
single integer chosen from the range to achieve precise results and ensure non-stochasticity during adversarial attacks. All the activation functions used in the MLP in AdaNCA GELU 2 and the input, hidden, and output dimensionalities of the MLP are all the same. We refer readers to Appendix C.6 for the details of the training and insert scheme of AdaNCA. All of our experiments are performed on four Nvidia A100 GPUs.
4.1 Results on Image Classification
We test all models on the ImageNet1 K validation set for clean accuracy. For the adversarial robustness evaluation, we choose common adversarial attack methods PGD [41, CW (5, APGD-DIR (9). and APGD-CE 19]. Moreover, also include the natural adversarial examples in ImageNet-A
7

---

05
64
Intorcdenn Scalss
Swin-B, -0.47
Swin-B-AdonCA, Kmean Kmeon=0.52 =0.52
Figure 5: Ablation on the (a) scales and (b) num- ber of kernels used our multi-scale Dynamic Interaction. Overly large scales can undermine the performance and so do too many or too few kernels. We choose S 2, .M 4 to balance between the clean accuracy and robustness.
Figure 4: Pair-wise layer similarities. Layer sets are marked in red boxes. Swin-B-AdanCA, has a clearer stage partition, which might be attributed to AdaNCA acting as an information transmitter between different layer sets.
For the PGD attack, we align with the settings used in 42 max magnitude e step size a 0.5, steps - 5. We refer readers to the Appendix C.7 for details of the other attacks. For testing the OOD generalization, and ImageNet-Skette (IM-SK) [67]. We report the mean corruption error (mCE) on ImageNet-C and accuracy on all other kinds of robustness benchmarks in Table I Our results highlight that AdaNCA-enhance ViTs consistently outperform corresponding baselines in various robustness tests well in terms of clean accuracy. Importantly, the enlarged baseline models sign) do not bring comparable improvements to AdaNCA, suggesting that the enhancements do not merely stem from the increase computational budgets. However, the existing method [23] that introduces local structure into ViTs can potentially undermine the adversarial robustness of the baselines. In Table 2 conuptiins in ImageNet-C. The results show that AdaNCA enhances robustness without the trade-off seen methods that ignore texture information [1 53 While these methods may improve mCE 5 In contrast, AdaNCA consistently improves robustness across most categories. We refer readers to Appendix C for more results.
(IM-A) 13
ImageNet-C. (IM-C) 28 ImageNet-R (IM-R) (30]
we conduct an in-depth study on the corruption errors of the different categories of common
for non-Blur noise, they often worsen mCE for Blur noise
4.2 Layer similarity structure
Our key assumption in Section 3.3 is that AdaNCA will not change the layer similarity structure due to its small size, and that is why we use pre-trained networks to conduct this analysis. Here, we examine the pair-wise layer similarities in Swin-B (38 and Swin-B- B-AdaNCA in Figure 4 Kmean is the mean of x from all layer sets. We refer readers to Appendix C.14 for more results. AdaNCA not only preserves the original layer similarity structure but also contributes
clearer stage partition, validating our assumption Section B.3] The results might be attributed to the fact that AdaNCA transmits information between different layer sets and thus layers inside each do not bother adapting to the layers outside the set.
4.3 Ablation studies
We conduct ablation studies ImageNet100, 100-class subset of TmageNetIK. Previous studies [5] 72 774 have shown that ImageNet 100 serves as representative subset of ImageNet K. Hence, we can obtain representative results for the self-evaluation of the model while efficiently using the computational resources. All the ablation experiments are based on a Swin-tiny (38) model. We insert it after the fourth layer to obtain the best robustness improvement, according to our analysis in Section 3.3 and Appendix A First, we ablate on two hyperparameters, the number of convolutional kernels used in the Dynamic Interaction stage (M) and the maximum scale (S) used in the multi- scale Dynamic Interaction stage. The Clean Accuracy and Attack Failure Rate are shown in Figure [5] The attack failure rate quantified using the same method as Section 3.3 and Appendix A Multi-scale interaction can contribute to the performance while overly large scale can lose local information and complicate the process of selecting the interaction neighbors. This issue also observed in previous work [62]. Increasing the number of kernels benefits the performance while too many

---

Table 3: Ablation studies design choices. Each of the AdaNCA design choices contributes to improved performance and robustness. Baseline is a Swin-tiny tiny 38 Bold indicates the best model.
model trained on ImageNet-100,
Exp. Type
Recur
StocU
RandS
Dynin
Params (M)
FLOPS (G)
Accuracy (f)
Attack Failure Rate (U)
Baseline
x
27.59
4.5
86.56
12.29
1
28.97
87.36
19.04
x
27.94
86,92
19.56
Ablation
x
/
27.94
4.7
87.12
19.34
x
27.93
4.7
86.72
21.98
Ours
27.94
4.7
87.18
22.35
kernels undermine the robustness. According the results, we choose S 2, M 4. We then perform ablations on several design choices:
Recurrent update (Recur). Ablation on unrolling the recurrence with average time step in AdaNCA into T independent AdaNCA with time step being 1.
Stochastic update (StocU). Ablate the stochastic update during training, leading to globally synchronized update of tokens (45)
Random step (RandS). Change the recurrence time step from randomly chosen integer in range [Ti, T2] to [(T + 72) It cannot be turned on without recurrence.
Dynamic interaction (DynIn). Ablate the Dynamic Interaction so that the interaction results are simply summed together. The number kernels remains the same.
As shown in Table 3 the highest robustness improvement is achieved with all components. Among them, turning off the recurrence leads to the largest drop in the robustness, as recurrence allows the model to explore more cell states than finishing the update in single step. While it achieves the highest clean aceuracy, it uses <4x more parameters than our method, and the improvement of the clean performance is likely due more parameters. Without any of the two sources of randomness, stochastic update and random steps, the model cannot adapt to the variability of the inputs and thus exhibits vulnerabilities against adversarial attacks. Finally, turning off our Dynamic Interaction will cause drops in both clean accuracy and robustness, as tokens cannot decide their unique interaction weights and thus cannot generalize to noisy inputs. We provide extended ablation studies in Section D.2 in the Appendix.
4.4 Noise sensitivity examination
drawback of adversarially robust models their increased sensitivity noise in specific frequency bands [60]. For instance, while adversarially trained models are robust against adversarial attacks, they can sensitive to noise from a larger frequency- range compared to standard models [60]. Here, we use the method and data from [60] to examine the noise sensitivity of ViTs. Specifically, we evaluate the classification performance on set of images that are mixed with noise of varying magnitudes and frequencies. The noise is sampled from Gaussian distribution with zero mean and the standard deviation indicates magnitude. Then, it is filtered within various pretial-rrquency bands, resulting in different frequencies of noise. Higher classification accuracy on specific noise type indicates that the model less sensitive that noise. Figure figure66presenss presents the results, including human data sourced from [60). Our findings demonstrate that AdaNCA enhances ViTs by reducing the sensitivity to noise with certain frequency components, equipping ViTs with more human-like noise-resilient abilities. Crucially, AdaNCA improves the model robustness differently than adversarial training since the AddaCCA-enhance ViTs do not exhibit increased sensitivity to the noise. We quantitatively validate the conclusion and refer readers Appendix C.16 for the details.
5 Limitation
AdaNCA has certain limitations. First, the AdaNCA-equipel ViTs cannot adapt unseen recurrent steps of AdaNCA, which limits the generalization ability. For example, if the training step range of AdaNCA is [3,5], it cannot produce meaningful results with the test step being 6. AdaNCA

---

Swin-Base
Swin-B-AdaNCA
Human Classification Accuracy
Magnitude of Noise
0.02
ConViT-Basse
ConViT AAdaTT-BAAdaNA
-40
0.08
28
Frequency of Noise (cycles/image)
Figure 6: Classification accuracy of humans and VITs on noisy images. The images are perturbed by Gaussian noise with different standard deviations (Magnitude of Noise) and are filtered with different
bands (Frequency of Noise). AdaNCA improves the accuracy on images with certain types of noise (dotted black boxes), indicating that it makes ViTs less sensitive to them. Quantitative results are presented Appendix C.16.
introduces a non-negligible computation into the original architecture. Our experiments are conducted on ImageNetIK with the image size of 224 224. Whether AdaNCA can lead to impressive improvements on larger-scale problems, e.g., ImageNe122K, remains question. The size of the images can also affect the efficiency of token interaction.
Broader Impact
AdaNCA contributes to more robust ViTs, facilitating their usage in real-world scenarios. We bridge two powerful models, NCA and ViT, on large-scale image classification, potentially encouraging research dedicated to their synergistic combination in more practical settings. Our findings on AdaNCA improving network redundancy can stimulate more works on architectural robustness in deep learning that involves increasing the redundancy to enhance robustness. In the context of this paper, we believe that AdaNCA does not introduce any significant negative implications.
7 Conclusion
We have proposed AdaNCA, an efficient Neural Cellular Automata (NCA) that, when inserted between their middle layers, improves ViT performances and robustness against adversarial attacks as well as out-of -it-o---iittitttionn inputs. We design our model by connecting NCA and ViT, in terms of token interaction modeling, and we propose Dynamic Interaction improve the computational efficiency of standard NCA. Exploiting the training strategies and design choices in NCA, i.e., stochastic update, random steps, and multi-scale interaction, we further improve the AdaNCA-enhanced ViTs' clean accuracy and robustness. To decide the placement of AdaNCA, we propose the Set Cohesion Index that quantifies the network redundancy via layer similarity and conclude that AdaNCA should be inserted between two layer sets that consist of redundant layers. Our results demonstrate that AdaNCA consistently improves ViTs performances and robustness. Evidence suggests that the mechanism by which obtain improvement reduces the sensitivity of ViTs to certain types of noise and makes the moise-resilient ability of ViTs similar to that of humans.
10

---

References
[1] The MNIST database of handwritten digits.
[2] Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein. Square at- tack: query-efficient black-box adversarial attack via random search. In European Conference on Computer Vision, pages 484-501. Springer, 2020.
[3] Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses adversarial examples. In International Conference on Machine Learning, pages 274-283. PMLR, 2018.
[4] Srinadh Bhojanapalli, Ayan Chakrabarti, Daniel Glasner, Daliang Li, Thomas Unterthiner, and Andreas Veit. Understanding robustness of transformers for image classification. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10231-10241, 2021.
[5] Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium on Security and Privacy (SP), pages 39-57. IEEE, 2017.
[6] Aditya Chattopadhay. Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian. C Generalized gradient- based visual explanations for deep convolutional networks. In 2018 IEEE winter conference applications of computer vision (WACV), pages 839-847. IEEE, 2018.
[7] Yixin Cheng, Grigorios Chrysos, Markos Georgopoulos, and Volkan Cevher. Multilinear operator networks. The Twelfth International Conference on Learning Representations, 2023.
[8] Xiangxiang Chu, Zhi Tian, Bo Zhang, Xinlong Wang, and Chunhua Shen. Conditional positional encodings for vision transformers. The Eleventh International Conference on Learning Representations, 2022.
[9] Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-fre free attacks. In International Conference on Machine Learning, pages
PMLR, 2020.
[10] Zihang Dai, Hanxiao Liu, Quoc V Le, and Mingxing Tan. CoAtNet: Marrying convolution and attention for all data sizes. Advances in Neural Information Processing Systems, 34:3965-3977, 2021.
[11] Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. In International Conference on Machine Learning, pages 7480-7512. PMLR, 2023.
[12] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large- scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255. IEEE, 2009.
[13] Josip Djolonga, Jessica Yung, Michael Tschannen, Rob Romijnders, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Matthias Minderer, Alexander D' Amour, Dan Moldovan, et al. On robustness and transferability of convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16458-16468, 2021.
[14] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 6 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020.
[15] Arthur Douillard, Alexandre Ramé, Guillaume Couairon, and Matthieu Cord. DyTox: Trans- formers for continual learning with dynamic token expansion. In Proceedings of the IEFF/VVF Conference Computer Vision and Pattern Recognition, pages 9285-9299 2022.
11

---

[16] Stéphane D' Asscoli, Hugo Touvron, Matthew L Leavitt, Ari S Morcos, Giulio Biroli, and Levent Sagun. ConViT: Improving vision transformers with soft convolutional inductive biases. In International Conference on Machine Learning, pages 2286-2296. PMLR, 2021.
[17] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained enns are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations, 2018.
[18] Robert Geirhos, Bern-Herrik. Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shorteut learning in deep neural networks. Nature Machine Intelligence, (11)):65-88 673, 2020.
[19] Jacob Gildenblat and contributors. Pytorch library for cam methods. https://github. com/
jacobgil/pytorch- -rad--rddddd cam 2021.
[20] William Gilpin. Cellular automata as convolutional neural networks. Physical Review E, 100 (3):032402, 2019.
[21] Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Daniel A Roberts. The unreasonable ineffectiveness of the deeper layers. arXiv preprint arXiv:2403. 17887, 2024.
[22] Jianyuan Guo, Kai Han, Han Wu, Yehui Tang, Xinghao Chen, Yunhe Wang, and Chang Xu. CMT: Convolutional neural networks meet vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12175-12185, 2022.
[23] Yong Guo, David Stutz, and Bernt Schiele. Robustifying token attention for vision transformers. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 1 17557- 17568, 2023.
[24] Xing Han, Tongzheng Ren, Tan Nguyen, Khai Nguyen, Joydeep Ghosh, and Nhat Ho. Designing robust transformers using robust kernel density estimation. Advances in Neural Information Processing Systems, 36, 2024.
[25] Yena Han, Tomaso A Poggio, and Brian Cheung. System identification of neural systems: If we got it right, would we know? In International Conference on Machine Learning, pages 12430-12444, PMLR, 2023.
[26] Ali Hassani, Steven Walton, Jiachen Li, Shen Li, and Humphrey Shi. Neighborhood attention transformer. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 6185-6194 2023.
[27] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[28] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations, 2018.
[29] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELUs). arXiv preprint 1rX1v:1606.084155 2016.
[30] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8340-8349, 2021.
[31] Gao Huang. Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October / /1-14, 2016, Proceedings, Part IV 14, pages 646-661. Springer, 2016.
[32] John Kalkhof, Camila González, and Anirban Mukhopadhyay. Med-NCA: Robust and lightweight segmentation with neural cellular automata. In International Conference on Infor- mation Processing in Medical Imaging, pages 705-716. Springer, 2023.
12

---

[33] John Kalkhof, Arlene Kühn, Yannik Frisch, and Anirban Mukhopadhyay. Frequeney-time diffusion with neural cellular automata. arXiv preprint arXiv:2401.06211, 2024.
[34] Hoki Kim. Torchattacks: A pytorch repository for adversarial attacks. arXiv preprint arXiv:2010.01950 2020.
[35] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning, pages 3519-3529. PMLR, 2019.
[36] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[37] Yawei Li, Kai Zhang, Jiezhang Cao, Radu Timofte, Michele Magno, Luca Benini, and Luc Van Goo. LocalViT: Analyzing locality in vision transformers. In 2023 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages
IEEE, 2023.
[38] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin Transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012-10080 2021.
[39] Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, et al. Swin Transformer v2: Scaling up capacity and resolution. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 12019, 2022.
[40] Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 11976-11986, 2022.
[41] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference Learning Representations, 2018
[42] Xiaofeng Mao, Gege Qi, Yuefeng Chen, Xiaodan Li, Ranjie Duan, Shaokai Ye, Yuan He, and Hui Xue. Towards robust vision transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12042-12051, 2022.
[43] Alexander Mordvintsev and Eyvind Niklasson. f--NAA: Texture generation with ultra-compact neural cellular automata. arXiv preprint arXiv:2 I1 13545,, 2021.
[44] Alexander Mordvintsev, Ettore Randazzo, Eyvind Niklasson, and Michael Levin. Growing neu- ral cellular automata. Distill, 2020. doi: 1.223915ddistill.000 https://bssilllpbbb2220zvvvwrr
ca.
[45] Eyvind Niklasson, Alexander Mordvintsev, and Ettore Randazzo. Asynchronicity in neural cellular automata. In Artificial Life Conference Proceedings 33, volume 2021, page 116. MIT Press One Rogers Street, Cambridge, MA 02142-1209, 2021.
[46] Eyvind Niklasson, Alexander Mordvintsev, Ettore Randazzo, and Michael Levin. Self- organising textures. Distill, 6())e00027-003, 2021.
[47] Maximilian Otte, Quentin Delfosse, Johannes Czech, and Kristian Kersting. Generative adver- sarial neural cellular automata. arXiv preprint arXiv:2108.04:: 2021.
[48] Ehsan Pajouheshgar, Yitao Xu, Tong Zhang, and Sabine Süsstrunk. DyNCA: Real-time dynamic texture synthesis using neural cellular automata. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2023.
[49] Ehsan Pajouheshgar, Yitao Xu, Alexander Mordvintsev, Eyvind Niklasson, Tong Zhang, and Sabine Süsstrunk. Mesh neural cellular automata. ACM Trans. Graph., 2024. doi: 10.1 3658127. URL https:://do...... org/10. 10. 1145/3658127
[50] Ehsan Pajouheshgar, Yitao Xu, and Sabine Süsstrunk. NoiseNCA: Noisy seed improves spatio-temporal continuity of neural cellular automata. arXiv preprint arX1v:2404.00279, 2024.
13

---

[51] Rasmus Berg Palm, Miguel Gonzalez Duque, Shyam Sudhakaran, and Sebastian Risi. Varia- tional neural cellular automata. In International Conference on Learning Representations 2022, 2022.
[52] Zhiliang Peng, Wei Huang, Shanzhi Gu, Lingxi Xie, Yaowei Wang, Jianbin Jiao, and Qixiang Ye. Conformer: Local features coupling global representations for visual recognition. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 367-376, 2021.
[53] Xinkuan Qiu, Meina Kan, Yonghin Zhou, Yanchao Bi, and Shiguang Shan. Shape-biased enns are not always superior in cut-of-distributioo robustness. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2326-233333 2024.
[54) Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jon Shlens. Stand-alone self-attention in vision models. Advances Neural Information Processing Systems, 32, 2019.
[55] Ettore Randazzo, Alexander Mordvintsev, Eyvind Niklasson, Michael Levin, and Sam Greydanus. Self-classifying mnist digits. Distill, 2020. doi: 10.23915/dissill.00077.00
[56] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115:211-220 2015.
[57] Mark Sandler, Andrey Zhmoginov, Liangcheng Luo, Alexander Mordvintsev, Ettore Randazzo, al. Image segmentation via cellular automata. arXiv preprint arXiv:2008 04965, 2020.
[58] Baifeng Shi, Yale Song, Neel Joshi, Trevor Darrell, and Xin Wang. Visual attention emerges from recurrent sparse reconstruction. In International Conference on Machine Learning, pages 20041-20056 PMLR, 2022.
[59] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):192--1988 2014.
[60] Ajay Subramanian, Elena Sizikova, Najib Majaj, and Denis Pelli. Spatial-frequency channels. shape bias, and adversarial robustness. Advances in Neural Information Processing Systems, 36, 2024.
[61] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfel- low, and Rob Fergus. Intriguing properties neural networks. In 2nd International Conference on Learning Representations, ICLR 2014, 2014.
[62] Mattie Tesfaldet, Derek Nowrouzezahhrai, and Chris Pal. Attention-based neural cellular au- tomata. Advances in Neural Information Processing Systems, 35:8174-8186 2022.
[63] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pages 10347-10357. PMLR, 2021.
[64] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in Neural Information Processing Systems, 30, 2017.
[65] Ashish Vaswani, Prajit Ramachandran, Aravind Srinivas, Niki Parmar, Blake Hechtman, and Jonathon Shlens. Scaling local self-attention for parameter efficient visual backbones. In Proceedings of the JEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 12894-12904, 2021.
[66] viddggg. Code repo for: RVT: Towards robust vision transformer. https: http::///tttttttt com/
Stddggg/Robuust Vision- Transformes 2021.
14

---

[67] Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global repre- sentations by penalizing local predictive power. Advances Neural Information Processing Systems, 32, 2019.
[68] Yujing Wang. Yaming Yang, Jiangang Bai, Mingliang Zhang, Jing Bai, Jing Yu, Ce Zhang, Gao Huang, and Yunhai Tong. Evolving attention with residual convolutions. In International Conference Machine Learning, pages 10971-109800 PMLR, 2021.
[69] Ross Wightman. Pytorch image models.
atttpss//githabb.com///gggg
pytorch- image-models 2019.
[70] Haiping Wu, Bin Xiao, Noel Codella, Mengehen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. CvT: Introducing convolutions to vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2021.
[71] Tete Xiao, Mannat Singh, Eric Mintun, Trevor Darrell, Piotr Dollár, and Ross Girshick. Early convolutions help transformers see better. Advances Neural Information Processing Systems, 34:30392-30400, 2021.
[72] Shipeng Yan, Jiangwei Xie, and Xuming He. DER: Dynamically expandable representation for class incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3014-3023 3023, 2021.
[73] Jianwei Yang. Chunyuan Li, Pengchuan Zhang, Xiyang Dai, Bin Xiao, Lu Yuan, and Jianfeng Gao. Focal attention for long-range interactions in vision transformers. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 30008-30022 Curran Associates, Inc., 2021. URL http::tt///otcccttttttttt www.gss..enrpppcmmm wwwpuuuipiscppppprrepeeee /pppee__iiieeeepprrr////////
fc Ecc.6121bbbb2220000bb900001-3133
pdf
[74] Tan Yu, Xu Li, Yunfeng Cai, Mingming Sun, and Ping Li. S2-mlp: Spatial-shift mlp architecture for vision. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 297-306, 2022.
[75] Kun Yuan, Shaopeng Guo, Ziwei Liu, Aojun Zhou, Fengwei Yu, and Wei Wu. Incorporating convolution designs into visual transformers. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 579-588, 2021.
[76] Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. takens-to-token vit: Training vision transformers from scratch on imagenet. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 558-567, 2021.
[77] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transform- ers. Proceedings of the IEEE/CVF Conference on Computer Vision and Paltern Recognition, pages 12104-12113, 2022.
[78] Daquan Zhou, Zhiding Yu, Enze Xie, Chaowei Xiao, Animashree Anandkumar, Jiashi Feng, and Jose M Alvarez. Understanding the robustness in vision transformers. In International Conference Machine Learning, pages 27378 273944. PMLR, 2022.
15

---

Appendix: Table of Contents
Establish the correlation between AdaNCA placement and robustness improvement
17
A.1 Dynamic programming for AdaNCA placement
18
Notes on statistical significance
18
ImageNetlK experiments
20
C.1
AdaNCA settings for each model
20
C.I.I Discussion on the choice of AdaNCA steps
................................
20
C.2
Discussion on the model choices
21
Discussion on the hyperparameters of MLP inside AdaNCA
21
The effect of AdaNCA placement robustness on ImageNetIK
Details of architecture change in Swin- Siin-Base* and Con -iseBase'
C.6
Training details
22
Adversarial attacks
22
C.7.I
Choice of adversarial attacks
22
C.2.2
Black-box attack
23
C.8
The effect of the range of the steps in random step training
23
............
C.9
Reason for using ImageNet22K model for RVT
23
C.10
Visualization of attention maps
24
C.11
Integrating AdaNCA into pre-trained ViT models
24
C.12
Additional results on comparison with current methods
25
C.13
Additional results on same model architecture but with more layers (* models)
25
C.
14
Additional results on layer similarity structure
26
C.15
Additional results category wise mCE on ImageNet-C
26
C.
16
Additional results on noise sensitivity examination
26
77
Datasets information and license
28
18
Model and code license
28
Extended ablation studies
29
D.1 The dropout- like strategy
29
D.2 Dynamic Interaction
29
D.3 Multi-scale Dynamic Interaction
Differences between traditional NCA and AdaNCA
31
16

---

RVT-S-AdaNCA
FAN-S-AdaNCA
Swin- --ia-T-AdaNCA
Learning rate
2.5e-4
2.5e-4
2.5c-4
Batch size
256
256
256
Model EMA decay
0,99992
0,9999
Stochastic depth
0.1
0.25
0.1
Random erase prob
0.25
0.3
0.25
Gradient clip
None
None
5.0
MixUp
0.8
0.8
0.8
Label smoothing
0.1
0.1
0.1
Min learning rate
le-5
1e-6
1e-5
Weight decay
0.05
0.05
0.05
Table 4: Hyperparameters used in equipe- ViT training for analysis in Section 3.3 on Im0geNet100.
PGD
CW 3
APGD-DLR 9
APGD-CE
Max magnitude
0.5
/
0.5
0.5
Steps
5
20
5
Step size
0.25
/
/
0.0
/
/
0.25
/
P
0.75
0.75
BoT
1
Table 5: Hyperparameters different adversarial attacks used in the analysis Section 3.3 on ImageNet100 as well as our ablation studies.
A Establish the correlation between AdaNCA placement and robustness improvement
Our motivation for investigating the relationship between AdaNCA and robustness stems from an empirical observation, namely that making neural networks more monolithic contributes to their robustness (28 One of these monolithic ideas is redundancy networks. It has been observed that several consecutive layers output similar results [4], hence building what we call "a layer set." We hypothesize that AdaNCA, as adaptors inside ViTs, should connect different sets inside ViT and transmit information between them. In this way, layers inside set will no longer bother adapting to other layers outside the set, improving the redundancy and thus robustness. this end, propose the layer redundancy quantification and network redundancy quantification K,, as well as the robustness improvement measurement 7 in Section 3.3
For the robustness improvement, we insert AdaNCA into all possible positions of 3 ViT models, Swin- tiny (38], FAN-small --aasl-llhybrid 778, and -mallsall-pluus [42] and train them along with the baseline models on the ImageNet 100 dataset [5] 72, (74). representative 100-class subset of ImageNetIK. The total amount of models 34, including baselines and AdaNCA models. All AdaNCA- models have a 1 1%-%% parameter increase and 5% more FLOPS. The training hyperparameters are given Table 4 All models are trained for 300 epochs with the Cosine scheduler. Here, we only consider adversarial robustness for simplification,
it is shown that adversarial robustness correlated with image corruption robustness within one backbone architecture [42]. We use a white-box version of AutoAttack [9] in the AdaNCA placement analysis in Section 3.3 which
comprises PGD [4D, CW (5), APGD-DLR 99, and APGD-CE (9) The hyperparameters of the four attacks are in Table
We show the qualitative results our experiments on the Set Cohesion Index and robustness improvement in Figure and the quantitative results in Figure Qualitatively, Y generally follows the trend K. except for the top layers and last layer. Specifically, 7 for the top three layers consistently exhibits pattern where it decreases in the second layer and increases in the third layer, regardless the trend in K, which always increases. We hypothesize that the proximity of the top layers to the input image causes AdaNCA to similarly impact the model's I's robustness by adapting the
17

---

input to the subsequent layers. Furthermore, the position immediately before the final layer likely serves as a transitional stage for output, adapting to different output strategies (e.g.. FAN has an additional class attention head while Swin and RVT use average pooling on all tokens to generate features for classification). The distinct behavior of the last layer compared to other layers has also been noted in previous research [2T] Hence, we exclude the models (AdaNCA applied in the top three layers and before the final layer) from our analysis. The resulting number of models is 19. To conduct cross-model quantitative comparison, we normalize all 7 well as all I. in single type of model. Each of the models thus has set of Tnorm E [0, 1] as well as Knorm E [0. 1]. We plot all sets of 'norm and Knorm from 3 models in Figure 8 Note that the coordinate (1.0,1.0) has two overlapping points (Swin-tiny and FAN-small-hybrrin
hence only 18 points are visible in the figure. Thorm significantly correlated with Knorm (r 0.6938, p 9.8e 4), as report in Section 3.3
A.1 Dynamic programming for AdaNCA placement
Given U AdaNCAs, the VIT is expected be partitioned into U + 1 sets. The dynamic program- ming involves filling an array D, where D[i[[u] represent the maximum value of K. achievable by partitioning the first layers into u sets. The can be obtained via Equation 11
Diilu max jz
1] + k(j, 1))
(11)
The boundary conditions are: 1) D| = <(1, i), which is partitioning the first i layers into set; 2) [u] = 0. By recording the partition position, we can find U partition points for inserting the AdaNCA. The pseudo-code is presented in Algorithm
Algorithm 1 find_ tal___pptimapppaee pppiiaalpartition(SSmiaaa Matrix S, Number of Stages ())
1: n <- length of S
2: Initialize dp with dimensions (n + 1,2f + 1), filled with -00
3: Initialize partition with dimensions (n 1,44 +1), filled with zeros
4: [o[0][0] + 0 [Base )ase)
5: for each i from to n do
6: for each u from to min(i, () do do
7: for each from 0 to
8:
current value + dp j if current value > dp j then
x(ji i 1) +],j
9:
10:
dp[i][u] < current value
partition(]](u <
12: end if
end for
14: end for
15: end for
16: Initialize stages as empty list
17: current_layer + n
18: current stage t 2
19: while current stagettstsgeeeee > 0 do
20: start layer partition(current layer][eurrent, _ttge] + 1
21: Append (start_layer, current_ layer) to stages
22: current ourrent_layer partition (errititnccurrett.cmmm larrrent_lyyrrrcrrrreee stage]
23: current stage <- pourent_stage
24: end while
25: stages stages[::-1]
26: return stages
Note that here the layer index starts from 0, and does the input requirement of
B Notes on statistical significance
We do not report the error bar for training models with different seeds on ImageNet as it would be very expensive (Table [0). However, we follow the seed used in the released code for each model.
18

---

Swin-tiny
FAN-small-hybrid
RVT-small-plus
Layer Radundancy
1.49
0.34
Robustness
5.33
C13
6.3
Layer indices
00
Pair-wise layer similarity
Figure 7: Top: Visualization of layer redundancy (IC(1) = st(1, i+ ++1)++((+)), L)) and corresponding
robustness improvement 7.. Bottom: Visualization of pair-wise layer similarity of the ViTs.
05..698,,p00000
RV RVT-small-po plus
FAN-small-hybrid Swin-tiny
(horm
0.2
K
Figure 8: The relationship between the layer redundancy K. and robustness improvement 7 by inserting AdaNCA in the corresponding position. "norm and Knorm are obtained by normalizing and K. within the results of each model. Note that the coordinate (1.0,1.)) has overlapping points (Swin-tiny and FAN-small-hybrid), Quantitatively, Tnorm is significantly correlated with Kooorm 0.6938, 0.001). The linear fit is shown orange.
19

---

Table 6: AdaNCA settings for each VIT.
AdaNCA positions (after the index)
# Layers
Training steps
Test steps
Stochastic update
RYT-B-AdaNCA
4,8
[2,41,13,5]
3,3
0.1
FAN-B-AdaNCA
0,6,9
[2.2],[3.5]]22.44
2,4.3
Swin-B-AdaNee
24
2,8,22 5,9
[2.2).
2,4,3
0.1
ConViT-B B-AANC
12
[3,5],1244]
3,3
0.1
Table 7: Increasing the number of recurrent steps of AdaNCA during testing can contribute to the model performance.
# Params (M)
FLOPS (G)
Clean Acc. (1)
Attack Failure Rate (1)
Step=4 27.94
4.7
87.18
22.35
Step=5
27.94
4.8
87.22
23.46
We test the adversarial attack using seeds and find the difference between different seeds negligible (standard deviation: PGD 0.08, CW 0.01, APGD-DLR 0.01, APGD-CE 0.02). As all our results adversarial attacks have differences larger than 0.1, they are statistically significant.
For ImageNet100 experiments in the ablation study, we train the model times and find the clean accuracy changes within a small range (standard deviation 0.08), and the robustness test results barely change (standard deviation 0.02). We also test the robustness improvement within one model using seeds and the standard deviation is 0.008.
For the AdaNCA placement analysis, our correlation result is statistically significant (p < 0.001).
C ImageNet1K experiments
C.1 AdaNCA settings for each model
Based [ insert AdaNCA into the four chosen baseline models. The specific settings for each AdaNCA are shown Table 6 Note that insert position means before the network, since FAN-B model uses complex patch embedding layer, semantic information rather than the simple convolution patch embedding used in other models. In practice, find inserting AdaNCA after it can contribute to the model performance. Moreover, add the drop path operation after each AdaNCA, and the drop path rate follows the one of the layer that AdaNCA follows. Hence, the regularization is pretty strong even without the stochastic update. However, as shown in our ablation studies in Section 4.3 the stochastic update indeed contributes to the model performance. Our FLOPS computation for AdaNCA models are based Test steps.
treat it as unique layer that encodes
C.1.1 Discussion the choice of AdaNCA steps
Our choice of AdaNCA steps differs from all previous NCA models, which typically iterate for hundreds of steps. We make this compromise as the computational costs increase linearly as the step grows. We aim to minimize the increase the number of parameters and FLOPS for scalability and we do not want the source of improvement to merely stem from the increase in the size and computation of the models. We show that more NCA steps will indeed contribute to the model's performance in Table 7 while it introduces extra FLOPS. The model is the same as in Section 4.3 which is trained using range of steps being [3, 5). We want to underscore that it the architecture and evolution scheme that defines an NCA as introduced in Section 3.1 instead of the number of its recurrent steps. Admittedly, fewer steps will lead to a coarser path to the target state, and can potentially undermine the model's 's performance. Notably, with our scheme, AdaNCA has achieved generally strong robustness improvements compared
baselines. It indicates that our choice of the recurrent steps of AdaNCA is a good balance between computational costs and performance.
20

---

C.2 Discussion on the model choices
In our experiments, we choose RVT 42 FAN 78 Swin 38 and Con ViT 16].
Our choice not
only includes different types of ViTs (Regular, Hybrid, Hierarchical) and robust as well as non-robust architectures (RVT, FAN and Swin, Con ViT), but also covers other aspects of ViT characteristics. the final token maps to obtain the ID feature for
First, RVT and Swin use average pooling classification, while FAN and Con ViT adopt a separate class token. has been proved that those two strategies of classification can have a significant influence on the model robustness 42) Despite this, AdaNCA are effective in both ways for classification. Moreover, all the architectures already contain certain kinds of local structures. RVT contains depth-wise convolution within MLP and convolutional patch embedding layer. FAN has a ConvNeXt [40] head for patch embedding. Swin has shifted window attention. VIT has cervoolutin-aattentin
coupled gated positional self-attention layers. Local structures have been proven effective in improving model robustness [42] Therefore, partially eliminate the possibility that AdaNCA is effective simply because of introducing local information in ViT's without any local inductive bias, such as the original ViT [4 or DeiT [63].
As shown in our ablation studies in Section 4.3 AdaNCA not only contributes to more parameter- efficient models but also improves the model robustness compared to not using the training strategies or design from NCA. Furthermore, in our choices, Swin does conduct weight exponential moving average (EMA), while the other 3 models perform. It indicates that AdaNCA can be effective with or without model EMA.
C.3 Discussion on the hyperparameters of MLP inside AdaNCA
In AdaNCA, we set the hidden dimensionality of the MLP the same one as the input dimension. is different from the common design choice of MLP in ViT that uses a 4-time larger hidden layer than the input layer. Moreover, it also deviates from the design of the previous NCA that uses more than 8 times larger hidden layer size than the input one. Instead, we set the hidden dimensionality to be the same as the input one. The reason is that AdaNCA introduces additional parameters and computation in ViTs and we want the extra computational overhead to be as low possible. The lowest dimensionality that will not cause inevitable information loss is the same one as the input dimension. Therefore, we design the MLP in AdaNCA as non-compression-onn-eee
structure.
C.4 The effect of AdaNCA placement on robustness on ImageNetIK
Our algorithm for deciding the placement of AdaNCA is based on the correlation between the robustness improvement and the Set Cohesion Index in Section All the experiments are conducted on ImageNet100 100 to efficiently use the computational resources. Here, we compare two schemes for deciding the placement of AdaNCA on ImageNetIK on Swin-B (38) and FAN-B [78] to demonstrate result validity. The first scheme is No-Prior, which does not involve the knowledge of the layer similarity and performs the most reasonable placement choice. For Swin-B, the choice for placing 3 AdaNCA is to insert them between each stage pair defined by the transition between different embedding dimensionalities, namely after layer 2, layer 4, and layer 22. Coincidentally. this choice aligns with what we obtain from the dynamic programming algorithm, which is place AdaNCA after layer 2, layer 8, and layer 22. For FAN-B, although it is hybrid VIT, we only consider inserting AdaNCA into its main network where all layers share the same structure. In this case, the No-Prior chooses to uniformly insert AdaNCA between the 16 layers, that is after layers 5 and 10. Our algorithm decides the placement should instead be after layers and Table & shows the results of two Swin-B- B- AdaNCCA models. Inserting AdaNCA with the No-Prior scheme can still contribute to the model performance and robustness, but not as effective as using our proposed method.
C.5 Details of architecture change in Swin-Base' and Con ViT-Base
For Swin-Base* we add two extra layers in stage 3 with embedding dimensionality being 512, since in the original design most of the computational budgets are dedicated to stage 3, and two layers form a complete Swin operation. For Con ViT-Bass" we add an extra Gated Positional Self Attentonnn (GPSA) layer.
21

---

Table 8: Comparison between different schemes of AdaNCA placement. No-Prior: insert AdaNCA without knowledge of layer similarity structure but making the most reasonable choice, that is to insert them between the architectural stage transition where the embedding dimensionality changes for Hierarchical ViT like Swin [38], or insert them uniformly between layers that have the same structure like the main network in FAN 78]. AdaNCA consistently improves the model performance and robustness compared to the baseline models, while the improvement can be maximized by using our AdaNCA placement scheme based on Algorithm
Model
Insert Schenme
Params (M)
ELOPS
ImageNet Top-1
PGD
CW
APGI-DDER
Robustness Benchmarks APGD-CE
IM-C
IM-A
IM-R
IM-SK
Swin-B 58
87.8
83.4
21.3
13.4
54.2
46.6
32.4
90.7
Swin B-AdaNCA
Ours
90.7
16.3
83.7
24.1
20.5
25.1
24.8
51.5
36.0
48.2
35.5
BAN-B [3]
50.4
11.7
83.9
15.0
7.6
10.4
13.1
46.1
39.6
52.7
40.8
BUN AdaVCA
No-Prier
51.7
12.4
84.1
173
10.2
13.5
16.3
44.7
42.8
53,5
41.0
BUN-B-AdoYCA
Table 9: Hyperparameters used in AdaNCA-equippel ViT training.
RVT-B-AdaNCA FAN-B-AdaNCA Swin-B-AdaNCA ConViT-B-AdANAA
Learning rate
le-3
2e-3
le-3
le-3
Batch size
1024
2048
1024
1024
Model EMA decay
0.99996
0.99992
/
0.99996
Stochastic depth
0.35
0.5
0.1
Random crase prob
0.25
0.3
0,25
0.25
Gradient clip
None
None
5.0
None
MixUp
0.8
0.8
0.8
0.8
Label smoothing
0.1
0.1
0.1
0.1
Min learning rate
le-5
le-6
le-5
le-5
Weight decay
0.05
0.05
0.05
0.05
C.6 Training details
use different training schemes for the four selected baseline models, closely following the parameter settings for each of them (link to reference configuration files: RVT-B FAN-B Swin- BoCTBB) Most training settings are in line with DeiT (63) since all models used build the training upon the scheme of training DeiT but with minor changes. Some specific training settings are listed in T
All models are trained for 300 epochs. Our ablation studies on the size of the model (Swin-Ba BBabl, Con ViT-B-abl) also follow the same training settings. The detailed time consumption for training each model using 4 Nvidia-A100 A100 80G GPUs is in Table TO The excessive time of RVT is because of the pisc augmentation (42
C.7 Adversarial attacks
We test all AdaNCA-equipes setting of each attack is given Table We try setting the Expectation Over Transformation (EOT) to and the results do not change significantly (For Swin-B, APGD-DLR, EOT=1: 25, 124, EOT=3: 1211. Hence, we fix the EOT to I.
ViTs using four adversarial attacks based the code (34). The detailed
C.7.1 Choice of adversarial attacks
We choose PGD [1 since it is the most popular method for examining model adversarial robustness
after architectural changes [58 42). However, is relatively easy to overcome the PGD attack
Table 10: Training time of each AdaNCA-equipped VIT.
RVT-B-AdaNCA
FAN-B-AdaNCA
Swin-B-AdaNCA
ConVit-B-AdaNCA
Training time (hours)
400
180
120
100
22

---

Table 11: Hyperparameters in different adversarial attacks.
PGD
41
CW S
APGD-DLR 9
APGD-CE 9
Max magnitude
1.0
/
1.0
1.0
Steps
5
20
Step size
0.5
/
/
K
0.0
/
0.5
/
p
0.75
0.75
EOT
1
Table 12: Results of ViTs and AdaNCA models under Square
attack on ImageNet100.
Placement (after)
Clean Acc.
Square
Attack Failure Rate
Swin-T 38
86.56
16.18
18.69
Swin-T-ANe T-AANA
4
87.18
31.74
36.41
FAN-S 778
/
87.30
29.44
33.72
FAN-S-AdaNCA
88.22
42.24
47.88
RVT-S 42]
/
87.18
33.14
38.01
RVT-S-AdaNCA
87.50
39.12
44.71
using obfuscated gradients through methods such random inference, noisy architecture, non- differentiable components (3. Our AdaNCA does not fulfill the requirement for producing obfuscated gradients since it is fully differentiable and still want to examine whether recurrence would result in corrupted gradient information due to the drawback of cross entropy loss (9). Morcover, the step size in PGD can largely affect the result. Hence, we choose Auto-PGD family (9] to automatically decide the step size and incorporate the new Difference of Logits Ratio (DLR) loss, resulting in APGD-CE and APGD-DLR, respectively. We also want to include an optimization- based adversarial attack and thus select the CW [5] attack.
off all randomness during test. However,
C.7.2 Black-box attack
We also consider the black-box attack, specifically Square [2). However, due to the extreme com- putational cost of performing Square ImageNet1 K, instead conduct Square attack three models, Swin-tiny (Swin-T) [38), FAN-small-hybree (FAN-S) (78), and RVT-small-phe -plus (RVT-S), on ImageNet100. The three models are used in our AdaNCA analysis in Section 3.3 The maximum magnitude e 6/255 and the number of queries is 1000. Results are shown in Table I2 The AdaNCA placements
line with the highest robustness improvement placements demonstrated
in Figure 7
C.8
The effect of the range of the steps in random step training
Our random stap training strategy contributes to the model robustness as shown Table 3 In all of our experiments,
adopt range of 2 in the random step setting. Here, examine the effect of increasing the range. We use the same setting as described Section 4.3. Results are given in Table [3 We can observe that more choices of the recurrent steps worsen the model performance.
This might stem from too much noise introduced into the training process which leads to underfitting. Therefore, only adopt range of 2 in experiments.
C.9 Reason for using ImageNet22KK model for RVT
We use the LmageNet22K-pretraanee
model for RVT-Base DYT-Base-plus in our main results. This is because we do not find an official release the ImageNet K-trained weights. Moreover, our trainable parameter count on the model in the released code differs from the one reported in the original paper (ours:
88.5M, in-paper: 91.8M) 42 66). Hence, self-training might deviate from the results the official paper. We tried to contact the author for an ImageNet version of the model but did not succeed.
23

---

Table 13: Results of Swin-Tiny and the AdaNCA-cnhanced models with different ranges of random steps on ImageNet100.
Range
Clean Acc.
AutoAttack
Attack Failure Rate
Swin-T 38
86.56
10.64
12.29
[3,5]
87.18
19.52
22.35
Swin-T-AdaNCA
[2,6]
87.02
19.02
21.85
[1,7]
86.84
16.20
18.65
Table 14: Integrating AdaNCA with pre-trained VIT on ImageNetIK. All integration schemes perform worse than training Swin-B-AdaNCA from scratch.
Clean Ace. (1)
IM-A (1)
APGD-DLR (t)
IM-R (1)
IM-SK (1)
Swin-B Swin-B-AdaNCA
83.4
35.8
15.6
46.6
32.4
83.7
36.0
25.1
48.2
35.5
Freeze All
83.1
30.9
11.5
46.2
32.5
Boundary Training
83.1
32.8
19.5
47.1
33.9
Finetune All
83.3
34.9
19.9
46.9
33.7
We test the result of the released model (ImageNet22K version) on PGD attack and get an accuracy of 30.47. In the paper, the authors report this number to be 29.9 the ImageNet1 model. Hence, assume that the results of adversarial attacks can be used as a representative, though might be slightly optimistic. Importantly, this does not falsify our claim that TAPADL modification undermines the adversarial robustness of RVT, since TAPADL-RVT already falls short in PGD attack when compared to the ImageNetIK version of RVT-Base-plus. However, for the O.O.D test, ImageNet22K model can differ lot from the ImageNet1 K models, as most data would be counted as in-distribution w.r. ImageNet22K. Therefore, we use the official values reported in the original paper [42 on those benchmarks. Moreover, for the comparison other than adversarial attacks, we use TAPADL-RVT 23 as a proxy of the RVT model.
C.10 Visualization of attention maps
We aim qualitatively show that AdaNCA helps ViTs perform correct classification when facing noise. Here, we use GradCAM+ 6 to visualize the attention map of Swin-Base [38] and Swin-B- AdaNCA on the clean images well as images containing adversarial noise. We use APGD-DLR to generate the adversarial images. Results are shown in Figure P We can observe that AdaNCA- enhanced Swin model focuses more on the objects, while the baseline model attends areas unrelated to the object on the images when facing adversarial noise.
C.11 Integrating AdaNCA into pre-trained VIT models
In our experiments, we train all models from scratch. Implementing AdaNCA as plug-and-play module for pre-trained ViT models would certainly improve the training speed. To explore this, we experiment by inserting AdaNCA into pre-trained Swin-base model on ImageNetIK by 1) freezing all VIT layers; 2) training only the boundary layers, where boundary layers are the layers before and after the insert position of AdaNCA; 3) Finetuning all layers. Results are given in Table [4] None of the schemes perform as well as training from scratch. The results indicate that the current NCA is able to adapt to such a scheme. However, it may struggle to effectively transmit information between two pre-trained VIT layers, as these layers have already established strong connections. In contrast, training the model from scratch allows NCA and ViT to synergistically adapt to feature variability, resulting in better overall performance. However, worth exploring the future since the fine-tuning scheme can contribute to the performance.

---

Swin-Base
Swin-B-AdaNCA
Clean
Adversarial
Clean
Adversarial
Salamander
Goldfinch
Dipper
Figure Visualization of attention maps Swin-Base [88 and Swin-B-AdaNCA using GradCAM++ [6] clean images and images with adversarial noise. AdaNCA helps ViTs focus more the object when facing noise.
Table 15: Comparison with VITCA. ViTCA performs worse in clean accuracy and is on-par with AdaNCA in robustness. Note that ViTCA has more parameters than our scheme Dynamic Interaction. The training setting the same as in Section in the main paper.
Params (M)
FLOPS (G)
Clean Acc.(1) Attack Failure Rate (1)
Dynamic Interaction
27.94
4.7
87.18
22.35
ViTCA, scale=1
28.48
4.8
86.78
19.47
VITCA, scale=2
29.08
4.9
86.72
22.28
C.12
Additional results on comparison with current methods
We have presented the comparison between our AdaNCA and the current SOTA method, TAPADL models 223, Section 4.1 Here, we provide an additional comparison with the ViTCA model 62] In ViTCA, tokens interact with each other through local self-attention. We aim compare our proposed Dynamic Interaction module with the interaction learning scheme ViTCA. Results are given in Table 15 Despite having more parameters, the ViTCA-like scheme performs worse in clean accuracy and is on-par with AdaNCA robustness. This indicates that is promising to explore the possibility of incorporating ViTCA into framework.
C.13 Additional results on same model architecture but with more layers (x models)
we not train them with more layers and thus do not have a version of these two models. Instead, we train the smaller version of models on Im0geNet100. We choose RVT-small-plus (RVT-S) and FAN-small-hybrrd (FAN-S) two baselines, which is in line with our other experiments on ImageNet100. We add one extra
Due to the computational cost RYT 42 and FAN 78 (Table 10
25

---

Table 16: Results of different VITs on ImageNet100. * means the same model design but with more layers. The improvement AdaNCA brings does not merely stem from the increase in the number of parameters or FLOPS.
Params (M)
FLOPS (G)
Placement (after)
Clean Acc.
Acc. AutoAttack
Attack Failure Rate
FAN-S
78
25.8
6.7
87.30
18.70
21.42
FAN-S 78
27.7
7.1
/
87.4
19.40
22.88
FAN-S-AdaNCA
26.1
5
88.22
27.76
31.47
RVT-S 42
23.0
/ 87.18
29.80
34.18
RVT-S' 42
24.8
5.1
/ 87.52
32.14
36.72
RVT-S-AdUNCA
23.3
4.9
9
87.50
36,90
42.17
Table 17: Comparisons of corruption enror on each corruption type of ImageNet-C. AdaNCA consistently improves the performance of different ViT models.
Weather
Digital
Impulse
Defoces
Bright
Elastic
Swin-B Swin-B-AMuNCA
45.7
43.8
55.5
47.4
68.8
62.6
34.5
TAPADI RVT
34.6
36.1
34.0
52.3
44.0
57.1
RVTEB-AdaNCA
339
33.9
60.5
Con'ii [6]
46.9
36.0
37.6
35.5
56.5
64.4
51.5
65.0
42.6
40.5
28.0
61.7
45.6
518
Con CA
44.3
34.7
38,9
34.2
531
60.4
40.5
37.2
333
87.1
42.4
49.4
DAN [8]
36.5
36.8
34.7
528
65.9
47.7
56.9
00.6
43.4
34.3
62.8
50.6
FAN-BAGINCA
44.7
353
36,0
33.7
$1.9
45.3
$5.6
39.7
43.3
37.0
331
62.0
47.9
layer to RVT-small-plus and one extra layer to the FAN network in FAN-small-hybrid, resulting in RVT-S' and FAN-S*. We compare it with our AdaNCA-equipped model where AdaNCA is after layer in RVT-S and after layer 5 in the FAN-S, aligning with Table T2 We use the same AutoAttack as described in Section measure the robustness improvement and attack failure rate. Results are shown in Table [6 We can observe that the increase in the number of parameters and FLOPS does not account for the improvement in robustness.
C.14 Additional results on layer similarity structure
Here, we show the layer similarity structure for FAN-B-Hybre 78 and ConViT-B 16 with the Image weights and the weights after training with AdaNCA. For RVT-Base-plus we train it with the released code [66] but with a different number of parameters, as discussed in Section C.9 This training is only for obtaining the layer similarity structure, and we find it stable after 30 epochs. The results are shown Figure 10 practice, we ignore the stage partition found by the dynamic programming algorithm in Section A. that is after the first layer or before the last layer since want a stage to contain more than layer. The results further validate our assumption that AdaNCA is used and more different from each other. Interestingly, not all AdaNCA contributes to drastic clearer stage partitions. For example, the first AdaNCA in FAN-B is not effective in making the stage clear as the second one. However, the overall Set Cohesion Index increases, and the layer similarity structures in all 4 networks do not change. We believe the deep learning architecture research our results, in that developing more architectural changes which contribute to stage partition in the network and examine the effect on network robustness and generalization. More importantly, use a single non- parameerii metric for quantifying the layer similarities. Introducing advanced output similarity quantification methods might lead new findings and other fascinating results.
adaptor between stages and transmits information between them, making them more
benefit from
C.15 Additional results on category-wise mCE on ImageNet-C
We provide additional results on category-wise mCE on ImageNet-C in Table 17
Our test follows
the data transformation used in 42
C.16
Additional results on noise sensitivity examination
We use the data from 60) the experiment, human subjects are asked to classify the noise- contaminated images, and the classification accuracy is recorded. Then, the images are fed into the
26

---

Swin-B, B,
=0.466
Swinn--AAdaCA, Kmoon 00.509
FAN-B, =00055I
FAN-B-AdonCA, Ameen 00.444
15
RVT-B Kynen RVT-B-AdanCA, Kmeun *0.129
ConVIT-B Kmean =0.072
ConViT-B-AdonCA Kmoun =0.103
Figure 10: Layer similarity structures of different VITs. Layer set marked in red boxes. AdaNCA is inserted between each pair of stages, Kynea is the mean of x of all stages marked out.
TAPADL-RVT
RVT-B-AdaNCA
Human Classification Accuracy
Magnitude of Noise
0.02
60
0.04
FAN-B-Hybrid
FAN-B -AN-B-A
0.18
1.25
2B
96
112
Frequency of Noise (cycles/image)
Triquency of latte
Freguenty Mave
Figure 11: Noise sensitivity examination on humans and ViTs. AdaNCA contributes to more human-
like noise-resilient ability and reduces the sensitivity of ViTs certain types of noise (yellow boxes).
models trained on ImageNetIK, K, whose outputs are then transformed into 16-way ImageNet label [56 and compared against the ground truth labels. Their classification accuracy is also recorded and used for comparison with human results. We present the additional results on MAN-B-Hybrig [78
and the SOTA model TAPADL-RVI 23 compared the AdaNCA--quipper model in Figure
Moreover, we adopt quantitative metric on the test. In the original experiment [60] Gaussian curve fit based the discretized accuracy map. However, such method ignores the continuous change the classification pattern. Hence, we propose a simple metric that examines the similarity between the accuracy map of humans and of models. Specifically, given the ground truth accuracy map (human performance) Agt, and model accuracy map Am. the similarity is defined as:
1 Σ (m, Agrle N
T= 100
(12)
c,f
6, represent the magnitude and frequency of the noise, respectively. higher indicates that model can achieve more similar accuracy maps with humans averaged across all noisy images. N is the total number of types of noise added. Since all models as well as humans perform similarly on low moggiagi noise, we ignore the first two noise levels 0, 0.02). Hence, in our experiments,
27

---

well as the Table 18: Quantitative results on the frequency preference examination experiments. We report the similarity of accuracy maps between model results and humans.
Accuracy map similarity to ground truth r
ResNet50 27
72.23
ResNet50-Ady 66
52.12
TAPADL-RVT
23
81.91
RVT-B-AdaNCA
82.17
FAN-B-Hybrid [78
83.52
FAN-B-AdaNCA
84.75
Swin-B 38
77.07
81.34
ConViT-B [66
79.69
ConViT-B-AdaNCA
81.56
N 21. We report in Table 18
We also include the results of ResNet50 [27]
(ResNet50-Adv, trained with L2-bounded adversarial noise where the maximum magnitude of noise is 0.25), and present the visualizations of the accuracy maps of those two models in Figure. Critically, our results indicate that the adversarially trained model cannot lead improved accuracy similarities, resulting in less human- like noise-resilient ability. This is in line with the conclusion of the original work [60], validating our proposed metric. Contrary to adversarial training, we obtain more numan-like decision patterns than the compared models, validating our claim in Section 4.4
12-adversarialll-traa -rrinne version [60]
C.17 Datasets information and license
[2] This dataset contains 1.28M training images and 50000 images for validation. We report the topl accuracy on the 50000 validation images. License: Custom (research, non commercial)
ImageNetIK
ImageNet-C (28]. This dataset contains 15 types of 2D image corruption types that are generated by different algorithms. The metric on this dataset mean corruption error, whose lower value represents a more robust model against those corrupted images. License: CC BY 4.0.
ImageNet-A 13.. This dataset contains naturally existing adversarial examples that can drastically decrease the accuracy of ImageNetl K-trained CNNs. It is a 200-class subset of the ImageNet1 K dataset. License: MIT license.
ImageNet-R 330 This dataset contains different artistic renditions of 200 classes from the original ImageNet object classes. The original ImageNet dataset discourages non-real-world images, and thus the artistic renditions render the images to be O.O.D. License: MIT license.
ImageNet-SK 667] This dataset contains 50000 images with 1000 classes that match the validation set of the original ImageNet dataset. All the images black-and-white sketches instead of real-world photographs the object class. License: MIT license.
C.18 Model and code license
Swin-B [3]
MIT License.
FAN-B Nvidia Source Code License-NC.
ConViT-B [6]
Apache 2.0 License.
RVT 42 Apache 2.0 License.
TAPADI 23
MIT License.
Code for adversarial attacks 34 MIT License.
PyTorch Image Model 69 Apache 2.0 License.
Code for GradCAM+
19 MIT License.
28

---

Table 19: Stochasticity during testing will give false sense of adversarial robustness, We conduct the same experiments twice, which result in Trials and 2. The training setting the same in Section 4.3 in the main paper.
Clean Acc. (1)
CW Acc. (t)
Test-time Synchronous
87.18
9.48
Test-time Stochasticity, Trial
87.10
10.72
Test-time Stochasticity, Trial
87.06
10.60
Table 20: Removing in Equation 7 during training and directly testing with synchronized update. Our dropout-like compensation scheme improves AdaNCA's performance.
p
Clean Acc. (1)
Attack Failure Rate (1)
Original No
87.18
22.35
86.84
21.94
D Extended ablation studies
In this section, we discuss the design of our AdaNCA and provide additional ablation studies on the designs of AdaNCA.
D.1 The dropout-like strategy
In Section 3.1.1, we introduce our approach of using the dropout-like strategy to perform the stochastic update during training and testing. While previous NCA works claim that the stochasticity be preserved [44 46] or can be simply switched off [62] during testing, the proposed dropout-like scheme necessary our case. First, We highlight that stochasticity during testing can hinder the evaluation of adversarial robustness by producing obfuscated gradients [3], leading to the circumvention of adversarial attacks. Table 19 illustrates this issue, where we the classification accuracy under CW attack. Stochasticity also results in inconsistent outputs. This is problematic for practical image classification tasks where reliable output is critical, unlike applications focusing on visual effects or collective behaviors (55). The change in clean accuracy in Table
that given the same image, the stochasticity can lead to different decisions, hindering the deployment of the trained models in real-world scenarios. To further demonstrate the necessity of our scheme, we remove the compensation of the output magnitude during training, i.e., we remove the in Equation 7 during training and directly test the model with synchronized update. The results are given Table 20 The drop in clean accuracy and robustness suggests that downstream VIT layers struggle with varying NCA output magnitudes, indicating the usefulness our proposed dropout-like method.
46 62
D.2 Dynamic Interaction
Our motivation for developing the Dynamic Interaction module is the high dimensionality of feature vectors in modern ViT models. We underscore that any operations involving linear transformations of the concatenated interaction results will lead drastic increases in computational costs, as shown in Table 2T Note that the concatenation scheme nearly doubles the FLOPS for RVT compared the baseline, leading to difficulties in training. Such an increase renders scalability a challenge. Admittedly, more parameters can contribute to better performance, as shown in Table 22 Note, however, that our Dynamic Interaction achieves 70% of the improvements of the original NCA concatenation scheme (10.06/14.6) 62) in robustness improvement and 60% improvement in the clean accuracy (0.62/1.06) with merely 10% of the parameters and FLOPS (0.35/2.99 for # Params and 0.2/2.3 for FLOPS). Therefore, our Dynamic Interaction scheme provides a good trade-off between performance and computational costs. It is capable of scaling up, allowing us to insert AdaNCA into even larger models, such as current vision-language models where the token dimensionality even higher. Moreover, we qualitatively showcase in Figure that Dynamic Interaction helps robustify AdaNCA when facing noisy inputs.
29

---

Table 21: The concatenation scheme of the original NCA results in large increase in the computa- tional costs when the token dimensionality is high, leaving scaling up to large datasets and model sizes a challenge.
# Params (M)
FLOPS (G)
Swin-B
87.8
15.4
Swin-B-AdaNCA
90.7
16.3
Swin-B-AdaNCA, Concat
115.5
24.7
RVT-B
88.5
17.7
RVT-B-AduNCA
91.0
19.0
RYT-B-AdaNCA, Concat
112.3
34.0
Table 22: Performance of using our Dynamic Interaction and the concatenation scheme of original NCA ImageNer100. Numbers in parentheses indicate performance improvement compared the Baseline. Our method achieves a large portion of the performance improvement (60% 70%) brought by the concatenation scheme with much fewer computational costs (10%).
Params (M)
FLOPS (G)
Clean Acc.(t) Attack Failure Rate (1)
Baseline
27.59
4.5
86.56
12.29
Dynamic Interaction
27.94 (+0.35)
4.7 (+0.2)
87.18 (+0.62)
22.35 (+10.06)
Concat
30.58 (+2.99)
6.8 (+2.3)
87.62 (+1.06)
26.91 (+14.62)
Without Dynamic Interaction
Clean
Adv
With Dynamic Interaction
Clean
Adv
Time
Figure 12: Visualization the developing token maps of AdaNCA using PCA. Without Dynamic Interaction, AdaNCA cannot recover from the noisy adversarial inputs. Equipped with it, the model can stick to the evolution path similar to the clean inputs.
30

---

Table 23: Overly large neighborhood size during token interaction will undermine the model per- formance as complicates the process of figuring out which neighbors to interact with and adds too much noise during the process. While increasing the model capacity can alleviate such an issue, AdaNCA still struggles to handle too-noisy information when the scale is too high. Note that the larger filter scheme achieves on-par results with our method when the scale is 2, indicating that it does not provide additional useful information for token interaction at this scale.
Params (M)
FLOPS (G)
Clean Acc.(1)
Attack Failure Rate (1)
Original, scale=3
27.94
4.7
86.60
20 44
Larger filter,scale=3
28.04
4.7
86.74
21.13
Larger Wy receptive 3ield,scale=3
27.98
4.7
86.66
20.32
Original, scale=2 Larger filter,scal=22
27.94
4.7
87.18
22.35
27.96
4.7
87.14
22.31
D.3 Multi-scale Dynamic Interaction
Our motivation for developing multi-scale interaction is perform more efficient token interaction learning over local scales since ViT already contains global information. Enlarging the neighborhood size will: 1) Complicate the process of selecting the neighbors to interact with; 2) Introduce excessive noise, making it difficult for tokens to accurately acquire neighbor information. 3) Repeatedly acquire the global information provided by ViT. The recurrence further amplifies the noise. In our ablation study in Section 4.3 we notice that increasing the number of scales to 3 will undermine the performance. Such an issue is also observed in previous work 62 where local self- attenttnnn is used for interaction learning. While in theory both Dynamic Interaction and self-attention can discount far-away information as the tokens can decide their own interaction neighborhood, overly large neighborhood size can lead to the problems mentioned above. Note that our model gains performance when S (5 x neighborhood), indicating the usefulness of our multi-scale module. We further explore whether the multi-scale issue can be solved by increasing the model capacity. Specifically, we develop two additional schemes. The first is to replace the dilation in Dynamic Interaction by simply using larger filters. The other one enlarges the receptive field of the weight computation network War in Equation 9 Instead of using two 3 x 3 convolution layers, we change the first layer to be a x convolution. As a result, the receptive field of War becomes 7 x 7, matching the one when S = 3. We use the same training setting as in Section 4.3 Results are given in Table Although increasing model capacity might alleviate the noise issues, AdaNCA struggles with overly large neighborhoods.
E Differences between traditional NCA and AdaNCA
While AdaNCA is inspired by NCA, it does not fully exploit the designs of traditional NCA. We discuss three key differences between AdaNCA and ÑCA below:
1. Different initial states. NCA in image generation starts from either constant or randomly initialized cell states, typically referred to as the seed states, while AdaNCA receives the outputs from previous ViT layers. Hence, AdaNCA handles structured inputs at the very beginning.
2. Different usage of cell states. Traditional NCA does not use all cell states for accomplishing the downstream tasks. After certain steps of evolution, subset of the cell states is extracted to perform the given task. The unused cell states, termed hidden states, facilitate cell commu- nications they can be used to store additional information 44 The dimensionality of the hidden states is typically several times that of the input. When the cell dimensionality is high, is the case in ViT, adding too many hidden states will bring too much computational costs. Moreover, cells can store effective enough information when they have high dimensionality. Therefore, we do not adopt the hidden states design in AdaNCA.
3. No pooling strategy. A critical component of traditional NCA is the pooling strategy. Instead of starting from the seed states in every epoch, NCA fetches the starting states from a pool, where the output states from previous epochs are stored. Through this strategy, NCA can explore much longer time steps without suffering from gradients or memory issues. While is tempting incorporate the pooling trick into our method, two major concerns hinder its
31

---

practical implementation: 1) The previous NCA models start from their own outputs in the pooling strategy. In other words, the cell states in the pool are generated by the model itself. However, AdaNCA starts from the outputs of ViT layer. Such a difference renders the pool trick hard to implement. 2) Taking step back, even if there is a well-designed pooling strategy for AdaNCA, it will bring too much computational costs during testing. A single step of AdaNCA introduces non-trivial FLOPS during testing, as shown in Table [T] The ultimate goal of the pooling strategy is to ensure the stability of NCA in large time steps, while large time steps will slow down the inference.
32

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Our claim on AdaNCA improving ViT's robustness is validated in Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitation of our work discussed in Section 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
33

---

Justification:
We do not have theory assumptions and proofs in the paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We disclose all architecture and training details in Section 3.2
and C.6
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
34

---

Answer:
[No]
Justification:
We are preparing the code release and will publish the code soon.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
il for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ccpppp.ccppulli//uuuii wwwpc//ggue////eeeeeoooc CoodeSubeiiieeee ssi asionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We disclose all training details in Section and C.6
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Please see Section B in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
35

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the hardware and time consumption for our experiments in Sec- tion Con[C.6fn in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: research.
We have read the NeurlPS Code of Ethics and strictly follow it during the
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., s if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please see Section G
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
36

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper does not pose such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Please Section C.17 and C.18 in the Appendix.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
37

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: We do not include code or model release in the paper, but we are preparing it for full publication.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not have crowdsourcing and do not conduct research with human subjects.
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector:
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not have crowdsourcing and do conduct research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions NeurIPS Code of Ethics and
and locations, and we expect authors to adhere guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
38


Reviews:
[{"review_id": "i0dzOe9gbg", "reviewer": "Reviewer_h6zC", "summary": "This paper proposes a strategy for improving the image classification robustness of Vision Transformers (ViT) through the use of specialized networks that are inserted at strategically placed layers within the ViT model. These networks are called Adapter Neural Cellular Automata (AdaNCA) and are intentionally chosen due to the proven robustness characteristics of NCA on various tasks such as image generation and classification. Although there has already been a connection made between NCA and ViT models (ViTCA from Tesfaldet et al.), AdaNCA differentiates itself by not trying to be a ViT in and of itself (as ViTCA does), but by acting as an adapter that can be placed at various layers within a ViT to improve its robustness. Through exhaustive experimentation, the authors prove the viability of using NCA in a much larger scale setting than ever before (as far as I know) through this adapter-type approach, thus providing a new pathway for the NCA community to consider when it comes to practical applications. Another difference between ViTCA and AdaNCA worth mentioning is the downstream task at hand: small-scale image denoising vs. large(r)-scale image classification. \n\nA quick summary of what NCA are: they're a fairly recent (circa 2019-20) computational paradigm that build upon the much older model of Cellular Automata (CA). Both NCA and CA consist of a connected lattice of stateful cells whose states are recurrently updated through the repeated application of an update rule. The update rule consists of two stages: an interaction stage, where for each cell, information is gathered from its neighbouring cells; and an update stage, where this information is processed to produce a cell update, which can be applied in a residual manner or be directly treated as the new cell state. CA use a handcrafted update rule, with a popular one being Conway's Game of Life, while NCA use a learned update rule in the form of a neural net with convolutions in the interaction stage and an MLP in the update stage. The NCA update rule is trained via some downstream task, where the cell grid / lattice is evaluated against some target state after a certain number of cell updates. Due to the way NCA are trained (stochastic application of cell updates, repeated evaluation against target state at various points of cell lifetime via pool-based training, hidden states to facilitate cell message passing, etc.), they end up being fairly robust models, able to correct themselves in the presence of adversarial attacks (e.g., various types of structured or unstructured noise) and adapt to OOD situations.\n\nThe three main contributions of this paper are as follows:\n\n1. Presenting AdaNCA, a NCA-based small model that is able to be inserted at various points within a ViT to improve its robustness on image classification. On average, the classification accuracy gains on clean, noisy, and OOD data outweigh the added parameter and computational cost in a non-negligible (statistically significant) manner.\n2. Introducing a new type of cell interaction strategy called \"Dynamic Interaction\" (and a Multi-Scale variant) that uses far fewer parameters and FLOPS when compared to a vanilla concatenation approach (typically used by previous NCA), while being more parameter efficient than a simpler summation-based approach. This is the part where you combine the information from the various filter responses in the interaction stage before providing it to the MLP that performs the update stage.\n3. Proposing an effective strategy for picking the best ViT layers for inserting AdaNCA. This strategy is backed by an exhaustive analysis that proves the strategy's effectiveness in maximizing the robustness gains of inserting AdaNCA within a ViT. In short, they show that AdaNCA is best inserted between sets of layers where each set consists of highly redundant layers. Basically, AdaNCA helps information flow between two sets of layers where the two sets are not redundant from one another.\n\nThere are some smaller things I'm leaving out of this summary due to how extensive the paper is in some areas, but this is the overall approach.", "questions": "* L40-41: This statement is missing one crucial piece. It's not only the stochasticity and modulation of local information that make NCA robust against noisy input, it's also the recurrent steps within a single training step that makes them robust (coupled with the pool-based training that extends a cell grid's lifetime to many training steps). The recurrent steps allow the cells to explore a wide variety of states, many of them perturbed by the model itself (especially early on during training when the model is fairly untrained). Tab 3 in your paper also proves this, with the ablation of \"Recur\" causing the biggest drop in attack failure rate compared to the other ablations. So my suggestion is to state the importance of recurrent updates.  As a bonus, here's a paper I suggest taking a look at [1]. You'll be surprised to see the similarities between training an NCA (Mordvintsev-style) and training an Energy-based Model (EBM) using the technique in [1]. Just like the pool-based training technique that Mordvintsev et al. used, they use a sample replay buffer here. It shows how crucial the exploration and recurrence are for having a robust model.  [1] Du, Y., Mordatch, I. Implicit Generation and Generalization in Energy-Based Models. In NeurIPS 2019.\n* For the introduction (L26) and related works (L101-103, L89), I'd like to see a mention of how ViTCA (from Tesfaldet et al.) improves the robustness of ViT architectures. I understand that a brief mention of ViTCA was given in L111-113 but nothing about how it achieved a more robust ViT model (albeit with its own unique limitations). On that note, I find it a bit surprising that nothing is said about ViTCA under the context of ViT and robustness and how that compares with AdaNCA and its approach to ViT robustness. Both contribute to a more robust ViT but in different ways. I'd like to see a deeper comparison made between ViTCA and AdaNCA as well as an acknowledgement of ViTCA already having improved ViT robustness using an NCA (albeit with a different downstream task in mind).\n* L107: You forgot to cite Gilpin [2]! If I recall correctly, he was the first to realize CA under conv nets. I may be wrong, but either way, I think it's necessary to cite and comment on his work if you're bringing up Mordvintsev et al ([43] in the paper).  [2] Gilpin, W. Cellular Automata as Convolutional Neural Networks. In Physical Review E 2019.\n* Eq 3: Please put a quick explanation of the depthwise convolution operator as you did with the channel-wise concat operator in L142. Something like \"where (*) is the depthwise convolution operation.\" It's just to cover your bases in terms of equation clarity.\n* Eq 4 and L142-143: I would suggest clarifying how S_out is used to update cells. \"where cells get the updated states\" is not clear. Perhaps you may want to show that S_out can be used as a residual to update cell state S, or that S_out is itself the new cell state. Either way, it's important to give this extra bit of clarity to the reader.\n* The Neural Cellular Automata paragraph in Section 3: This section is great for the most part, but I would definitely like to see an explanation of how the NCA update rule is typically trained. So far, only the rough architectural design and computations have been explained, but not how to train them, which is incredibly important. For example, perhaps you can mention at the end that after a certain number of recurrent steps (within a train step), a subset of each cell's state may be evaluated against some downstream task's loss function, or something to that effect. You can also make it clear that one may not necessarily need to use all parts of a cell's state, briefly mentioning the hidden channels that Mordvintsev et al (and many other NCA) use, which has been shown to be extremely helpful in facilitating cell communication.\n* Related to the above point, I would recommend clarifying some crucial differences between AdaNCA and other \"more traditional\" NCA that more closely follow Mordvintsev et al's approach. In other words, I would suggest talking about why there are no hidden channels in AdaNCA, what the initial cell states are during training (i.e., Mordvintsev et al. and other NCA approaches start with either a constant or randomly initialized cell grid, whereas here with AdaNCA it seems like the initial cell grid state is just the activations from the layer it's inserted after—correct me if I'm wrong), and why there's seemingly no pool-based training.\n* L158-162: \"In previous NCA works, stochasticity is maintained during testing.\" ViTCA showed that this was not necessary and that it was possible to train with stochastic updates and test with synchronous updates. I would recommend mentioning this and contrasting it with how you've approached test-time synchronous updates. In particular, I'd mention how ViTCA does not use the dropout-style approach and merely switches to synchronous with no compensation.\n    * On this note, I would like to see an ablation between using the dropout-style compensation vs not using it. Can you quickly run an experiment to compare the two settings? Basically, remove the 1/p in \"Train\" in Eq 7 and see how it compares with having 1/p. I'm very curious to see how useful this is.\n    * L159-160 states \"Such a scheme is problematic in discriminative tasks since reliable numerical outputs are essential.\" Isn't the whole point of training the NCA update rule to be agnostic to asynchronous cell updates is so that they can be robust to such a scenario during testing? In other words, it's been shown that one can achieve more or less the same results in an asynchronous testing setting by merely updating the cells for more iterations to compensate for those that are behind. The texture synthesis work from Niklasson et al. (Self-Organizing Textures, citation [45] in the paper), showed this in their time synchronization experiment. ViTCA from Tesfaldet et al. showed this in their ablation study, where the results were more or less the same at 50% update rate compared to 100% if you doubled the number of cell updates. Self-Classifying MNIST Digits from Randazzo et al. also showed that stochastic updates are no problem when it comes to reliable discriminative outputs. So I'm wondering why you say stochasticity during testing is problematic for discriminative tasks when other works have shown that not to be the case.\n    * L165-166: \"Hence, the trained model can effectively handle the variability and unpredictability of the input, thus being robust against noisy input.\" I would suggest rewording this sentence. Specifically, I would highlight that it's not just the randomness of when cells get updated that contributes to their robustness, but also that the recurrent updates coupled with the stochastic masking allows the update rule to explore many cell states, teaching it how to recover from such states and move cells towards a good target state. It's also important to note the update rule's own influence on the perturbation of cells early in training, where the update rule has not been sufficiently trained and so acts as a source of noise to itself. So in essence, you're also training the update rule to be robust against perturbations caused by itself.\n* Fig 2 & 3: I would suggest either removing (b) from Fig 2 and have Fig 3 explain the architectural details, or remote Fig 3 and provide the architectural details in Fig 2. You'll save a lot of space as there's a lot of redundant information between the two figures. Also, I would put a more detailed explanation of the architecture rather than \"Overview of AdaNCA architecture\". That means explaining the computation pipeline and the different computational blocks so that the reader can easily refer to the caption to explain anything they don't understand in the figure. This will mean some repeat of what was said in the main manuscript, but it will immensely help with clarity when someone wants to quickly glean your paper and understand what's going on. A good rule of thumb is to have the gist of the paper be understood from reading the figures and captions alone. \n* Eq 8: Please put a quick explanation of the convolution operator.\n* For Multi-scale Dynamic Interaction, is W_I shared across scales? I'm assuming it is, but just wanted a quick confirmation if it is or not.\n* For Multi-scale Dynamic Interaction (and the single-scale version), I would like to see a comparison with a vanilla concatenation approach. I want to know if the cost of more parameters and FLOPS is worth it with the concatenation approach. Otherwise it's difficult to judge the usefulness of Dynamic Interaction. Comparing it with a simple filter response summation scheme and showing improvements doesn't really motivate Dynamic Interaction in the context of typical NCA approaches. If you can show that Dynamic Interaction (and Multi-Scale) is more computationally efficient than the concatenation scheme (the relative reduction of accuracy is made up for by a greater relative reduction of parameters and FLOPS), then it solidifies the benefits of Dynamic Interaction and possibly sets a new standard for the NCA interaction stage.\n* For Multi-scale Dynamic Interaction, have you tried comparing the weighted summation from W_I and W_Ms with a 1x1 conv approach to mix the depthwise info (basically this is how ConvNeXT does it, depthwise conv followed by pointwise conv)? Have you also tried comparing Dynamic Interaction with a non-depthwise spatial conv? If not, can you provide some quick commentary on what you believe the pros and cons would be for each approach and how it would compare with Dynamic Interaction?\n* For Multi-scale Dynamic Interaction, have you tried using a larger filter instead of a larger filter dilation? I have a feeling that the sparsity induced by the dilation coupled with the sparsity of cell state updates contributed to the model having difficulty with three scales.\n* For Multi-scale Dynamic Interaction, have you tried modifying the two-layer convnet that produces W_Ms such that its receptive field covers the same receptive field of the max dilation? In other words, if there are three scales, that means you would need the two-layer convnet to have a receptive field of at least 7x7, but the problem is that its receptive field is 5x5. Can you verify if it's required to match the two-layer convnet's receptive field with the max dilation in order to see an improvement with three or more scales?\n    * This could explain why it worked with two scales, as the max receptive field would be 5x5, which is within the receptive field of the two-layer convnet.\n* L224: I suggest explaining the \"K(r = 0.6938, p < 0.001)\" part.\n* Jointly training the ViT model and AdaNCA seems costly. Have you tried a training approach where you use a pretrained ViT model and have every layer but the one before and after AdaNCA insertion points frozen? Considering your layer set redundancy experiments, it seems like these \"boundary\" layers would be the only ones required to adapt to an AdaNCA that's also being trained. If this works, then this would really push the adapter narrative of AdaNCA.\n    * Another experiment on top of this would be to have all ViT layers frozen and train AdaNCA to act as a feature cleaner where OOD and noisy inputs have their features adapted to the part of the feature manifold that each layer set expects.\n* Table 1: I suggest providing citations beside each of the metrics under \"Adversarial Inputs\" and \"OOD Inputs\". You can simply reuse the ones provided in 4.1.\n* Table 1: I suggest bolding the lowest param count and lowest FLOPS within each row triplet.\n* Table 1: I suggest changing the bolding of your model to italicizing or some other visual indicator so as to not confuse the reader with the intention of the bolded numbers.\n* Table 1: I suggest explaining what the bold means in the caption.\n* Table 2: Same recommendations as the three above.\n* Table 3: I suggest explaining the bolding in the caption.\n* Table 3: I suggest removing the bolding from \"Ours.\" It's already obvious that it's your model.\n* Figure 6: I suggest not using yellow as the outline colour as it's very difficult to tell amongst all the other yellow-ish colours. Use a strong dotted black or some other colour-blind-safe colour.\n* Figure 6: I suggest clarifying what \"Frequency of Noise\" and \"Magnitude of Noise\" means in the caption.\n* Figure 6: I suggest specifying what kind of noise is being shown here.\n* Section 6: Negative impacts were not considered. Please provide brief commentary on potential negative impacts, even if you believe there are none. It's important to show that you've at least considered it.\n* Figure 4: I suggest briefly commenting in the caption on how AdaNCA does not negatively affect the layer-wise similarity in Swin-B. In other words, providing a smaller version of what was said in 4.2 about how AdaNCA preserves the original layer sim structure.\n* Figure 5: I suggest providing some brief commentary on (a) and (b) in the caption.\n* L288: \"while overly large scale can lose local information.\" How so? Theoretically, the model should learn to discount larger scales if they're not useful, so why does including more scales hurt performance and how would it cause a loss in local information? To me, this suggests that W_Ms isn't working the way it's intended to, so there must be some sort of bug or something else entirely. It's just really odd to me that merely adding a third scale is worse than keeping it at a single scale. I'm keen to hear your thoughts on this and if you've looked into this any deeper.\n* L300: I'd like to see commentary on each of the ablations. \"Our design choices contribute to model performance and robustness\" is not enough. There should be deeper commentary on each of the ablations, explaining any gains or loss in performance from each (e.g., why does ablating RandS and Recur lead to the highest accuracy?), and comparing relative gains between each (e.g., which component contributes the most to model performance?).\n* L298: \"Ablate the Dynamic Interaction that the...\" -> \"Ablate the Dynamic Interaction so that the...\"\n* L271: \"...networks to conduct the analysis.\" -> \"...networks to conduct this analysis.\"\n* \"FLOPs\" -> \"FLOPS\".\n\nI would like to see my questions and suggestions above addressed to the best of the authors' abilities. My largest concerns lie with the lack of a clear and detailed comparison with the only other NCA model that operates under a ViT setting (ViTCA from Tesfaldet et al.), the relatively insufficient evidence of the usefulness of Dynamic Interaction and its multi-scale counterpart, and the lack of commentary on the ablation study.", "limitations": "The authors have addressed most of the limitations of their approach, as shown in the Limitations section (Section 5) of the main manuscript and across various points of the main manuscript and appendix. I would like to point out that the potential negative impacts of their approach have not been considered, and so I recommend they comment on this in Section 6 (Broader Impact), even if there are none.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Originality:\n    * The authors embark on the challenging task of proving the feasibility of NCA to a crowd where many don't believe in their usefulness and where many more have not even heard of NCA. Instead of pushing tired areas of machine learning research, these authors have taken a risk in pursuing a niche and not-yet-proven area of machine learning research and delivering on showing promising results on practical applications that the community typically cares for. In relation to other NCA research, the authors take the first meaningful step in applying them to a much larger scale setting than before (image classification on ImageNet1K and various other versions of it).\n    * Several advancements are proposed: a Dynamic Interaction strategy that's quite original and seemingly effective (although I do have some gripes on how effective it really is), a dynamic programming algorithm for choosing the best layers to insert AdaNCA, new metrics (Set Cohesion Index, among other related ones), and the idea of using an NCA as an adapter to improve ViT robustness.\n* Quality:\n    * The submission is technically sound for the most part. Most claims are well supported by exhaustive experimentation and analyses, all appropriately chosen to further their narrative. \n    * The authors are honest about evaluating both the strengths and weaknesses of their approach. Although there are areas I have gripes with (which I'll mention in the Questions and/or Weaknesses sections).\n* Clarity:\n    * For the most part, this is well-written paper. Clear, concise, detailed, and thoughtful. The authors clearly put in a lot of time and effort in trying to leave no stone unturned and I commend them for that. The main manuscript and the appendix is not only exhaustive in trying to prove the benefits of their model, but the experiments chosen, the details of their experiments they listed, and the commentary provided, are all indicative of a thoughtful team that really cares to push this area of research forward.\n    * The appendix was clearly not a throwaway. I'd like to commend the authors in putting the same effort they did in the main manuscript for the appendix. Like the main manuscript, it was clearly and concisely written. Great work.\n* Significance:\n    * Are the results important?\n        * Yes. These results have given me a new perspective on how to apply NCA and I think the community at large would also appreciate these results. I believe this will convince some researchers, especially those specializing on model robustness, to pay closer attention to NCA due to the favourable cost-benefit tradeoff of using an NCA for improving the robustness of a ViT on image classification.\n            * However, I absolutely do want the authors to answer my questions provided in the Questions section below. Especially the part on the usefulness of Dynamic Interaction and its multi-scale counterpart and if it's possible to use AdaNCA with a mostly frozen and pretrained ViT.\n    * Are others (researchers or practitioners) likely to use the ideas or build on them?\n        * I am confident that this work will see adaptations of it in the near future, although I do have concerns regarding Dynamic Interaction, Multi-scale Dynamic Interaction, and if it's required to jointly train AdaNCA with a ViT from scratch. These concerns are detailed in the Weaknesses and Questions sections below.\n    * Does the submission address a difficult task in a better way than previous work?\n        * Yes. They propose an alternative and effective approach to utilizing NCA for improving robustness on downstream tasks (specifically on image classification, but I'm sure this can be applied on other tasks).\n    * Does it advance the state of the art in a demonstrable way?\n        * Although I'm not sure if it advances the state of the art, as the authors don't make any clear indication that it does, they do demonstrate that it improves many ViT-based models on image classification, particularly on noisy or OOD input. One of the models they apply AdaNCA to is amongst the state of the art (state of the art rapidly changes these days anyways), so I'm convinced this approach would also improve whatever the actual state of the art ViT-based model is on image classification.\n    * Does it provide unique data, unique conclusions about existing data, or a unique theoretical or experimental approach?\n        * As far as I can tell, yes.\n\nI just want to say that my favourite part of the experimentation in the main manuscript and the appendix was the layer similarity analyses. Everything from the motivation for focusing on layer similarity, to introducing the Set Cohesion Index, to the figures showing the similarity structure before and after AdaNCA is applied, to the plots and commentary proving a relationship between layer redundancy and robustness, to the dynamic programming algorithm for placing AdaNCA in the best spot based on this proven relationship, and to comparing it with a no-prior approach, were all well done.  Also, I'd like to commend the authors for providing the following:\n* Providing detailed training hyperparameter information in the appendix (Table 4, 5, 6, 8, 9, 10).\n* Providing detailed information of datasets in the appendix (C.14).\n* Providing model and code license information in the appendix (C.15).", "weaknesses": "* Originality:\n    * Citation and comparison with related works needs some improvement. My concern here is elaborated upon in the Questions section below. Fortunately for the authors, I think these are easily addressable.\n* Quality:\n    * Some incorrect / incomplete assumptions were made in various parts of the paper, particularly on parts commenting on where NCA get their robustness from. This concern of mine is detailed in the Questions section below.\n    * The dropout-like compensation technique shown in Eq. 7 was not ablated and thus was not sufficiently motivated. I go into more detail about my concerns about this in the Questions section below.\n    * The Dynamic Interaction strategy was not sufficiently motivated. It should be compared against the typical NCA approach of concatenation. Comparing Dynamic Interaction against a simple summation does not motivate the use of Dynamic Interaction over concatenation.\n    * The Multi-scale Dynamic Interaction strategy was not sufficiently motivated as it hurt performance at three scales, which indicates a potential fundamental flaw in the implementation. The authors provide a brief and insufficient explanation as to why it may be failing at three scales. Some more digging would be helpful here, as multi-scale approaches in neural nets typically go beyond two scales.\n* Clarity:\n    * All of my concerns regarding clarity are listed below in the Questions section. In short, there are some issues regarding redundancy and lack of explanation in some of the figures, tables, and equations; there's a lack of explanation of how NCA are typically trained; and there's a lack of commentary on the ablation study, making it difficult to appreciate the ablation results shown in Table 3.\n* Significance:\n    * The Dynamic Interaction strategy was not compared against the typical NCA approach of concatenation, and so its significance as a technique to increase NCA interaction stage efficiency is unclear. Furthermore, it could have been compared against other similar approaches instead of against a simple summation approach (which would obviously be worse). For example, comparing it against a 1x1 conv applied on depthwise conv outputs or comparing it against using a non-depthwise conv.\n    * The Multi-Scale Dynamic Interaction strategy hurt performance at three scales compared to a single-scale Dynamic Interaction, which limits the significance of Multi-Scale Dynamic Interaction. Theoretically, W_Ms should be able to zero out / ignore filter responses from useless scales, so why is it hurting performance when three scales are used?\n    * It seems limiting to have to train an entire ViT alongside AdaNCA from scratch. To really push the adapter narrative, I would suggest adapting AdaNCA to a frozen or partly-frozen ViT. I've suggested an experiment in the Questions section below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for the valuable and detailed suggestions as well as the acknowledgment of the originality, significance, and novelty of our work. We now address your questions below. \n\n- **Importance of recurrent updates**\n\nWe fully agree with you that the recurrent update scheme of NCA is crucial for the model to be robust. We will modify L40-41 and L165-166 to include this point. We want to clarify that we do not ignore the recurrence but mention it in L37-38 to state the intuition behind the advantage of using NCA, and also in our ablation study (Section 4.3) on Recur.\n\n- **Comparison with ViTCA**\n\nPlease refer to the global rebuttal \"AdaNCA and ViTCA\".\n\n- **Difference between AdaNCA and other NCA**\n\nWe agree that a discussion on this topic can help readers better understand our method, including the training, hidden state usage, initial cell states, and the pooling strategy. As we do not include them in AdaNCA training, we will add them to the Appendix. \n\n- **Stochasticity in AdaNCA**\n\nWe'd like to highlight that stochasticity during testing can hinder the evaluation of adversarial robustness by producing **obfuscated gradients** ([3] in the main paper), leading to the circumvention of adversarial attacks (L244-245 and Section C.7.1 in the Appendix). Table 5 in the R-PDF illustrates this issue, where we test the classification accuracy under CW attack. Stochasticity also results in inconsistent outputs. This is problematic for practical image classification tasks where reliable output is critical, unlike applications focusing on visual effects (Self-Organizing Textures (SOT), ViTCA) or collective behaviors (Self-Classifying MNIST (SCM)). The change in Clean Acc. in Table 5 indicates that given the same image, the stochasticity can lead to different decisions, hindering the deployment of the trained models in real-world scenarios. We also conducted your suggested experiment by removing the 1/p during training (Table 6, R-PDF). The drop in Clean Acc. and robustness suggests that downstream ViT layers struggle with varying NCA output magnitudes. Our dropout-like compensation scheme improves AdaNCA’s performance, which might be due to the different application scenario compared to previous NCA works.\n\n- **The usefulness of Dynamic Interaction and its multi-scale counterpart**\n\nFirst, we clarify that W_I is shared across scales. \n\n-- Comparison with the concatenation scheme\n\nPlease see the global rebuttal \"The usefulness of Dynamic Interaction\".\n\n-- Comments on other convolutions\n\nPros:\nBoth spatial and 1x1 conv incorporate channel mixing in the interaction stage, potentially improving the model capacity of NCA.\n\nCons:\nAdding channel mixing can cause a drastic increase in the number of parameters and FLOPS, hindering scalability. For example, using spatial conv will result in ~10M more parameters when the token is 384-dim and the number of kernels is 4 (the conv will be 1536x384x3x3 for a single scale, and we use two scales). It will also modify the paradigm used in ViT and NCA. That is, to separate token mixing from channel mixing. The modification can potentially lead to redundant information. \n\n-- The usefulness of multi-scale Dynamic Interaction\n\nOur motivation for using multi-scale interaction is to perform more efficient token interaction learning over local scales since ViT already contains **global information**. Enlarging the neighborhood size will: 1) Complicate the process of selecting the neighbors to interact with; 2) Introduce excessive noise, making it difficult for tokens to accurately acquire neighbor information. 3) Repeatedly acquire the global information provided by ViT. The recurrence further amplifies the noise. We point out that **ViTCA also suffers from the neighborhood size issue** in image generation (Table 5 in their Appendix). Theoretically, self-attention can discount far-away information as the tokens also perform a weighted sum on the neighborhood values with the weight generated by itself while it struggles with large neighborhood sizes. In contrast, our model gains performance when S=2 (5x5 neighborhood), indicating the usefulness of our multi-scale module. Although increasing model capacity (as you suggested by using larger filters or increasing the receptive field of W_Ms) can improve the performance (Table 7, R-PDF), AdaNCA struggles with overly large neighborhoods. We will add a paragraph to discuss this issue. We note, however, that our work pioneers the application of multi-scale interaction in large-scale image classification tasks.\n\n- **Integrating AdaNCA into pre-trained ViT**\n\nPlease see the global rebuttal \"Integrating AdaNCA into pre-trained ViT models\". \n\n- **Comment on the ablation study**\n\nDue to the space limit of the main paper, we did not include the analysis of the ablation study. Here, we provide our analysis below and we plan to add it as a Section in the Appendix: As shown in Table 3 (main paper), the highest robustness improvement is achieved with all components. Among them, turning off the recurrence leads to the largest drop in the robustness, as recurrence allows the model to explore more cell states than finishing the update in a single step. While it achieves the highest clean accuracy, it uses ~4x more parameters than our method, and the improvement of the clean performance is likely due to more parameters. Without any of the two sources of randomness, stochastic update and random steps, the model cannot adapt to the variability of the inputs and thus exhibits vulnerabilities against adversarial attacks. Finally, turning off our Dynamic Interaction will cause drops in both clean accuracy and robustness, as tokens cannot decide their unique interaction weights and thus cannot generalize to noisy inputs. \n\n- **Citation, clarification, and typos**\n\nWe highly appreciate your thorough and constructive feedback on the texts, figures, tables, and citations. We plan to incorporate all of them in our revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t7hML1dodD", "reviewer": "Reviewer_3x5z", "summary": "This paper introduces Adaptor Neural Cellular Automata (AdaNCA), a plug-and-play module designed to enhance the robustness and performance of Vision Transformers (ViTs). The innovation lies in integrating Neural Cellular Automata (NCA) as intermediary adaptors between the layers of ViTs. The paper demonstrates that AdaNCA can significantly improve the robustness of ViTs against adversarial samples and out-of-distribution inputs. The authors also propose a Dynamic Interaction mechanism to reduce computational costs and provide an algorithm to optimize AdaNCA placement within ViTs.", "questions": "- Would be good to elaborate a bit more on how the Dynamic Interaction works and why it is novel\n- How does the developing pattern of the NCA look over time? Are differences visible between a version with and without dynamic interaction?", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The integration of NCA into ViTs is a novel approach that addresses the robustness issue prevalent in current ViT architectures.\n- AdaNCA shows improvement in robustness with only a small increase in parameters\n- The results show improvements across 8 robustness benchmarks and 4 different ViT architectures.\n- AdaNCA demonstrates improvement in accuracy under adversarial attacks on the ImageNet1K benchmark", "weaknesses": "- The experiments are primarily conducted on image classification tasks. It would be ideal to see how AdaNCA performs in other domains or tasks, such as object detection or segmentation, to assess its generalizability.\n\n- The scalability of AdaNCA to very large-scale datasets and higher-resolution images is not yet studied.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your positive feedback on our work. We now address your questions.\n\n- **The generalizability and scalability of AdaNCA**\n \nWe fully agree with you that applying AdaNCA in other computer vision tasks and scaling it to larger datasets and higher-resolution images are worthwhile. Given the fact that AdaNCA can scale up to much larger sizes than all previous NCA models and its effectiveness in large-scale image classification tasks, we are confident in AdaNCA's competitiveness in tasks such as robust semantic segmentation. In fact, we are actively looking into robustness benchmarks such as CityScape-C [1] and ACDC [2] and we are keen to explore these problems in the future.\n\n- **Elaboration of Dynamic Interaction**\n\nWe are happy to explain our proposed Dynamic Interaction in more detail. Assume we have a token state $\\mathbf{S} \\in \\mathbb{R}^{H \\times W \\times C}$. After each $C$-dim token interacts with its neighbors using $\\mathcal{M}$ different depth-wise convolutions, we have $\\mathcal{M}$ tensors with the same shape as the input token state $\\mathbf{S}$. Our Dynamic Interaction first computes a token-wise scalar weight tensor $\\mathbf{W}_{\\mathcal{I}m} \\in \\mathbb{R}^{H \\times W \\times 1}$ using a two-layer CNN for each interaction results. Hence, the output of the CNN is of shape $H \\times W \\times \\mathcal{M}$. The input of the CNN is the token map. The CNN consists of two 3x3 convolution layers with a batch normalization layer in the middle. The interaction results are then computed using the right-most hand side of Equation 8, in which a weighted sum is performed on the $\\mathcal{M}$ interaction results. Each token has a different set of weights for aggregating the interaction results. Hence, the tokens dynamically adjust the weights of the combination on the interaction results based on both their own state and the states of their neighbors, which we term Dynamic Interaction. We will improve the texts of Section 3.2.1 to make it clearer.\n\n- **Novelty of Dynamic Interaction**\n\nPlease see the global rebuttal ”The usefulness of Dynamic Interaction”.\n\n- **Developing patterns of AdaNCA**\n\nPlease refer to Figure 1 in the R-PDF.\n\n\n[1] Michaelis, Claudio, et al. \"Benchmarking robustness in object detection: Autonomous driving when winter is coming.\" arXiv preprint arXiv:1907.07484 (2019).\n\n[2] Sakaridis, Christos, Dengxin Dai, and Luc Van Gool. \"ACDC: The adverse conditions dataset with correspondences for semantic driving scene understanding.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7p7wCiWAwI", "reviewer": "Reviewer_oE4b", "summary": "The authors propose the introduction of NCAs into VisionTransformers to improve the robustness against adversarial inputs as well as out of distribution data. This improves basic ViT architecture by up to 10% against specific adversarial attacks. In addition this modification, they propose Dynamica Interaction, for faster communication between cells/tokens. As the choice of the right layer to insert the NCAs is not trivial, they introduce a method to find the most efficient position based on the redundancy. The whole method has been evaluated against a good number of methods and on OOD data as well as adversarial inputs.", "questions": "- I very strongly encourage including the results for the original \"TAPADL\" [22], with the correct loss for training. I do not see it as a problem if you perform worse, which is likely the case, but it is absolutely necessary for completness. \n\n- Instead of doing this misleading modification to the TAPADL, did you try to add the ADL loss to your modified model? \n\n- How is it possible, that the NCAs have such a huge number of parameters? One major advantage of them is, that they do not need many parameters, so im curious why you chose to make them 1-2.5m parameters big.  \n\n- Why did you choose the number of steps for NCAs so low? It is very atypical for NCAs to run for 2-3 steps. \n\n- It is not one of my major points, but can you call something an NCA if runs for 2-3 steps? \n\n- Your statement about the release of the code is unclear to me. Will you release it in case of acceptance?", "limitations": "The authors have adequately addressed the limitations", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- I highly appreciate the effort of integrating NCAs into existing architectures, to combine the established method with the specific traits that NCA can bring, such as the authors have shown: robustness \n\n- The authors show generally strong robustness improvements when comparing the \"classical\" ViT variants, RVT-B, FAN-B, Swin-B and ConVitB with their AdaNCA variants \n\n- The proposed method for determining the best position for inersting the NCAs is sound \n\n- The manuscript is generally a well written paper and good to follow. The appendix adds valuable extra information", "weaknesses": "- My major concern is the choice of baselines. While generally a good selection has been made, i strongly dislike the choice of removing the additional ADL loss (according to [22] the ADL loss is more important than TAP). The only logical explanation I could find for this choice, is that it performs better than the NCA based method.  \n\n- This is especially misleading since you call your baseline \"TAPADL-RVT\". But the \"adl\" in this method stands for the loss.  \n\n- Considering that you do not use the \"adl\" loss Table 2 is particularly unfair, as these are OOD examples, something the loss particularly optimizes on.  Especially considering that the difference is not very big even without it. \n\n- Why does Table 2 once compare with the standard baseline and once with TADPADL? \n\n- The argumentation in appendix D, why the comparison with classical / concatenating NCAs is missing is unclear. Why would this require NCAs with 10 million parameters? In general it is not clear, why the NCAs are this big \n\nMore minor remarks: \n\n- \"enables the modeling of global cell representations\" in abstract is confusing \n\n- They choice of hyperparameters is unclear. Why is e.g. the EMA different inbetween AdaNCA setups?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your valuable feedback on our experiments. We will improve the clarity of the sentence in the abstract. We now address your major concerns below.\n\n- **Choice of baselines**\n\nWe fully agree with you that the original results reported in the TAPADL paper should be included for completeness, and we will add corresponding results to our manuscript. However, we'd like to clarify that we choose the **original TAPADL models**, instead of the one without ADL loss, and download them from the official code repository. We test them on our own machine, which might lead to different results from the original paper due to the difference between hardware and platforms. Importantly, the test results of our own code match the ones produced by their official code, which are the results reported in our paper. We **do not train** the TAPADL models as our focus is on the improvement of AdaNCA-enhanced ViTs compared with the baselines. We include the results from TAPADL models to showcase the strong robustness improvement that AdaNCA can achieve. In fact, our reported results **match most of the reported ones** in their main paper (citation [22] in our paper), with differences only in the IM-A results of both TAPADL models and the IM-R results of TAPADL-FAN. We assume these differences might originate from different data processing methods. However, we emphasize that we test our AdaNCA-enhanced model using the **same setting** as we test the TAPADL models, which follow the hyperparameters of the corresponding baseline models. \n\n- **Table 2 comparison with TAPADL**\n\nWe'd like to point out that we do not have access to the official RVT model trained on ImageNet1K, hence we use the TAPADL model as a proxy in non-adversarial robustness evaluation. We state this compromise in Section C.8 in the Appendix. We will add this clarification to our main manuscript. In all other similar comparisons, as provided in Table 13 in the Appendix, we use the standard baselines. \n\n- **The usage of ADL loss**\n\nWe clarify that we do not use ADL loss to train our models since we focus on our proposed architectural changes, as stated in L235-237. \n\n- **NCA parameters**\n\nWe highlight that AdaNCA works with cells that have much higher dimensionalities. For example, NCA in texture synthesis typically adopts a cell dimensionality of 16. ViTCA operates in 128-dim space. In ViTs, however, the cell dimensionality can be 768 or 1024, resulting in a large amount of parameters in the MLP of NCA. We choose the hidden dimensionality of the MLP to be equal to the input dimensionality, as we do not want to have an information bottleneck (Section C.3 in the Appendix). Hence, an NCA working with a cell dimensionality of 1024 can have 1024 x 1024 x 2 $\\approx$ 2M parameters in the MLP part. Moreover, we insert 2-3 AdaNCA models into ViT to achieve a good balance between the computational costs and performance improvement. Therefore, the final increase in parameters can fall in the range of 1-2.5M, as you indicated. Regarding the concatenation scheme, NCA parameters can exceed 10M as shown in the Global rebuttal \"The usefulness of Dynamic Interaction\". It is because the input dimensionality to the MLP is $\\mathcal{M}$x larger if we use $\\mathcal{M}$ kernels for interaction. Considering the above example where the cell dimensionality is 1024, the MLP now has the weight tensor of the first MLP being 4096x4096 with $\\mathcal{M}=4$ if avoiding information bottleneck (Section C.3 Appendix), which is already 16M. We believe more efficient instantiation of the Update stage in NCA is worth exploring, and we are happy to add a paragraph discussing this future direction in our paper. \n\n- **NCA steps**\n\nWe agree that our choice of NCA steps differs from all previous NCA models. We make this compromise as the computational costs increase linearly as the NCA step grows. We aim to minimize the increase in the number of parameters and FLOPS for scalability and we do not want the source of improvement to merely stem from the increase in the size and computation of the models. We show that more NCA steps will indeed contribute to the model's performance, as shown in the following table, while it introduces extra FLOPS. The model setting is the same as in our ablation study in Section 4.3.\n\n|        | # Params (M) | FLOPS (G) | Clean Acc. ($\\uparrow$) | Attack Failure Rate ($\\uparrow$) |\n|:------:|:--------:|:-----:|:----------:|:-----------------------:|\n| Step=4 |   27.94  |  4.7  |    87.18   |          22.35          |\n| Step=5 |   27.94  |  4.8  |    87.22   |          23.46          |\n\nWe want to underscore that it is the architecture and evolution scheme that defines an NCA as introduced in Section 3.1, instead of the number of its recurrent steps. Admittedly, less steps will lead to a coarser path to the target state, and can potentially undermine the model's performance. Notably, with our scheme, AdaNCA has achieved generally strong robustness improvements compared to the baselines. It indicates that our choice of the recurrent steps of AdaNCA is a good balance between computational costs and performance. \n\n\n- **Choice of hyperparameters**\n\nOur choice of the training hyperparameters follows the settings of corresponding baseline models, as listed in Section C.6, L718-719. Hence, their differences in hyperparameters result in the differences in our experiments. We want to highlight that our AdaNCA can adapt to different training hyperparameters, e.g., learning rates, EMA (on/off, different decay rates), batch size, etc, which indicate the strong adaptability of AdaNCA in different training settings as well as its promise in combining it with any ViT models.\n\n\n- **Code release**\n\nYes, we will release cleaned and well-documented code in case of acceptance, as well as the pre-trained models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZPxncJp9FJ", "reviewer": "Reviewer_ZhPF", "summary": "The work presents present a neural network architecture that combines vision transformers and neural cellular automata. The work shows that this hybrid architecture competitively at image classification task with a downsampled version of Imagenet (224x224). Furthermore, the results show a small improvement on performance against adversarial attacks and OOD test against the chosen vision transformers baselines.", "questions": "- How does AdaNCA differs from Vision Transformer Cellular Automata model from Attention-based Neural Cellular Automata, Tesfaldet et al. 2022?\n\n-What's the motivaiton of using channel-weighted convolution for the perception part part of the NCA (what you call \"dynamic interaction\")? Wouldn't this be equivalent to the transformation that the update MLP does on concatenated channels vectors from standard depth-wise conv layers?\n\n- Why training AdaNCA from scratch rather than adding it to a pretrained ViT model? Could you report performance on this?\n\n- The paper conjectures that it is the stochasticity of the NCA that may be positively contributing to the AdaNCA performance for adversarial attacks. Have you tried introducing noise during training to the ViT baselines to compare performance?\n\n- Line 163 : “NCA typically queries the cell states at a random time step T”. I don't understand what do you mean with \"NCA queries a cell state\".", "limitations": "- No code is provided.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper makes use of hybrid architecture (NCA and ViT), a combination that remains underexplored in the current machine learning literature.\n- The presented model performs competitively against the baselines.\n- The paper is clear, well written, and results are well presented with an thorough appendix detailing training setup.", "weaknesses": "- Readability of Table 1 could be improved: it is not indicated the values for adversarial attacks and OOD inputs: accuracy? error? In some cases high number are bold, in other low number are bold.\n\n- Please report whether the results are for the single best trained model.\n\n- The argument put forward by the authors to motivate the AdaNCA architecture is its competitive performance, however the magnitude of the performance improvement is limited, notably keeping in mind that the work is demonstrated on a downsampled version of Imagenet.\n\n- Since the architecture is presented as plug-and-play, it's unclear why the AdaNCA is not simply added to a ViT pre-trained models. Having a simple way of improving robustness of ViT with a quick training of the NCA would make the model much more impactful and scalable. \n\n\n## Typos:\n- Plug-in-play -> Plug-and-play", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your valuable comments and positive feedback. We now address your questions below.\n\n- **Comparison with ViTCA**\n\nPlease refer to the global rebuttal ”AdaNCA and ViTCA”.\n\n- **The difference between MLP processing concatenated channel vectors and Dynamic Interaction**\n\nPlease see the global rebuttal ”The usefulness of Dynamic Interaction”.\n\n- **Integrating AdaNCA into pre-trained ViT models**\n\nPlease see the global rebuttal \"Integrating AdaNCA into pre-trained ViT models\". \n\n- **Noisy training of baseline ViT models**\n\nWe are keen to explore the possibilities of integrating NCA training strategies into the current ViT training pipeline, as we believe there are connections between the two model families (Section 3.1.1). Nevertheless, we want to point out that our focus is the integration of NCA and ViT. We made a first attempt at incorporating one training strategy, the stochastic update as you indicated, into current ViTs. \n\n|                    | Clean Acc. ($\\uparrow$) |    Attack Failure Rate ($\\uparrow$)  |\n|:------------------:|:----------:|:-----------------------:|\n| Swin-Tiny Baseline |    **86.56**   |          12.29          |\n|   Swin-Tiny StocU  |    85.80   |          **13.40**          |\n\nThe stochastic update improves the model's robustness while undermining its clean performance. We assume this is because of the incompatibility between NCA which uses a local recurrent interaction scheme while ViT uses a global single-pass one. Despite this, it supports the notion that stochasticity during training indeed improves robustness.\n\n- **Magnitude of performance**\n\nWe wish to highlight that AdaNCA can contribute to over 10\\%  absolute improvement in adversarial robustness, as shown in Table 1, Section 4 in our main manuscript, with a small cost. For instance, ConViT-B-AdaNCA uses merely 3\\% more parameters and 7\\% more FLOPS to achieve more than **10x improvement** compared to the enlarged baseline model (* sign). More importantly, our method achieves on-par or better results than a recent advanced modification of ViTs, namely the TAPADL method. Therefore, AdaNCA provides a good trade-off between the performance and computational costs, facilitating its deployment. \n\n- **Clarification**\n\n-- Line 163, NCA queries cell states:\n\nWe are happy to clarify more details of the NCA evolution. Following Equation 6 in Section 3.1, NCA evolves in a recurrent manner for certain steps. Instead of evolving for a fixed number of steps, NCA typically runs for random steps, where the number is randomly selected from a range. After evolution, the cell states are used for downstream tasks. In our case, AdaNCA outputs activations for the next ViT layer. Hence, the usage of the cell states happens at a random step, and we refer to this process as NCA querying the cell states. We will improve the text and make it clearer.\n\n-- Readability of Table 1:\n\nThank you for your suggestion. We will improve the readability of Table 1 by adding arrows indicating the direction of performance growth. We clarify that the IM-C uses a different metric than other benchmarks that use accuracy, as stated in L257-258. It corresponds to a relative classification error compared to a pre-trained AlexNet, and thus lower is better. \n\n-- Single Best Model: \n\nYes, our model performance is from a single best trained model, to facilitate the release of the pre-trained weights. The best model we obtained almost all fall into the last 10 epochs of training, with some exceptions in our ablation studies. All models are from the last 15 epochs of training.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their valuable comments and the acknowledgment of our contributions. We are glad to note that all reviewers (h6zC, 3x5z, oE4b, ZhPF) agree on:

- **The novelty and significance of our method in integrating NCA into ViT.**
- **The effectiveness of AdaNCA in improving the performance of ViT, including clean accuracy and robustness against adversarial attacks and OOD inputs.**

Moreover, reviewers h6zC and oE4b remark that our method in determining the best insert positions of AdaNCA is sound. Reviewers h6zC and 3x5z mention the thoroughness of our experiments, and reviewer 3x5z recognizes the small cost brought by AdaNCA compared to the improvement it achieves. 

We identify some common questions from the reviewers: 

- h6zC, 3x5z, ZhPF: The usefulness of our proposed Dynamic Interaction and its difference from the existing NCA concatenation method.

- h6zC, ZhPF: Comparison between AdaNCA and ViTCA. 

- h6zC, ZhPF: The integration of AdaNCA into a pre-trained ViT model instead of training from scratch. 

We address these below. We conduct all experiments using the same setting as in our ablation study (Section 4.3), namely Swin-Tiny on ImageNet100, except for the integration of AdaNCA and pre-trained ViTs, which uses ImageNet1K and Swin-base. 

- **The usefulness of Dynamic Interaction**

Our motivation for developing the Dynamic Interaction module is the high dimensionality of feature vectors in modern ViT models, as stated in L43-51 and Section D in the Appendix. We qualitatively showcase in Figure 1, rebuttal PDF (R-PDF), that Dynamic Interaction helps robustify AdaNCA when facing noisy inputs. We underscore that any operations involving linear transformations of the concatenated interaction results will lead to drastic increases in computational costs, as shown in Table 1 in the rebuttal PDF. Note that the concatenation scheme nearly doubles the FLOPS for RVT compared to the baseline, leading to difficulties in training. Such an increase renders scalability a challenge. We agree that more parameters can contribute to better performance (Table 2, R-PDF, numbers in parentheses indicate performance improvement compared to the Baseline). Note, however, that our Dynamic Interaction achieves ~70\% of the improvements of the original NCA concatenation scheme (10.06/14.62) in robustness improvement and ~60\% improvement in the clean accuracy (0.62/1.06), with merely ~10\% of the parameters and FLOPS (0.35/2.99 for \# Params and 0.2/2.3 for FLOPS). Therefore, our Dynamic Interaction scheme provides a good trade-off between the performance and computational costs. It is capable of scaling up, allowing us to insert AdaNCA into even larger models, such as current vision-language models where the token dimensionality is even higher.

- **AdaNCA and ViTCA**

We acknowledge that ViTCA is the first Vision Transformer architecture to incorporate NCA. However, we highlight some key differences between AdaNCA and ViTCA. 

1. AdaNCA focuses on robustness in image classification, whereas ViTCA is primarily designed for image reconstruction. These different goals necessitate different structural designs in the two frameworks. 

2. ViTCA studies the behaviors of tiny NCA (~100K) on small datasets such as MNIST, while AdaNCA-enhanced ViT can deal with sizes larger than 90M and can operate on ImageNet1K.  

3. ViTCA's usage of self-attention for token interaction learning results in a higher parameter count compared to our proposed Dynamic Interaction approach (see Table 3 in R-PDF).

4. Tokens in ViTCA correspond to pixels instead of image patches. 

Nevertheless, we conducted experiments using ViTCA's local recurrent attention scheme to explore the potential benefits of integrating ViTCA into our framework. Our training setting is the same as in our ablation study (Section 4.3). Results are given in Table 3 in R-PDF. Despite having more parameters, the ViTCA-like scheme performs worse in clean accuracy and on-par with AdaNCA in robustness. This indicates that it is promising to explore the possibility of incorporating ViTCA into our framework. 

- **Integrating AdaNCA into pre-trained ViT models**

Implementing AdaNCA as a plug-and-play module for pre-trained ViT models would certainly improve the training speed. However, the current NCA might not be able to adapt to such a scheme. NCA may struggle to effectively transmit information between two pre-trained ViT layers, as these layers have already established strong connections. In contrast, training the model from scratch allows NCA and ViT to synergistically adapt to feature variability, resulting in better overall performance. To explore this, we experimented by inserting AdaNCA into a pre-trained Swin-base model on ImageNet1K by 1) freezing all ViT layers; 2) training only the boundary layers; 3) Finetuning all layers. Results are given in Table 4 in the R-PDF. None of the schemes perform as well as training from scratch. However, it is worth exploring in the future.

Meta-review: **Accept** with minor revisions: The paper demonstrates significant potential, especially in terms of improving robustness and efficiency. However, there are gaps in the experimental setup (e.g., missing baselines, more extensive comparisons) and in the explanation of the key contributions like Dynamic Interaction. These issues are not major roadblocks but should be addressed in a revision to align with NeurIPS' standards. The paper offers promising contributions to the community and could serve as a foundation for further research in robust ViTs.
Decision: Accept (poster)