Structured Initialization for Vision Transformers
Authors: Jianqiao Zheng
Xueqian Li
Hemanth Saratchandran
Simon Lucey
Venue: NeurIPS 2025 poster ()
Keywords: Transformers
Weight Initialization
Attention Mechanism
Convolutional Neural Networks
Inductive Bias
Primary area: deep_learning

Abstract: Convolutional Neural Networks (CNNs) inherently encode strong inductive biases, enabling effective generalization on small-scale datasets. In this paper, we propose integrating this inductive bias into ViTs, not through an architectural intervention but solely through initialization. The motivation here is to have a ViT that can enjoy strong CNN-like performance when data assets are small, but can still scale to ViT-like performance as the data expands. Our approach is motivated by our empirical results that random impulse filters can achieve commensurate performance to learned filters within a CNN. We improve upon current ViT initialization strategies, which typically rely on empirical heuristics such as using attention weights from pretrained models or focusing on the distribution of attention weights without enforcing structures. Empirical results demonstrate that our method significantly outperforms standard ViT initialization across numerous small and medium-scale benchmarks, including Food-101, CIFAR-10, CIFAR-100, STL-10, Flowers, and Pets, while maintaining comparative performance on large-scale datasets such as ImageNet-1K. Moreover, our initialization strategy can be easily integrated into various transformer-based architectures such as Swin Transformer and MLP-Mixer with consistent improvements in performance.

Structured Initialization for Vision Transformers
Jianqiao Zheng"
Xueqian Li
Hemanth Saratchandran
Simon Lucey
Australian Institute for Machine Learning The University of Adelaide
Abstract
Convolutional Neural Networks (CNNs) inherently encode strong inductive biases, enabling effective generalization on small-scale datasets. In this paper, we propose integrating this inductive bias into ViTs, not through an architectural intervention but solely through initialization. The motivation here is to have VIT that can enjoy strong CNN-like performance when data assets are small, but can still scale to ViT- like performance as the data expands. Our approach is motivated by our empirical results that random impulse filters can achieve commensurate performance learned filters within CNN. We improve upon current ViT initialization strategies, which typically rely empirical heuristics such using attention weights from pretrained models or focusing on the distribution attention weights without enforcing structures. Empirical results demonstrate that method significantly outperforms standard ViT initialization across numerous small and medium-scale benchmarks, including Food- 101d-101, CIFAR-10, CIFAR-100, STL-10, Flowers, and Pets, while maintaining comparative performance on large-scale datasets such as ImageNet- K. Moreover, initialization strategy be easily integrated into various transformer-base architectures such as Swin Transformer and MLP-Mixer with consistent improvements performance.
Introduction
Despite their success on large-scale training datasets, Vision Transformers (ViTs) often suffer notable drop in performance when trained on small-scale datasets. This limitation is primarily attributed to their lack of architectural inductive biases, which are crucial for generalization with insufficient data. In contrast, Convolutional Neural Networks (CNNs) possess strong inductive biases that allow them to perform well even with limited training data.
To bridge this performance gap, several strategies have been proposed. These include self-supervised pretraining on large-scale datasets X 26), advanced data augmentation techniques [3] (6, and hybrid
More recently. Zhang et al. (37) explored the use of pretrained weights to initialize ViTs. Building on this idea, subsequent works have shown that carefully designed network initialization strategies can enhance VIT performance on small-scale datasets without modifying the model architecture. In particular, Trockman and Kolter (28 introduced mimetic initialization that replicates the weight distribution of pretrained ViTs. Similarly, Xu et al. (33 proposed directly sampling weights from large pretrained models.
architectures that incorporate convolutional layers into Vision Transformers [22 16
35 15
While effective, these methods come with three notable limitations: (1) they focus on replicating the distribution of pretrained attention weights rather than structuring attention maps; (2) they rely access to pretrained models trained on large-scale data, which is often impractical in domain-specific scenarios; and (3) their effectiveness is often tied specific model architectures.
"iaaqqiao zheng @ adelaide.edu. ad..au. Code is available at alhttps://githim http::///ggcuuuucccooiiireeeeeee ccan//ssrriszjggssuuuueee
initialization
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Generative Initilization
Structured
Initialzzution (Ours)
Sampling
K
K
Q
K
Parameters
Kj
K
Solving
Attention Maps
Streetured
Impuuse
Filters
Figure 1: Illustration of conventional generative initialization and structured initialization (ours) strategies for the weights Q and K of the attention map in transformers. Conventional generative initialization involves sampling Q and K from certain distributions, such as Gaussian or Uniform, resulting unstructured attention maps. In contrast, our structured initialization imposes constraints on the structure of the initial attention maps, specifically requiring them to be random impulse filters. The initialization of Q and K computed based this requirement. Note that in both attention maps and random impulse filters, the pink cells indicate ones, while the gray cells represent zeros.
To overcome the limitations existing approaches, we propose a novel "structured initialization" strategy for ViTs that imposes convolutional structures on attention maps without requiring any pretrained models. Our approach grounded in our theoretical insights, aligned with the findings of recent work [3]. that randomly initialized depthwise convolution filters can match the performance of their trained counterparts in models such ConeMixer [27] and ResNet [T]. Inspired by this, we develop an initialization strategy for ViTs based random impulse convolution kernels, which impart locality and structure directly into the attention maps, as shown Fig. [ This structured initialization yields inductive biases characteristic of CNNs, enabling ViTs to generalize effectively small-scale datasets, while preserving their adaptability for large-scale applications. Unlike prior methods that modify ViT architectures by integrating convolutional components, our method preserves the original transformer structure, making broadly applicable across variety of ViT variants.
To conclude, our paper makes the following contributions:
We establish conceptual link between the structural inductive bias of CNNs and initializa- tion in ViTs, and provide a theoretical justification for using random convolution filters to initialize attention maps.
To the best of our knowledge, are the first to introduce the initialization strategy that explicitly structures attention maps ViTs, embedding convolutional inductive biases without modifying model architecture, enabling compatibility across diverse ViT variants.
We demonstrate ttateofftthe-a performance on small-scale and medium-scale datasets, including Food-101, CIFAR-10, CIFAR-I00, STL-10, Flowers, and Pets, while maintaining competitive results on large-scale datasets such ImageNet-1K 1K, and achieving improved performance across various ViT architectures, such as Swin Transformers and MLP-Mixer.
2 Related Work
Introducing inductive bias of CNN to ViT through architecture. Many efforts have aimed incorporate a convolutional inductive bias into ViTs through architectural modifications. 7 proposed to combine convolution and self-attention by mixing the convolutional self-attention layers.
[5 intrrddued hybrid models wherein output of each layer is summation of convolution and self-attention. [32] explored using convolution for token projections within self-attention, while [35 showed promising results by inserting a depthwise convolution before the self-attention map. 99] introduced gated positional self-attention to imply soft convolution inductive bias. Although these

---

Nx1
Z Embedding Basis
Patch Embeddings
Convolution Filters
Figure 2: Illustration of why random spatial convolution filters are effective. Patch embeddings
X R xx are typically rank-deficient and can be approximately decomposed k basis. Meanwhile, a linear combination of f linearly independent filters h can express any arbitrary filter in the filter space fx5, Based on these two observations, we derive the inequality D 2 f from Proposition
techniques have been proven effective, they aim to introduce the inductive bias of convolution through architectural choices. Our approach, on the other hand, stands out by not requiring any modifications to the architecture, retaining the generalizability to be seamlessly applied to different settings.
Initializating ViTs from pretrained weights. The exploration of applying inductive bias through initialization within transformer is limited to date. (37) posited that the benefit of pretrained
models in ViTs can be interpreted more effective strategy for initialization. [28] 29] recently
investigated the empirical distributions of self-attention weights, learned from large-scale datasets,
and proposed a mimetic initialization strategy. While this approach lies between structured and generative initialization, it relies on the pretraining results of large models. [23] 23 [4] directly sampled weights from pretrained large-scale models initialization for smaller models. While effective, the sampled weights must follow the distribution of these pretrained weights. A key difference in our approach that our method does not require offline knowledge of pretrained models. Instead, our initialized structure is derived from theoretical analysis of convolution layers.
Convolution attention. Since their introduction 33 8). the relationship between transformers and CNNs has been a topic of immense interest. I studied the structural similarities between attention and convolution, bridging them into unified framework. Building this, (5] demonstrated that self-attention layers can express any convolutional layers through careful theoretical construc- tion. While these studies highlighted the functional equivalence between self-attention ViTs and convolutional spatial mixing in CNNs, they did not delve into how the inductive bias of ViTs could be adapted through this theoretical connection. In contrast, our work offers a theoretical insight: random convolutional impulse filter can be effectively approximated by softmax self-attention.
3 Why Random Impulse Filters Work?
well established that both ConvMixer and ViTs use alternating blocks of spatial and channel mixing, where ViTs replace the spatial convolutions in ConvNets with attention mechanisms. A fun- damental difference, except for the receptive field, lies their parameterization: spatial convolutions with hundreds of channels typically use distinct kemels for each channel, whereas self-attention relies on a shared mechanism with only a limited number (~ 10) of attention heads. By introducing a key often rank-deficient, we demonstrate that long the spatial
observation that input embeddings kernels or attention patterns sufficiently span the kernel space, the uniqueness or repetition of individ- ual kernels becomes less critical. This insight, supported by recent empirical observation [3], offers a new perspective on the relationship between ViTs and CNNs, motivating the use of impulse-base based structures to embed convolutional inductive biases into the attention map initialization.
In recent work 3 Cazenavette el demonstrated remarkable performance of randomly initialized convolution filters in ConvMixer and ResNet when solely learning the channel mixing parameters. However, they failed to offer any insights into the underlying reasons. In this section, we provide theoretical analysis of how solely learning channel mixing can be sufficient for achieving reasonably

---

good performance. Our theoretical findings are significant as they establish conceptual link between the architecture of ConvMixer and the initialization of ViT, offering a deeper understanding of desired properties for spatial mixing matrices. Without losing generality, we have omitted activations (e.g., GeL.U, ReLU, etc.), bias, batch normalization, and skip connections in our equations for clarity.
Remark Let us define the patch embeddings or intermediate layer oulputs in ConvMixer as [= X2,..., XD], where D is the number of channels and N is the number of pixels in the vectorized patch x E R.. An interesting observation is the rank (stable rank, defined as Σ 0² /
of X is consistently much smaller than the minimum dimension min D) of X. indicating significant amount of redundancy in patch embeddings or intermediate layer outputs in deep networks.
This rank deficiency common in various deep neural networks [TO), especially ViT's [20 18 T2).
Let us define a 2D convolution filter as h e ER/___ In general, this kernel can be represented as circulant matrix H E RNNN such that h x where denotes convolution operator. The relation between the convolutional matrix and convolution filters is explained in detail in the appendix. ConvMixer block T RD + R is composed of a spatial mixing layer T : X + IR xD and a channel mixing layer T IR N + R
XD where T is defined
by a sequence of convolution filters H= (H,, , H2...., Hp] E ERRNNXA N H. ERRNN and Tea is
defined by a weight matrix W E DxD With input X = [X1, .2,..., xDl ERD T is
Tony X H) = H H2x2,....... HDXD],
Cony T T
[X; W) = -,W,
(2)
X T (X;H); W) = [ H2x2. HDxD W.
Definition 1 (M-k Spanned Set): Let V= (V1, V2...., ) C. R.d be a finite set of vectors. We say that V is M spanned (on W) ) if there exists a partition of V into at least one-olelap subsets:
== Vi UV2 U...U ...U
(4)
such that there exist -ddimensional subspace W C Rd in the span of each subset V; W C Sp)n(V;)
Vi= = ..... k.
(5)
A ConvMixer block T consists of spatial mixing layer Ts( HH) with convolution filters H and a channel mixing layer Tc. T is another ConvMixer block composed of T' ::( HH) and T c. Let l be the rank of input X. H is M-k spanned on W, for any set of filters H' C W and any T there always exists a Tc such that T(X)=T((X)
Proposition 1
For simplicity, we include the full proof in the appendix. Note that since H are convolution matrices, their span lies in R instead of N where f is the kernel size. In practice, the fact that D z f2 indicates that randomly initialized spatial convolution kernels are f2-k spanned and satisfy Proposition I as illustrated in Fig. 2 Consequently, any trained results of T and T's can be achieved by solely traming the Tc, while keeping a fixed spatial mixing layer Ts. Hence, the following corollaries can be obtained, and Corollary D explains the phenomenon found in [3], mentioned at the beginning of the section. Corollary 2 related experimental evidence of these corollaries and our proposition is given in the appendix.
inspires our proposed structure initialization for ViTs. The
Corollary Randomly initialized spatial convolution filters perform as well as trained spatial convolution filters since the fe-k spanned condition in Proposition 1 is satisfied.
Corollary Random impulse spatial convolution filters perform as well as trained spatial convolu- tion filters since the f2-k spanned condition in Proposition 1 is satisfied.
Corollary Spatial convolution filters with all ones (referred to as "box" filters) will not perform well since they are 1-k spanned and can produce only averaging values.
Structured Initialization for Attention Map
4.1 Expected Initialized Attention Map Structure
Cine and VIT share most of the components in their architectures. The gap in their performance on small-scale datasets stems from their architectural choices regarding the spatial mixing matrix.

---

Although depthwise convolution (ConvMixer) and multi-head self-attention (ViT) may appear distinct at first glance, their underlying goal remains the same: to identify spatial patterns indicated by the spatial mixing matrix. As defined in Sec. 3, similar to the spatial mixing step in ConvMixer defined Eq. (T). the spatial mixing step of multi-head attention can be expressed as
S (X; M) M ................................ MIx, M2xa+1..... M2X2d.
M (1-1411
MA
(6)
where d represents the feature dimension in each head, typically set D/h, with h being the number of heads, and the matrices M, for multi-head self-attention can be expressed as follows:
M, = softmax (XQ, K.X"),
where Q; K, ERDx are attention weight matrices.
is worth noting that in Eq. (T), the spatial matrices H are in convolutional structure, resulting in a despite each H, having size of N x N. This structural constraint
span of /x/ instead of R ensures that CNNs focus on local features but struggle to capture long-range dependencies. In contrast, the span of spatial matrices M in Eq. (6) is IR x allowing for greater learning capacity without these limitations. However, a randomly initialized Q and K contain no structural information, resulting in random matrices depicted in the bottom left of Fig.
Leveraging this insight, we propose to initialize the attention map for each head in ViT to a con- volutional structure denoted in the bottom right of Fig. [ Our initialization strategy preserves both the advantage of locality and the capacity to learn long-range information. For clarity and brevity, the following discussions will focus only from Eq. and Eq. (T. our structured initialization strategy can be represented as
head of multi-head self-attention. Therefore,
VIT MX: init T (X; M) = M softmax (XQ K
H.
(8)
Why using impulse filters? Any random convolution filters that satisfy proposition couuld be choice of initialization of attention maps to introduce inductive bias. However, random convolution filters H usually contain both positive and negative values, while the output of the softmax function is always positive, making Eq. (8) unreachable. One straightforward option is to use random positive convolution filters with a normalized sum of one, following the property of softmax. However, this approach often proves inefficient as the patterns are too complicated for a softmax function to handle with QK being of low rank. In (25, the authors found that the softmax attention map serves as a feature selection function, which typically tends to select single related feature. In convolution filters, this selection can be parameterized as impulse filters. According to Corollary 2 random impulse filters are also f2-k spanned. In conclusion, when initializing softmax attention map, the most straightforward and suitable choice random impulse convolution filters.
Pseudo input. The advantage of self- attentttention is that its spatial mixing map is learned from data. The real imput to an attention layer is P + X for the first layer and X (the intermediate output from the previous layer) for the following layers. However, during initialization, there is no prior information about the input. To address this problem, we simply use the initialization of positional encoding P as the pseudo input in the initialization of Q and K, replacing the actual input data P + X or intermediate outputs. Remember that this only happens when we solve the initialization to avoid data-dependent initialization, while in the training stage, we make no change to the ViT architecture.
With the use of impulse filters and the pseudo input, Eq. (8) becomes
M mnit softmax PQQ K Init
PT)
H
impulse
4.2 Solving Qinit and Kinit
There exist numerous approaches to solve Eq. 9 for Qum and iii with known H Hmpulse and P. Here, we apply SVD-based method. First, we change Eq. 9 to exclude the Softmax function as
PQ
T P = aH, init
impulse + +Z
(10)
where Z ~ NO. II). Then to solve for Q and K we put P to the right-hand side of the equation using the pscudo inverse P (p P) -ipce
since P is randomly initialized, it will be
5

---

of full rank. Therefore, we get
K init init = P (aH impuls + BZ)P inv
(11)
With Eq. we do SVD on the right-hand side and get a low-rank approximation of Qini and inti
U.s, V = SVD (Pinv (aH + +Z) )im),
(12)
Qii U UV :d], Kini = VV
(13)
Finally, we do normalization to get the final Qinit and Kni as
Qnii QQiiill ||r
Qinit
Knir
K
(14)
K ||
In VIT models with d 64, we use 3 x 3 convolution filters by setting - We find that the random values in Z are not decisive for our initialization strategy. To ensure a clear impulse structure, the initialization is primarily governed by selecting an appropriate ratio of a to B. Empirically, we choose a large ratio such that a:3 = 40:1 with 7 = 2. Notably, the exact values of a and are not strictly constrained due to the normalization step, which scales these parameters. The pscudo code for our initialization strategy can be found in the appendix.
Experiments and Analysis
In this section, we show that our impulse initialization can improve the performance of ViT on small-scale datasets in Sec. [5.1] while it does not limit the learning flexibility of ViT models on large-scale datasets in Sec. 5.2 In Sec.5.6) we show that even for a pretrained method like weight selection [33], the pretrained model initialized with our impulse structure can provide better pretrained weights with relatively smaller- scalerrr-seeeeeee dataset. Finally, in Sec. 5.4 we show that besides ViT', the concept of introducing architectural bias into initializations can also be generalized to other models like Swin Transformers and MLP-Mixer. Note that all experiments were conducted on a single node with 8 Tesla V100 SXM3 GPUs, each with 32GB memory, if not specified. Specifically, all the experiments on the small-scale datasets took about three hours to train each model, while the experiments on the ImageNet- IK took about two days. Please note that all results were reported based on experiments with retrained models. As result, minor discrepancies may exist between our reported results and those in published papers. However, the key focus of this analysis remains on the improvements achieved through different initialization strategies.
5.1 Small and Medium-Scale Datasets
In this section, we compare the performance of ViT-Tiny under three initialization strategies: the default (31], mimetic 28 and our impulse initialization. Experiments are conducted on medium-
scale datasets (~~50K training images), including Food- 101 [2], CIFAR-10. and CIFAR- 100 [13] as well as small-scale datasets (~55K training images), such as STL-10 10 44, Flowers [19), and Pets [22]. We follow the training recipe from 333, which proven to be useful in training ViT-Tiny on small and medium-scale datasets. Their codes are based on the timm library B1. By default, all the weights are initialized with truncated normal distribution. For fair comparison, all the experiments were run with identical codes except for the choice of initialization methods. Please also note that although [33] initializes the model weights from large pretrained models, we did not adopt any pretraining step for this experiment. In contrast, we apply default, mimetic, and our impulse initialization methods to ViT-Tiny models and training these models from scratch.
The experimental results are presented in Tab. T For simplicity. we keep the statistical results on using different seeds with stochastic filter generations with 5 different runs in Appendix F and Fig. 6 Our method consistently yields substantial improvements over the default initialization across all evaluated datasets. On medium-scale datasets, achieves performance gains of 2%.5%, while on small-scale datasets, improvements can reach approximately 8%, and in some cases, up to 20%. While mimetic initialization also improves the performance, our impulse initialization shows superior efficacy, attributed to the convolutional structure integrated in the attention initialization. Notably, as the dataset size decreases, the performance gap between our method and the default (or mimetic) initialization gets larger. This observation validates that the convolutional inductive bias introduced by our initialization becomes increasingly important when there is less data for the model learn the spatial dependencies.

---

Table 1: Classification accuracy of ViT-Tiny with different initialization methods on different datasets. Green number indicates an increase in accuracy. Note that we compare the performance to the default initialization method (shaded in gray). represents small-scale datasets, and A represents medium-scale datasets. The datasets are ranked based on their training scales.
Method
Datal
Food-101
CIFAR-10
100
STL-10
Flowers
Pets
Default [3]
77.95
92.29
71.67
61,86
64.60
26.58
Mimetic 28
81.78
3,831
93.50
1.21
75.16
3.49|
68.54
6.681
71.62
7.021
47.63
21.051
Ours (impulse)
81.85
3.901
94.67
2,381
77.02
5,351
70.21
8,351
73.18
8,581
50.84
24,261
Table 2: Classification accuracy of ViT-Tiny, ViT-Small and ViT-Base on ImageNet- IK dataset with different initialization methods. indicates large-scale datasets. Please note that for the last column (shaded in yellow), we report the experimental results on ViT-Base with specific settings to make the default initialization ViT comparable to the results reported in concurrent papers.
Model
ViT-Tiny
ViT-Small
ViT-Base
ViT-Base*
Method
Default 31
72.71
79.68
81.24
81.89
Mimetic 28
72.90
0.191
80.26
0.581
80.56
0.68
80.56
1.33| 0.241
Ours (impulse)
72.76
0.051
80.40
0.721
81.83
0.59
82.13
5.2 Large-Scale Datasets
In this section, we compare the performance of ViT-Tiny, ViT-Small, and ViT-base with default, mimetic, and our impulse initialization on a large-scale dataset- ImageNet-IK (over 1M training images). We follow the training recipe from DeiT [26] classic and efficient training recipe for training ViT models on ImageNet-IK. We directly use the original ViT structure and training codes in the timm library, except for adding our implementation of initialization. All the models were trained with the same hyperparameters starting from scratch without any pretraining or distillation.
We show the comparison results on the ImageNet- K dataset in Tab. 2 Detailed training hyperpa- rameters and training curves can be found in Appendix [ In particular, we find that the rapid update for the baseline ViT codebase (i.e., timm library) and the difference in GPU hardware settings result in small discrepancy in the default initialization-basee ViT-Base model between our main result (~81.22 .24) and the results reported in concurrent papers (~88.899). Therefore, we have included an additional column (shaded in yellow) in Tab. Vab.[je [ for ViT-Base" model that uses 16 Tesla V100 GPUs with an extra 0.3 color jittering data augmentation for a clearer comparison.
Despite different training settings and comparisons, our method maintains comparable performance with default initialization, demonstrating that the convolutional inductive bias introduced during initialization does not hinder the model's 's flexibility in learning data-driven dependencies. This indicates that while the transformer architecture begins training with structurally imposed spatial priors, the attention mechanism retains full capacity to learn optimal feature representations when sufficient training data is available. Furthermore, the convolutional structure we introduced in the attention map initialization not only accelerates early convergence but also improves the robustness to variations training hyperparameters, We provide more quantitative results in Appendix [E]
5.3 Training Curves and A Analysis
In Fig. we show the training accuracy curves across 300 epochs for different initialization methods of ViT-Tiny on CIFAR-IO and ViT-Base on ImageNet-1K IK. For the small model on medium-scall dataset, our impulse initialization consistently outperforms the default or mimetic initialization throughout the entire training process. On the large-scale dataset with training large ViT-Base model, our impulse initialization method and the mimetic initialization have shown faster convergence rate than the default initialization at the beginning of the training. However, the mimetic initialization shows degraded performance even to the default initialization at the last 100 training epochs, indi- cating limited ability in large-scale model training. On the contrary, our method does not limit the learning ability of the large-scale ViT models, showing a significant advantage in the final training stage where the performance surpass both the default and the mimetic initialization methods.
7

---

100
Zoumed-in view
Accuracy (%)
Accuracy
200
300
Ours (impalsc)
Ours (imppulnc)
Minotiic
Minntlic
Defont
0
Defunt
Epochs
300
Epochs
300
(a) ViT-Tiny on CIFAR-10
(b) VFT-Base on ImageNet-
Figure 3: Training curves of ViT-Tiny on CIFAR-10 and ViT-Base on ImageNet- using default, mimetic, and impulse initialization. The zoomed-in box shows the training curve in the final training stage from epoch 200 to epoch 300.
Table 3: Classification accuracy of different Swin Transformers and MLP-Mixer different datasets with different initialization methods. Red/green number indicates accuracy decrease/increase. We compare the performance to the default initialization method (shaded in gray).
Model&
Swin-B ImageNet-IK
Swin-T CIFAR-I0
MLP-Mixer
Method
CIFAR-I0
Default
83.14
89.85
87.00
Mimetic 28
83.14
89.24
0.61
Ours (impulse)
83.55 0.411
91.19 1.34
88.78
1.781
In summary, experiments both small-scale and large-scale datasets show that our initialization method effectively balancing prior architectural knowledge integration with data adaptability- incorporating convolutional inductive bias during initialization provides structural guidance to capture dependencies when training data is limited, while still allowing the attention mechanism to retain full learning flexibility in scenarios with abundant data.
5.4
VIT Variants
Beyond the original VIT architectures, we extend simplified version of our initialization strategy to the Swin Transformer. Notably, Swin Transformers incorporate relative positional encoding within each attention block-4 design choice that aligns with our core objective instilling convolutional structure into attention maps. For these models, we achieve this by directly initializing the relative positional embeddings with our impulse pattern.
We evaluated our approach with Swin Transformer-Base (Swin-B) architecture on the ImageNet-IK IK following the training recipe the original Swin Transformer paper [7]. To further demonstrate the flexibility of our approach, we applied our impulse initialization to the MLP-Mixer. For experiments with Swin Transformer- Tiny (Swin-T) and MLP-Mixer, we follow training settings in 34 which is specifically designed for training different models on CIFAR-10.
The results are summarized in Tab. 3 For models with strong learning capacity but limited inher- ent inductive biases at initialization, introducing a structured initialization consistently enhances performance without compromising the model's capacity to learn complex data dependencies. In particular, while Swin Transformer incorporates convolutional inductive bias through windowed self-attention, applying our structured initialization further improves the performance by 1.34% on smaller-scale datasets such as CIFAR-10 10, with no degradation on large-scale datasets like ImageNet- IK. MLP-Mixer, which replaces the spatial convolutions in ConvMixer with MLP layers, typically struggles to train effectively on small-scale datasets. However, initializing the spatial MLPs with a convolutional structure leads to a 1. 1.78% performance gain on CIFAR-10. In contrast, mimetic initialization iriialization-deiigneeed to replicate empirical weight distributions from pretrained negligible benefits or degrades performance, highlighting limited generalizability outside the specific pretrained ViT structures.
8

---

Ours
(impulsc)
Head
Ours (impulse)
Head?
Mimetic
All heads
Default
Layer
Layer 6
Layer 12
Figure 4: Visualization of attention maps in VIT-T using ours, mimetic [28] and default 33 initializations. Red boxes highlight zoomed-in details of the 16 x 16 upper left corner in attention maps. White boxes indicate the 8 x bub-blocks of the zoomed-in attention maps. Our structured initialization method offers distinct attention peaks aligned with the impulse structures across different heads. Head 1 offers peak +1 offset from the main diagonal. Head 2 offers peek -8 offset (equivalent to - --maemaasaiaa size) from the main diagonal. Both mimetic and random initialization methods initialize all the attention heads identically. Specifically, mimetic initialization primarily strengthens the main diagonal of the attention map for each head, while random initialization shows minimal structural patterns with flatter peak values.
5.5 Attention Maps
The initialized attention maps produced by default, mimetic, and our impulse initializations are shown in Fig- 4 For better visualization, we use 32 CIFAR-10 images as input with a 4 4 patch size, yielding the token size as 8x8-64. observe that the default initialization generates near-identical attention values with little distinguishable spatial structure, while the mimetic initialization introduces diagonally dominant values by adding an identity matrix during initialization. Notably, these two initialization methods initialize all the attention heads in the same way. In contrast, initialization assigns different impulse patterns to each attention head, producing spatially diverse activations with off off-diagonal peaks specific impulse positions.
5.6 Pre-trained Model
In this section, we demonstrate that our initialization method remains compatible with existing pretraining pipelines. Building on the work of Xu al. [33), who showed that weight selection from K
ViT-Small models effectively initializes ViT-Tiny architectures, we extend this approach to ImageNet-IK pertaining. We pretrained three ViT-Small models on ImageNet-IK using three initialization strategies: default, mimetic, and our proposed impulse initialization. From these, we derived three ViT-Tiny initialization variants pariants-trmmed " -edauullt"" "1K-mimetie and " "'K-impulse" "aallsse""sssi using the same weight selection methodology on smaller-scale datasets. When training these initialized ViT-Tiny models on small and medium-scale datasets (results shown in Tab. 4. we observed that: (1) Switching from ImageNet-21K to ImageNet- - lesls training data patta-petrainnng with default initialization typically incurs a performance drop of ~ %. (2) Mimetic initialization fails to mitigate this data degradation, (3) The ImageNet- IK pretraining model with impulse- impulse-based

---

Table 4: Classification accuracy of ViT-Tiny on small-scale datasets with the weight selection method. Here, the weights selected for the experiments were pretrained on the ImageNet- IK (shortened as 1K, shown above the purple dashed line) and ImageNet-21K (shortened as 21K, shown below the purple dashed líne) datasets with different initialization methods. Please note that pretraining on ImageNet-21 with default initialization is the original weight selection method [33] We compare
the performance to the default initialization method pretrained on ImageNet- K (shaded in gray).
Datal
Food-101
CIFAR-10
CIFAR-100
STL-10
Method+MMdet-
DK+Def (33
85.42
96.61
79.64
82.58
Miimeete (28
86.32
0.901
96.37
0.24|
79.86
0.221
82.39
0.191
IK+Ours (impulsc)
87.43
2.011
97.19
0,581
80.92
1.28
83.89
1.31
21K+Default
33
87.14
1.721
97.07
0.461
81.07
1.431
83.23
0.651
initialization achieves performance parity with MMageNet-211K pretrained baselines. This highlights the robustness of our method even under reduced pretraining data regimes.
Limitations and Broader Impacts
There exist several limitations in our initialization method: (1) Positional encoding. Although to its simplicity and
positional encoding is a natural and effective choice for pseudo-inputs- data-independent nature- even simpler alternatives may exist for initializing the Q and K matrices. In this work, we focus solely on the initialization of Q and K, but a more comprehensive initialization strategy that also considers patch embeddings and positional encodings could offer greater control over the structure of attention maps. Notably, since positional encoding is only added at the input attention maps may diminish with increasing network depth, as illustrated in Fig. 4 (2) Hard constraints. Our Corollary of Proposition i hased on the presumption that the filters are f2-k spanned, which is usually characteristic inherent in CNNs. However, in ViTs, the limited number of heads may be inadequate to span the filter space of a small kernel. Finding better adaptations in this scenario remains a challenge. (3) Value initialization. Our method does not consider the initialization for the value weights V and the projection matrix.
layer, the influence
Broader impacts. This work advances the understanding of how structured initialization influences the performance of transformers, particularly in nooource-constrinne settings. Our research findings enables more efficient training of neural networks on small-scale datasets, which may benefit domains such as medical imaging, environmental monitoring, robotics, or education, where data is limited or expensive to collect. Furthermore, improving initialization strategies reduce computational costs, contributing to more sustainable Al practices.
However, as with most advances in artificial intelligence, these techniques carry risk of misuse or harmful societal applications. For example, by applying our initialization method to more powerful models with fewer resources could make larger models more easily accessible for malicious purposes. These concerns further remind us to carefully consider the broader societal impacts of our research and make sure its benefits outweigh potential harms.
7 Conclusion
In this paper, we propose structured initialization method with convolutional impulse filters for attention maps in VITs. Our method preserves both the advantage of locality within CNNs and the capacity to learn long-range dependencies inherited from ViTs. We also provide a thorough theoretical explanation of the spatial and channel mixing ConvMixer and ViT, building connections between the structural bias in CNNs and the initialization of ViTs. Our results on small-scale datasets validate the effecctivnness of the convolutional structural bias, while on-par performance on large-scale datasets indicates the preservation of architectural flexibility. Our initialization also accelerates early-stage convergence but also enhances the model's robustness to variations training hyperparameters. Furthermore, we demonstrate that our method consistently provides benefits across a range of architectures and even under pre-training strategies.
10

---

References
[1] Andreoli, J.M.: Convolution, attention and structure embedding. arXiv preprint arXiv: 190.01905 (2019)
[2] Bossard, I.,, Guillaumin, M., Van Gool, L.: Food-101-mining discriminative components with random
forests. In: Computer vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part VT 13. pp. 446-461. Springer (2014)
[3] Cazenavette, G., Julin, J., Lucey, S.: Rethinking the role of spatial mixing, arXiv preprint arXiv:2503.16760 (2025)
[4] Coates, A., Ng. A.. Lee. H.: An analysis of single-layer networks in unsupervised feature learning. In: Proceedings of the fourteenth international conference artificial intelligence and statisties, pp. 215-223. JMLR Workshop and Conference Proceedings (2011)
[5] Cordonnier, J.B., Loukas, A., Jaggi, M.: On the relationship hetween self-attention and convolutional layers. In: Fighth International Conference on Learning Representations-ICI.. 2020 (2020)
[6] Cubuk, E.D., Zoph, B., Shlens, J., Le, Q V.; Randaugment: Practical automated data augmentation with reduced search space. In: Proceedings of the IEEE/CVF conference computer vision and pattem recognition workshops. pp. 702- 703 (2020)
[7] Dai, Z., Liu, H., Le, Q. Tan, M.: Coatnet: Marrying convolution and attention for all data sizes. Advances in neural information processing systems 34, 3965-3977 (2021)
[8] Dosovitskiy, A., Beyer, I.., Kolesnikow, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer. M.. Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: Transformers for image recognition at scale. ICLR (2021)
[9] d' Asccoli, S., Touvron, H., Leavitt, M.L., Morcos, A.S., Biroli, G., Sagun, L.: Convit: Improving vision transformers with soft convolutional inductive biases. In: International Conference on Machine Learning. pp. 2286-2296. PMLR (2021)
[10] Feng, R., Zheng, K., Huang, Y., Zhao, Jordan, M., Zha, Z.J.; Rank diminishing in deep neural networks. Advances in Neural Information Processing Systems 35, 33054-33065 (2022)
[11] He. K., Zhang. X., Ren, S., Sun, Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778 (2016)
[12] Ji, Y., Saratchandran, H., Moghaddam, P. Lucey, S.: Always skip attention. arXiv preprint arXiv:250.00199 (2025)
[13] Krizhevsky, A., Hinton, G., cl.: Learning multiple layers of features from tiny images (2009)
[14] Li, A., Tian, Y., Chen, B., Pathak, D., Chen, X.: On the surprising effectiveness of attention transfer for vision transformers. Advances in Neural Information Processing Systems 37, 113963-1 113990 (2024)
[15] Li, K., Wang, Y., Zhang. J., Gao, P., Song. G., Liu, Y., Li, H., Qiao, Y:: Uniformer: Unifying convolution and otlf-attention for visual recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence (2023)
[16] Liu, Y., Sangincto, E., Bi, W., Sebe, N., Lepri, B., Nadai, M.: Efficient training of visual transformers with small datasets. Advances in Neural Information Processing Systems 34, 23818 2--83303000 (2021)
[17] Liu, 7., Lin, Y. Cao, Y., Hu, H., Wei, Y., Zhang, 7., Lin, S., Guo, B.: Swin transformer: Hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 10012-10022 (2021)
[18] Naderi, A., Saada, T.N., Tanner, J.: Mind the gap: a spectral analysis of rank collapse and signal propagation in transformers. arXiv preprint arXiv:2410.07799 (2024)
[19] Nilsback, M.E., Zisserman, A.: Automated flower classification over a large number of classes. In: Indian Conference Computer Vision, Graphics and Image Processing (Dec 2008)
[20] Noci, L., Anagnostidis, S.. Biggio, L., Orvieto, A., Singh, S.P., Lucchi, A.: Signal propagation in transformers: Theoretical perspectives and the role of rank collapse. Advances in Neural Information Processing Systems 35, 27 1981112721 (2022)
[21] Pan, X., Ge, C., Lu, R., Song, S., Chen, G., Huang, Z., Huang, G.: On the integration of self attention and convolution (2021)

---

[22] Parkhi, O.M., Vedaldi, A., Zisserman. A., Jawahar, C.V.: Cats and dogs. In: IEEE Conference on Computer Vision and Pattern Recognition (2012)
[23] Samragh, M., Farajtabar, M., Mehta, S., Vemulapalli, R., Faghri, F. Naik, D., Tuzel, O., Rastegari, M.: Weight subcloning: direct initialization of transformers using larger pretrained ones, arXiv preprint arXiv:2312.09299 (2023)
[24] Samragh, M., Mirzadeh, I,, Vahid, K.A., Faghri, F., Cho, M., Nabi, M., Naik, D., Farajtabar, M.: Scaling smart: Accelerating large language model pre-tree pre-training with small model initialization. arXiv preprint MrXiv:2409.1299 (2024)
[25] Tarzanagh, D.A., Li, Y., Thrampoulidis. Oymak, S.: Transformers as support vector machines. In: NeurlPS 2023 Workshop on Mathematics of Modern Machine Learning (2023)
[26] Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., Jégou, H.: Training data-efficient image transformers & distillation through attention. In: International conference on machine learning. pp. 10347- 10357. PMLR (2021)
[27] Trockman, A., Kolter, J.Z.: Patches are all you need? arXiv preprint arXiv:2201.097) (2022)
[28] Trockman, A., Kolter, J.Z.: Mimetic initialization of self-attention layers. International Conference on Machine Learning. pp. 14456-34468. PMLR (2023)
[29] Trockman, A., Willmott, D., Kolter, J.Z.: Understanding the covariance structure of convolutional filters. arXiv preprint arXiv:2210.03651 (2022)
[30] Vaswani, A., Shazeer. N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L.., Polosukhin, Attention is all you need. Advances in neural information processing systems 30 (2017)
[31] Wightman, R.: Pytorch image models. (2019). https://doi.orgg org/ 1 /zeemoo..4414500
[32] Wu, H., Xiao, B., Codella, N., Liu, M., Dai, X., Yuan, L., Zhang, L.: Cvt: Introducing convolutions to vision transformers. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 22-31 (2021)
[33] Xu, Z., Chen, Y., Vishniakov. K., Yin, Y., Shen, Z., Darrell, T., Liu, L.,, Liu, Z.: Initializing models with larger ones. In: International Conference on Learning Representations (ICLR) (2024)
[34] Yoshioka, K.: vision-transffrmmen-cccfaalll Training vision transformers (vit) and related models on cifar- 10.
wrw.:oonnntrraafffmmmmemm cifarl0 (2024)
[35] Yuan, K., Guo, S., Liu, Z., Zhou, A., Yu, F., Wu, W.: Incorporating convolution designs into visual transformers. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 579- 588 (2021)
[36] Yun, S., Han, D., Oh, S.J., Chun, S., Choe, J., Yoo, Y: Cutmix: Regularization strategy to train strong classifiers with localizable features. In: Proceedings of the IEEE/CVF international conference computer vision. pp. 6023-6032 (2019)
[37] Zhang, Y., Backurs, A., Bubeck, S., Eldan, R., Gunasekar, S., Wagner, T.: Unveiling transformers with lego: a synthetic reasoning task. arXiv preprint arXiv:2206.0430) (2022)
12

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We have clearly summarized the contributions of our paper at the end of the introduction section. The theoretical and experimental results shown in the paper accurately reflect the contributions of our proposed method.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We have clearly stated the limitations of our proposed method and the broader impact in the main paper.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
13

---

Answer: [Yes]
Justification: We have provided the complete, correct, detailed proof and the full set of assumptions for every theoretical result in both the main paper and the appendix. And have numbered and cross cross-referenced all the theorems, formulas, and proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
We have provided sufficient experiments and the detailed initialization strate- gies to validate the effectiveness of our proposed method in the main paper and the appendix. We have also released the full code to the public.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
14

---

reproducible, they 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: All the data used in the paper are open-sourced datasets. We have provided the implementation details and have released the code to reproduce the results claimed in our paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cem
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ /twps.cccppbllc//uuiiiiie Swwelee/ddoeSSedeiieeee cettsummiissooonll wsiinnoliccyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have provided all the training and testing details, including datasets, hyperparameters, hardware settings etc. in the main paper and the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [NA]
Justification: We have reported the error bars for the main results in the appendix. For larger model training, we skip the error bar report due to the expensive computation. Nevertheless, we have set the experimental seed for all the experiments and have ensured accurate and the same experimental settings and hyperparameters for different methods the same experiment. We have also provided the detailed implementation strategies and have released the full code for reproducing the main results in the paper.
Guidelines:
15

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi-
dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative етror rates).
If error bars are reported in tables or plots. The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have provided the computing resources used in the experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: We confirm that our research, in every respect, conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
16

---

Justification:
We have discussed both potential positive and negative societal impacts in the broader impact section in the main paper.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-us dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All the datasets, codes, and models used in the paper are open- And we have clearly referenced these assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
17

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperssi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We did not introduce any new assets in the paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our research does not involve any crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our research does not involve any crowdsourcing or research with human subjects.
18

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does involve LLMs as any important, original, or non standardd components.
Please refer to our LLM policy for what should or should not be described.
19

---

Appendix: Structured Initialization for Vision Transformers
A Frequently Asked Questions
Q: Does introducing CNN structural biases to Transformer contradict the advantage of its structure?
A: Our method focuses on introducing the architectural inductive bias from CNNs to initialize the attention map without changing the Transformer architectures. We have emphasized this argument in the abstract and introduction, stating that unlike previous arts [35] 15| 7 32 [6] that directly introduce convolutions into attention, potentially damaging the structural advantages of Transformers, our method only introduces architectural inductive bias in attention map initialization, maintains the inherent structural flexibility in ViTs. Therefore, our method preserves the Transformer architectures and still allows the ViTs to learn flexible, dynamic global relationships. This is also one of the central innovations of our method. In addition, we have also provided experiments on large-scale applications to validate the stable performance of our method that preserves the architectural flexibility of ViTs.
Q: Why is the proposed method better than transfer learning on large-scale pretrained models or a hybrid architecture combining CNN and attention?
A: As stated in Sec. I we would like to emphasize the advantages of our proposed structured initialization method: 1) Unlike previous methods that do transfer learning on large-scale pre-trained ViTs, our method involves no pre-training of large-scale models on large-scale datasets, which may not be readily available; 2) Our method shares the advantages of both CNNs and ViTs.
In general, introducing inductive bias in CNNs for initializing the attention map helps ViTs begin learning from a more reasonable/stable starting point, while the default random initialization can lead to noisy starting point, especially when training on small-scale datasets. Notably, our initialization method does not alter the ViT structure, maintaining the advantages of the Transformer architectures in learning dynamic, long-dependent global features. contrast, methods that directly combine the architectures of CNNs and ViTs alter the Transformer architectures, potentially compromising its architectural advantages. Please also refer to the theoretical analysis of random filters in Sec. and the convolutional representation matrix in Appendix C of the main paper for more theoretical explanations.
For pretrained models, we noted in the introduction section that their reliance on access to pretrained models makes them impractical for domain- sppcific scenarios. For example, pretraining a large model on extensive datasets only to deploy a smaller model on limited data is inefficient and often unrea- sonable. Furthermore, their effectiveness heavily depends on the performance of the specific model architectures, offering no inherent advantages from using pretrained weights. Nevertheless, even on pretrained models, our initialization still outperforms other methods. For additional quantitative evidence, please refer to the experimental analysis in Sec. 5.1 and Sec. 5.6
Q: Why use an impulse filter? Does enforcing strict structure in initialization degrade performance?
A: While one might expect that a rigid initialization could limit the model's flexibility, our exper- imental results in Sec. demonstrate that the impulse structure does not hurt the training. On the contrary, it consistently improves the performance, particularly on small and medium-scale datasets. This is attributed to the structured initialization introducing beneficial inductive bias, which guides the model toward learning useful representations in the early stage of training.
Moreover, the hyperparameters a, B, and y control the norms of Q and K, ensuring that the attention maps exhibit a well-defined convolutional structure at initialization, while maintaining sufficiently flexible adapt and learn from data during training. This design shows a central innovation of our method, and its effectiveness is consistently supported by experiments across both small and large-scale datasets.
Although, as suggested by Proposition D any set of random convolutional filters can be used to initialize attention maps, it is difficult to obtain both clear attention map structure and sufficient
20

---

training flexibility from random non-impulse filters, especially when analyzed through low-rank approximation via singular value decomposition (SVD) and the Softmax operation. Thus, when aiming to initialize attention maps with convolutional prior, impulse filters remain the most straightforward and robust choice.
B Proof for Proposition
Let w = [1. u...... wDl" E Dx be the channel mixing weights for one output channel and H1, H,,,..,, HD are the corresponding spatial convolution filters for each channel. Therefore, the result E R after spatial and channel mixing can be represented as,
Σ W H
(15)
With Remark !
we can suppose the rank of X RJ ZA is k, where [Z,...., 2x] and h < D, as We then obtain
illustrated in Fig. 2
D
&
w H.z = Σ H2Zz.
y ~
(16)
j=1
where azs refers to the row j. column i element of A, and H, Σ Wi 0zj H,
Remember that a linear combination of f linearly independent filters h can express any arbitrary filter in filter space R/x,, where h serves as the basis. Consequently, any desired H,, H H be achieved by only learning the channel mixing weights w. Therefore, obtain the following proposition.
C Convolutional Represetation Matrix
In Sec. [] we interchangeably use the terms convolution filter h and convolution matrix H. Addition- ally, we represent the impulse filter as a convolutional matrix. Here, we offer a detailed explanation of the relationship between the convolutional filters and the convolutional matrices.
Let us define 2D convolution filter as h E RRx with elements
h1
hy
h
(17)
hy1
hyf
When h convolved with an image x e R HxW this convolution operation is equivalent to matrix multiplication
vee(h x) H vec(x),
(18)
where is composed from the elements in h and zeros in the following format:
Fi
F
Fr
0
0
0
F
F
2
Fr
0
0
H
(19)
0
F1
F2
Fr
0
0
0
F1
F2
Fr
where
hai
h22
bif
0
0
0
ha
h
hat
0
F;
(20)
0
h41
h.2
baf
h.2
h4g
1ri==1,,..... Fis are circulant matrices and H is block circulant matrix with circulant block (BCCB). Note that convolutions may employ various padding strategies, but the circulant structure remains consistent. Here, we show the convolution matrix without any padding example.
21

---

Table 5: Ablation study on training settings of ViT-Base on ImageNet- IK dataset. Note that we compare the performance to the default initialization method (shaded in gray).
Scaled LR
Repeated Augment
Default
Mimetic
Ours (impulsc)
7c-4
1.0
76.25
79.12
2.871
80.48
4.231
le-3
1.0
80.09
80.17
0.081
81.55
1.071
le-3
3.0
.1.24
80.56
0.68
81.83
0.591
D
Pseudo Code for Solving Qinit and Kinit
Algorithm I Convolutional Structured Impulse Initialization for ViT
Input: P
Input positional encoding D Hyperparameters Initialized attention parameters
Input: d, ox 3,
Output: Qinit; Kinit
1: N, D + shape(P)
2: Himpulse IImpulsCCnn,Marii((,,,
Build 2D impulse convolution matrix
3: M + a-Hampulse + ,3
>> Get ideal map
4: X LayerNorm(2 (P)
Get pseudo input
5: PIny +- (X X) X
b. Get pscudo inverse of P Get patterns QK? SVD of M
6: M + P PIIMP T
inv
7: U,S, V SVD(M)
8: Qinit + Uvs, Kini +- Vvs
9: Quii + Qmiil:: d], Kini t- Kinit;;: d]
Kinit +- Kint/norm(K /nom/oommm 11: Qinit Qinit: Kinit <- 7Knn
10: Qinin Qini//nrrm(
12: return Qinit, Kinit
E Training Details and Training Curves for ViT-Base
At first, we found it challenging to reproduce the ViT-Base performance reported in DeiT [26]. even when strictly following their specified training settings. Upon a careful examination of code differences across various versions of the timm [3 library, we identified two critical discrepancies that likely contributed to the performance gap: (a) Learning rate scaling strategy: In DeiT, the learning rate is linearly scaled with respect to the batch size: Irscated = Irhuse batch size In contrast, the current version of timm uses a square root scaling rule as the default for the Adam optimizer:
512
batch This discrepancy the default setting leads to different effective learning Insaated = Iruuse x V
512
rates, even when all the other hyperparameters are identical, and can substantially affect performance. (b) Repeated data augmentation setting: DeiT emphasizes the importance of the repeated data augmentation strategy, stating drop in top-1 accuracy from 81.8% to 76.5% when it is disabled. However, they did not not specify the exact augmentation weighting value in their original paper. After inspecting their specific version of code, we discovered that the default value for the repeated data augmentation was set to 3, whereas we used in our main experiments, which may partially explain the performance discrepancy. We also aligned several other minor settings, such the minimum learning rate during warm-up and at the end of training, as well as an additional 10 epochs for cool- down. However, we believe that these factors have only marginal effect on the final performance, while the two aforementioned reasons remain the primary contributors to the observed discrepancy.
We present an ablation study these two main settings difference in Tab. In Fig. S we also present the training accuracy curves across epochs for different initialization methods with three different training configurations. Our impulse initialization consistently outperforms the default initialization throughout the entire training process, with particularly significant advantage the final training stage. While the mimetic initialization shows relatively faster initial convergence,
22

---

80
Zaumer-in view
Taomed-in view
Zaomed-in view
82
Accuracy
80
Accuracy
A Accarscy
75-
200
300
200
300
200
300
Ours (impalse)
Ours (impplise)
Ours (impulee)
Mimetic
Mimetic
Mimetic
Defanl:
Defauit
Defanlt
Epochs
300
Epochs
300
Epochs 300
(a) Scaled LR: 7e-4
(b) Scaled LR: 1e-3
(c) Scaled LR: 1e-3
Repeated Augementation: 1.0
Repeated Augmentation: 1.0
Repeated Augmentation: 3.0
Figure 5: Training curves of ViT-Base using default, mimetic, and impulse initialization under three different training configurations. The zoomed-in box shows the training curve in the final training stage from epoch 200 to epoch 300.
Table 6: Detailed scale of all the datasets used in the paper. represents large-scale datasets, represents medium-scale datasets, and represents small-scale datasets. The datasets are ranked based on their scales following [3]. "+' means we use both train and validation dataset in training. Note for Flowers and Pets datasets, we use both training and validation data for training.
#Train images in each class
#Train images in total
#Images total
Dataset
#Classes
ImageNet-IK
1K
IK
1.3M
1.4M
Food-10
101
750
75K
101K
CIFAR-10
10
5K
50K
60K
CIFAR-100
100
500
50K
60K
STL-10
10
500
5K
13K
Flowers
102
10+10
2K
8K
Pets
50+50
3.7K
7.3K
ultimately degrades performance in the final training stage. Our structured initialization method demonstrates robustness across different training configurations, consistently yielding over 80%
accuracy in all cases.
Additional Results for Small and Medium-Scale Datasets
F.1
Dataset Scale
For completeness, we provide the detailed dataset scales used the main paper in Tab. ordering of the datasets follows that in 33
The
F.2 Additional Statistical Results
Here we provided the mean and standard deviation of runs ViT-Tiny with different initialization methods on small and medium-scale datasets in Fig. G which serves as additional statistical results of Tab. I in the main paper. Notably, our initialization method still outperforms other methods across small and medium-see scall datasets. Especially for the smaller-scale datasets, our methods shows larger performance improvements, aligning with our findings in the main paper.
Model Convergence Rate
We would like to clarify that while impulse filters mimic convolutional locality, they do not reuse weights like CNNs do-which may explain the fast convergence CNNs. Interestingly, we did observe a faster convergence when using the mimetic impulse initialization to replace the vanilla ViT model. We give an example of the training accuracy of ViT-Tiny with different initialization methods CIFAR-10 during the optimization stage, shown in Tab. 7 We can clearly faster
23

---

100
92.27 15.111a0 14.60
Default Mimetic
90
82.1181.64
Ours (impulse)
77.93
76.50
74.59
72.00
71.84
70.24
Aceuaay (%)
67 .22
64.30
62.26
60
48.85
50
46.82
40
28.71
30
Food-101
CIFAR-10
CIFAR-100
STL-10
Flowers
Pets
Figure 6: Classification accuracy of ViT-Tiny with different initialization methods different datasets (mean of 5 runs). Our method consistently outperforms the default and the mimetic initializations. Except for the smallest-scale dataset Pets, all the methods show robust performance across different datasets with small error bars. Note that from left to right, the dataset scale decreases.
Table Comparison performance when training ViT-Tiny on CIFAR-IO at different epochs. Our method consistently achieves higher accuracy than both default and mimetic initialization baselines.
Epoch
Default
Mimetic
Ours (Impulse)
10
47.95
49.05
54,62
50
61.51
63.62
69.51
100
76.87
77.64
81.87
convergence of mimetic and our impulse initialization methods compared to the default initialization used in vanilla ViTs. One intuitive explanation is that the mimetic initialization yields ~4x the norm values of the default initialization, while our initialization method has ~2x the norm values of the default initialization. Different norm values will affect the gradient magnitudes and early training dynamics.
E.4 Performance Improvement Increases the Model Size Increases.
In the main paper Tab. we also observe that as the model scale increase (i.e., the number of heads increases), method becomes relatively more expressive. Theoretically, explained in Sec. 31 to replace the spatial kernels with fixed filters, the initialized attention heads must span the spatial filter basis. Therefore, for 3x3 filter used in the experiments, this requires at least 9 independent heads to span the kernel space. Please note that in Cinerixer, each channel uses an independent filter, which satisfies this criterion. However, VIT models share the same filters within each head, which need a separate discussion: 1) ViT-Tiny has 3 heads, ViT-Small has 6 heads-inees insufficient to span to the kernel space; 2) ViT-Base has 12 heads- for replacing the spatial kernels. This analysis fully supports this performance improvement increase observation. However, as the motivation of the paper also suggested, training even larger ViT models on small datasets notoriously challenging due to optimization instability and out of the scope of this paper.
G Experimental Validation of the Propositions and Corollaries in Sec.
In this section, we conduct a series of preliminary experiments on CIFAR-10 validate the propo- sitions and corollaries in Sec. of the main paper, with particular focus demonstrating that in

---

Table Classification accuracy(%) ConvMixer (depth 8) with different filter sizes, embedding dimensions on CIFAR-10.
Kernel Size
Embedding Dimension 256
Embedding Dimension 512
Trained
Random
Impulsc
Box
Trained
Random
Impulsc
Box
91.76
90.72
90.68
81.70
92.82
92.15
92.20
81.90
92.69
90.87
90.41
80.57
93.90
92.72
91.91
81.19
8
92.34
88.12
87.82
78.95
92.96
90.09
89.61
80.10
ConvMixer, fixed random impulse spatial filters can achieve comparable performance learned filters. All experiments adhere to the training protocol proposed in [34], which is specifically tailored for evaluating diverse architectures on small-scale datasets such as CIFAR-10.
To support our theoretical findings in Sec. concerning the effectiveness train ConvMixer [27] models with embedding dimension of 256, a depth of 8, and patch size of 2 on the CIFAR-I0 dataset, using spatial filter sizes of 3, 5, and 8. We also include a variant with embedding dimension of 512 to examine the impact of feature width. We evaluate the end-to-end trained ConvMixe alongside three initialization strategies: random (Corollary [ impulse (Corollary 2). and box (Corollary 3. Note that in all three cases, only the spatial convolution filters are initialized the models are evaluated without any training. The box filters use all-one values, effectively performing average pooling. The results summarized in Tab.
random filters, we
In conclusion, these experimental results reveal several key insights. First, comparing across columns (L.e., initialization methods), both random and impulse initializations achieve performance comparable (290%) to that of fully trained models, whereas the box initialization leads to significantly worse performance (~80%). This discrepancy be attributed to the deficient rank of the box filters, which fail to span the full fa-dimensional filter space, unlike random and impulse filters that are capable of forming complete basis.
Second, when comparing across rows (i.e., kernel sizes) with an embedding dimension of 256, the performance gap between trained and untrained (random or impulse) filters grows with kernel size, from 1% (size 3) to 2% (size 5) and 5% (size 8). This occurs because larger kernels require more distinct filters effectively span the filter space. However, the fixed embedding dimension constrains the number of such filters, reducing their ability to match the input rank as shown in Proposition I Notably, when the embedding dimension is doubled 512, this performance gap narrows. In particular, for a kernel size of 3, the random and impulse initializations nearly match the performance of the trained filters, suggesting that sufficient embedding width compensates for the limitations of fixed filters.
H
Attention Maps
Here we provide additional visualization of the attention maps for all 12 layers Fig. 7 In particular, our structured initialization method offers different attention peaks on various heads, showing alignment with the impulse structures, while the mimetic initialization only presents main-diagonal peaks, and the default initialization shows little to no patterns. As stated in the main paper, both mimetic and default initialization methods use identical initialization for all attention heads.
25

---

Ours (impulse)
Ours (impulse)
Mimetic
Default
Head
Head
L2
L4
L6
18
LIO
L12
Full
Detail
Full
Detail
Full
Detail
Full
Detail
Figure 7: Visualization of attention maps in ViT-T using our impulse initialization method, mimetic (28 and default 31 initializations.
26


Reviews:
[{"review_id": "etupFiHeoz", "reviewer": "Reviewer_7orw", "summary": "The paper introduces a new initialisation method for non-CNN architectures, based on CNN like filters. They use impulse filters for this and provide theoretical justification. The initialization strategy differs from prior work by initializing different attention heads with different filters. On small-scale datasets, this initialization improves performance significantly, similar to architectural inductive biases. However, unlike architectural inductive biases, they don't limit scalability on larger scale datasets (up to IN-1K).", "questions": "- Are there downsides to the structured initialisation?\n- Would any benefit of initialisation for fine-tuning a pre-trained model remain when pre-training is scaled up?\n- The paper claims better robustness to hyperparameter selection, but doesn't show it in the main paper. Maybe it's better to include it in the paper as it's part of the main message (including the conclusion)?\n- Can we be sure that the initialization doesn't limit performance compared to random initialization when training on even larger scale datasets than IN1K? Or different types of training, such as self-supervised objectives. I think it's likely, but it's not sure.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The paper is clearly written.\n- Inducing inductive bias through initialization rather than the architecture is an interesting and promising direction, as it benefits from the best of both worlds: good on small-scale without limiting large-scale.\n- The paper finds theoretical justification for findings in recent work that show for ConvMixer like architecture, the spatial mixing doesn't have to be trained to achieve good performance.\n- The paper shows their initialization even outperforms prior weight selection methods which use pre-trained models.\n- The pretraining results are promising, highlighting that the benefit of the initialization remains even after pre-training.\n\nWeaknesses:\n- Though the paper shows they can improve ViT on small-scale datasets, they don't provide a comparison with ConvMixer (or similar). Thus it's unclear whether their method makes ViT actually useful on a small-scale, as one otherwise might still better use a CNN.\n- DeiT-B 224 without distillation shows 81.4 accuracy on ImageNet-1K. In your reproduction it's 76.25. What explains this difference? It's a bit concerning to me, as there might be a problem in the training pipeline, which might change the conclusion, though unlikely.\n- The paper argues this initialization is useful in domain-specific use-cases, as there limited data is available. However, this is never verified. Though experiments are done on small scale generic datasets, no pre-training is used there (except for one table with IN1K pre-training, which is arguable too limited). Therefore, it seems to me important, that it is verified that current off-the-shelf large-scale pre-trainings are indeed less effective on domain-specific data (which is dissimilar from generic pre-training datasets), and structured initialisation is more effective. I do understand this might be challenging and don't think this is crucial for this paper to be accepted.", "was_revised": true, "final_justification": "I will maintain my score of accept because the paper is technically solid, introduces a novel initialization method with strong empirical results, and provides valuable insights into the role and impact of initialization. While the practical utility of the approach remains debatable, the contribution is likely relevant and informative for the community.", "rebuttal": "We sincerely thank the reviewer for the constructive and thoughtful comments. We are happy to know that our paper is well written, and the initialization idea is interesting, and has great theoretical and quantitative justifications. We will address each concern in detail.\n\n## **Weaknesses:** ##\n\n### **1. Comparison with CNNs (** ***e.g.*** **, ConvMixer) on small-scale datasets**\n\nWe would like to clarify that it is difficult to give a definite and general comparison over whether CNNs or Transformers are better in performance because of different model architectures, training configurations, application scenarios, *etc*. However, to give an intuitive comparison and to validate the effectiveness of our method in aiding training ViTs on smaller-scale datasets, we did a simple experiment to compare ConvMixer and ViT. Specifically, we design 1) ConvMixer with depth=8, embedding dimension=256, and convolutional filter size=3; 2) a ViT with depth=8, embedding dimension=256, and number of heads=16. We show the accuracy on CIFAR-10 below:\n\n| Embedding Dimension | ConvMixer (trained) | ViT (default) | ViT (mimetic) | ViT (impulse) |\n| :---: | :----: | :----: | :----: | :---: | \n| $256$ | $\\mathbf{91.76}$ | $\\underline{84.62}$ | $88.97$ | $\\mathbf{\\underline{90.59}}$ |\n\nIn this table, we show that the baseline ConvMixer with trained filters achieves the best result. When trained with vanilla ViT with default initialization, the performance drops **\\~7\\%**. Using mimetic initialization improves the accuracy by **\\~4.4\\%**, while using our impulse initialization method significantly improves the accuracy by **\\~6\\%**, making the performance of ViT (**90.59**) comparable with ConvMixer (**91.76**).\nPlease note that for a fair comparison, we force the ViT to have the same depth, size, and embedding dimension. However, in practice, ViTs have different model configurations. For example, a ViT-Tiny model typically has 3 heads, 12 layers with an embedding dimension of 192.\n\nNonetheless, this table demonstrates the effectiveness of our structured initialization in introducing CNN inductive bias into the ViTs and improving the performance under the limited data settings, making the accuracy comparable to ConvMixer. Meanwhile, having such a ViT structure unchanged enables the scalability and flexibility to learn more complex features. We will update these intuitive results to the revised version.\n\n### **2. ImageNet-1K performance discrepancy explanation**\n\nWe kindly point the reviewer to **Appendix Section F** for a detailed explanation of this performance gap. To summarize, we have not only provided a detailed explanation, but also provided two different experiments in **Table 5** and **Figure 4** for comparisons of using different training configurations. Both intuitive and quantitative experiments validate that no matter how the training configurations change, our method maintains the best performance with the biggest performance improvements over the vanilla ViT. These results indicate the effectiveness and the robustness of our method under different training configurations. We will update the main paper with the updated results reported in the Appendix.\n\n### **3. Verifying the effectiveness of the proposed method in domain-specific use cases**\n\nWe would like to clarify that we have already verified that using our initialization method is effective in domain-specific use cases where only limited data is present. Our paper focuses on showing that integrating the domain-specific prior knowledge, *i.e.*, CNN locality,—which is important in visual feature learning—into the large vision model, *i.e.*, ViTs as a structured attention map during initialization is effective through extensive experiments on different limited data settings. Similar to visual priors, we argue that other domain-specific knowledge can also be integrated into the model architectures with the same analogy to aid training such a model with limited data.\nIn addition, we have already shown in **Section 5.5** and **Table 4** that applying our initialization method in the pretraining stage can generate better pretrained weights for models that rely on pretraining on large-scale datasets (weight selection model [33]), making the performance comparable to such model that pretrained on even larger-scale datasets such as ImageNet-21K.\n\n## **Questions:** ##\n\n### **1. Downsides of the proposed method**\n\nWe point out to the reviewer that we have included a detailed limitation analysis in **Appendix Section A** due to the page limit. We will make this section to the main paper for the final version.\n\n### **2. Would any benefit of initialization remain when the pretrained model is scaled up?**\n\nWe kindly refer the reviewer to **Section 5.5** and **Table 4** of the main paper for a detailed analysis of applying different initialization strategies on pretrained models. Please note, the pretrained model we used (weight selection [33]) is a ViT-Small model pretrained on the ImageNet-21K dataset. Our method preserves on-par if not better performance than the vanilla pretrained model on datasets with various scales. This experiment validates the benefit of our proposed initialization when the pretrained mode is scaled up.\n\n### **3. Robustness to hyperparameter selections**\n\nWe kindly refer the reviewer to **Appendix Section F** (**Table 5** and **Figure 4**) for a detailed comparison of how robust our method is to different training configurations, such as hyperparameter selections. We would like to emphasize that under different training configurations, our method maintains the best performance while achieving the biggest improvements over the vanilla ViT model.\n\n### **4. Will the performance on larger-scale datasets or using self-supervised learning continue to be good?**\n\nWe would like to emphasize that our method is a “plug-in” module that only changes the initialization and preserves the ability of the large model to learn complex relationships with adequate data. In the paper, we have already shown in **Section 5.2** and **Table 2** that when different scaled ViTs trained with a large-scale dataset ImageNet1K, our method still achieves the best performance compared to the default or the mimetic initialization methods. In addition, training on larger-scale datasets makes the optimization less sensitive to the initialization. Therefore, when trained on even larger-scale datasets or using self-supervised learning, the ability of the model to learn more complex features will not be limited, and the performance of the model will not be downgraded."}, {"review_id": "yfwYeysdT1", "reviewer": "Reviewer_Vn6j", "summary": "The paper proposes structured initialization for Vision Transformers, in which the authors replace the usual random-weight or mimetic initializations of Q and K with carefully designed random impulse convolution filters that embed the locality bias of CNNs directly into the initial attention maps. They justify this choice theoretically by showing that, owing to the rank-deficiency of patch embeddings, a set of impulse kernels spans the same functional space as trained spatial filters, so learning can be delegated to the channel-mixing layers while the spatial bias is “baked in.” Empirically, this single-line initialization change boosts ViT-Tiny accuracy by 2~8 points on six small/medium benchmarks (Food-101, CIFAR-10/100, STL-10, Flowers, Pets), matches or betters baseline and mimetic schemes on ImageNet-1K, and transfers cleanly to Swin Transformer and MLP-Mixer variants.", "questions": "None.", "limitations": "None.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Pros:\n1. The idea of injecting inductive bias purely through initialization is highly innovative. It elegantly sidesteps the trade-offs of hybrid architectures, preserving the model's full expressivity while providing a much-needed structural prior for training.\n2. The experiments are well-designed, covering a wide range of dataset sizes, multiple architectures. The results are consistently and significantly positive.\n\nCons:\n1. The authors commendably acknowledge that their ImageNet baseline performance is slightly lower than in the original papers - DeiT. While their focus on relative improvement is valid, it slightly tempers the impact of the absolute numbers reported.\n2. In the large-scale dataset experiments shown in Table 2, as the model size increases, the performance improvement with impulse becomes more significant. Is there any detailed analysis on this? If the gain is solid, it would be worth a deeper investigation. Based on this observation, it may be valuable to try running ViT-base on the small/medium-scale datasets in Table 1 to see if even greater improvements can be achieved.\n3. The work focuses exclusively on impulse filters. While justified by Corollary 2 and the feature selection argument, it is unclear if other simple structures (e.g., Gaussian kernels, DoG filters) could offer different and potentially useful biases.", "was_revised": true, "final_justification": "Thanks to the authors for the detailed response to my concerns. As my questions have been satisfactorily addressed, I will accordingly increase my score.", "rebuttal": "We sincerely thank the reviewer for their positive and constructive feedback. We appreciate the recognition of the novelty and effectiveness of our approach. Below, we respond to each concern that was raised by the reviewer.\n\n## **Weaknesses:** ##\n\n### **1. ImageNet-1K performance discrepancy.**\n\nWe kindly point the reviewer to **Appendix Section F** for a detailed explanation of this performance gap. To summarize, we have not only provided a detailed explanation, but also provided two different experiments in **Table 5** and **Figure 4** for comparisons of using different training configurations. Both intuitive and quantitative experiments validate that, regardless of the changes in training configurations, our method maintains the best performance with the largest performance improvements over the vanilla ViT. These results indicate the effectiveness and robustness of our method under different training configurations. In addition, the reviewer has also acknowledged that the reported “relative improvements are valid”.\n\n### **2. Performance improvement increases as model size increases.**\n\nIn short, we would like to point out that the performance improvement increase is due to the increase in the number of heads. Theoretically, as explained in **Section 3**, to replace the spatial kernels with fixed filters, the initialized attention heads must span the spatial filter basis. Therefore, for a **3x3** filter used in the experiments, this requires at least **9** independent heads to span the kernel space. \nPlease note that in ConvMixer, each channel uses an independent filter, which satisfies this criterion. However, ViT models share the same filters within each head, which need a separate discussion:\n\n- ViT-Tiny has **3** heads — insufficient to span to the kernel space;\n- ViT-Small has **6** heads — insufficient to span to the kernel space;\n- ViT-Base has **12** heads — sufficient for replacing the spatial kernels.\n\nThis analysis fully supports the observation that as the model scale increases (*i.e.*, the number of heads increases), our method becomes more expressive and the performance improvement of our method increases. We will make this analysis clearer in the revised version.\n\nHowever, as the motivation of the paper also suggested, training even larger ViT models on small datasets is notoriously challenging due to optimization instability and is out of the scope of this paper. We believe that the results in **Table 1** have sufficiently validated the arguments of the paper that our proposed initialization method is effective for training transformers on limited data settings. \n\n### **3. Using filters beyond impulse filters.**\n\nWe acknowledge that other filters, such as Gaussian kernels or DoG filters, encode useful locality information and could perform equally well, as shown in our theoretical analysis, especially in **Corollary 2**. However, we use impulse filters in our method based on both practical and theoretical reasons as stated below.\n\n1) **Practicality**: As discussed in **Section 4** “why using impulse filters” (**Lines 163–173**), attention maps are generated by the softmax function, which can only be positive and naturally leads to peaked distributions—closely aligned with the structure of impulse filters. Designing initialization schemes that yield Gaussian-structured attention maps is significantly more difficult due to the non-linear softmax operation.\n\n2) **Theoretical simplicity**: Impulse filters provide a minimal and orthogonal basis to span the spatial domain, making them a simple, clear, and controllable way to introduce spatial bias while preserving the architectural flexibility of the model.\n\nIn a word, the most straightforward and suitable choice is random impulse convolution filters.\nIn future work, we are interested in exploring structured initialization with alternative attention mechanisms, which may better support the use of Gaussian-like or more complex filters."}, {"review_id": "fFjuts1QCN", "reviewer": "Reviewer_Cjkb", "summary": "This paper introduces a novel initialization strategy for ViTs that leverages the inductive biases of CNNs to improve the training of ViTs on small-scale datasets. The core idea is to initialize the attention maps based on random impulse convolution kernels, using random impulse convolution filters to initialize the query (Q) and key (K) matrices, thereby incorporating the architectural bias of CNNs into the initialization process without altering the Transformer architecture. Both the theoretical and empirical justification demonstrate the technical soundness of the method, and extensive results on various datasets reflect the effectiveness.", "questions": "Please refer to the weaknesses section.\n\nMinor-suggestions: \nLine 63: missing a blank after [21, 15]\nLimitations: although the authors discuss the limitations, it is better to include potential plans to address them.\nImplementation details: more details about the experimental settings and hyper-parameters should be provided.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Paper strength**:\n1. Combine CNN and ViT Advantages: The proposed method combines the inductive biases of CNNs with the flexibility of ViTs, unlike methods that rely on pre-trained models, this structured initialization does not require large-scale pre-training, making it accessible for tasks with limited data.\n2. Easy-to-implement and Maintains Transformer Flexibility: The proposed structured initialization dose not change the architecture, which is somewhat easy to implement and preserving ViTs’ ability to learn dynamic, long-range global features.\n3. Theoretical analysis and extensive evaluation: the theoretical elaboration provide fundamental insight and extensive results reflect the effectiveness of the proposed method.\n\n\n**Paper weakness**:\n\n1. About the weights of value (V) and projection layers: The current method focuses on the initialization of the Q and K matrices, neglecting the initialization of value weights (V) and the projection matrix. A more comprehensive initialization strategy that includes more components could potentially enhance the overall quality of ViTs, as well as provides more in-depth analysis.\n2. Why only focusing on limited-data scenarios? I wonder the practical meaning of this method to enable ViTs to perform comparable with CNNs under limited data. In other words, Transformers are better for scaling and CNNs are more suitable in relatively limited data applications, since CNNs perform well under such setting, why this paper focuses such setting instead of exploring the scaling capability of ViTs under large-scale settings?\n3. Integration with architectural design method: Since the proposed method requires no architectural modification, it is encouraged to involve methods with dedicated model designs to further testify the flexibility and compatibility of the method.\n4. Multi-scale structural initialization : considering the importance of multi-scale and hierarchical nature of images, it is suggested to explore multi-scale structural initialization and evaluate if there are consistent performance gains.\n\n**Justification**:\n1. In my opion, CNNs are relatively more suitable when limited data is available and Transformers have better scalibility, this paper's motivattion to enable Transformers to be as effective as CNNs under such scenarios is not suffecient. The essentiality and timeliness of this research is somewhat limited.\n2. The empirical evaluation could not comprehensively reflect the generalization and efficacy of the proposed method.\n3. Could the proposed method be scaled for larger model parameters with more training data, this is doubt to me.\nTherefore, I tend to vote for rejection.", "was_revised": true, "final_justification": "Most of my concerns are addressed in the authors' rebuttal, i raised my score to weak accept.", "rebuttal": "We thank the reviewer for giving a detailed evaluation of our work. We are glad that the reviewer acknowledged that our method is easy to implement and effective. We provide the detailed response below.\n\n## **Weaknesses:** ##\n\n### **1. Initialization of $V$ and projection matrices.**\n\nWe would like to remind the reviewer that our proposed initialization method theoretically originated from the architecture of CNNs, while in the CNN structure, there is no reference to guide the initialization of value $V$—only $Q$ and $K$ are responsible for ***spatial mixing***. In addition, unlike previous methods that initialize $Q$, $K$, and $V$ separately, initializing the attention map itself ensures the incorporation of architectural inductive bias of CNNs. In contrast, the $V$ and projection matrices mainly perform ***channel mixing***, which requires a different theoretical basis and motivations, and is out of the scope of this paper. Therefore, we use the default initialization adopted in the conventional Transformer to initialize other components such as FNN, normalization layers, and projection layers.\n\n### **2. Why focus on limited-data regimes?**\n\nWe would like to point out to the reviewer that the motivation of our proposed initialization method is based on the fact that in practical scenarios, it is highly desirable to have a model that 1) works well on smaller datasets without any pretraining, 2) scales seamlessly to larger datasets when needed, thus can be quickly and cost-effectively evaluate their suitability for larger learning tasks. In addition, such models are more accessible to researchers, educators, and students with limited computing resources to more readily investigate and explore the applications of ViTs. \n\nTherefore, we focus on showing that pure transformers (*e.g.,* ViTs) without architectural change can perform well in such limited data settings, while preserving the flexibility of learning global and dynamic features. As shown in the results, our initialization method significantly improves the performance of vanilla ViTs on smaller datasets while maintaining good performance on larger datasets without harming the scalability of the transformer architecture (ViTs).\n\n### **3. Integration with other specialized architectures**\n\nWe would like to emphasize that one of the advantages of our method is that no architectural modifications are needed, thus preserving the flexibility of the ViT structure, enabling good performance on large-scale datasets with larger models. Additionally, we acknowledge that our initialization method is a general and flexible “plug-in” module, and can be integrated with more specialized ViT designs. For example, we have already shown in **Section 5.3** and **Table 3** with the performance comparison on Swin-Transformer and MLP-Mixer—our proposed method achieves the best performance compared to default or mimetic initialization methods without changing the model architectures.\n\n### **4. Multi-scale structured initialization.**\n\nWe acknowledge that multi-scale or hierarchical structures are important in vision models, and exploring multi-scale initialization could be an interesting future work. However, we would like to emphasize that the focus of our paper is to demonstrate the effectiveness of using pure initialization to aid ViT training on small-scale datasets.\n\n## **Justification:** ##\n\n### **1 and 2. Motivation and impact of our research, generalizability, and effectiveness of our method.**\n\nWe would like to clarify that the motivation of our research work is reasonable, practical, and addresses real-world settings where 1) pretraining of the model is infeasible; 2) (training) data is limited; 3) model reuse is required across various model scales. We kindly refer the reviewer to the existing line of work **[7,9,14,15,21,23,24,28,29,32,33,35,37]** in studying how to enable training transformer models, especially ViTs, on smaller-scale datasets, which we have discussed in the introduction and related work sections of the paper, to better understand the motivations of our research. \n\nIn the paper, we have demonstrated the consistently good performance of our proposed method in 6 diverse smaller-scale datasets with significant improvements over the vanilla ViT models, as well as strong performance on the larger-scale dataset ImageNet-1K to confirm the scalability and the robustness of our method. We believe all these experiments have validated the generalizability and the effectiveness of our proposed method.\n\n### **3. Performance on even larger-scale models and datasets.**\n\nWe would like to emphasize that the focus of the paper is to introduce the architectural inductive bias from CNNs to initialize the attention map in Transformers to aid training on limited data settings. Although we have shown that our method maintains good performance on large-scale datasets such as ImageNet-1K, the focus is to validate the preservation of the transformer architectures and the flexibility of our method in learning dynamic global relationships despite introducing the CNN inductive bias during initialization. We would like to clarify that our method is a “plug-in” module that can be easily integrated into larger-scale models. However, discussions of such applications are beyond the scope of this paper.\n\n## **Minor suggestions and implementation details.**\n\nWe will correct the formatting issues. We would like to point the reviewer to **Section 5** and **Appendix Section F** for detailed training configurations. We will expand the appendix with more implementation and training details."}, {"review_id": "AG3wYL8FzG", "reviewer": "Reviewer_nTev", "summary": "Vision Transformers enable global attention across image patches. However, in the vision domain, local patches tend to exhibit strong correlations, and ViTs struggle to capture such local dependencies effectively. As a result, ViTs often require more training time and larger training datasets, which poses challenges for training on small datasets. This paper introduces a structured initialization method that mimics the behavior of CNNs, helping ViTs better capture local correlations from the start. The proposed initialization leads to improved accuracy compared to standard random initialization and is particularly valuable when training ViTs on small datasets.", "questions": "- There is a lack of explanation regarding the visualization of the attention maps. The paper states that the proposed method is superior because it does not exhibit purely diagonal behavior, unlike the baselines. However, this point is not well elaborated. Could the authors clarify why non-diagonal attention patterns are beneficial?\n- In Appendix B, the authors state that the proposed impulse initialization does not prevent the model from attending globally. However, the attention map visualizations suggest that the self-attention behavior in the proposed method is strongly influenced by the initialized impulse patterns. This appears to present a potential contradiction. Could the authors clarify how the model transitions from the localized, impulse-driven patterns at initialization to globally-aware attention during training?\n- In the training pipeline, you mention that for the small datasets, “We follow the training recipe from [Initializing models with larger ones]”. However, that paper appears to rely on pretrained models for training. Could you clarify what specific aspects of their approach you adopted for the small datasets? Did you use pretrained models, or were there other elements of their training procedure that you followed?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The paper is clearly written and well-organized.\n- The proposed initialization is simple to implement and effective, without requiring architectural changes or pretrained weights.\n- The method demonstrates consistent performance improvements across a wide range of datasets, particularly on small and medium-scale datasets.\n\nWeaknesses:\n- In Table 2, the reported accuracy for DeiT-Base is 76.25%, which is significantly lower than the 81.8% reported in the `timm` library. This 5–6% discrepancy suggests a potential issue with the experimental setup. The authors mention in the appendix that the gap is due to differences in learning rate scaling and data augmentation settings, and they report an improved accuracy of 81.24% for the default initialization under corrected settings. However, it is surprising that this corrected result is not included in Table 2 of the main paper. This omission is problematic, as the presented results show both the mimetic baseline and the proposed method underperform compared to random initialization from `timm`, which contradicts the paper’s stated goal of outperforming standard initialization. Moreover, the corrected accuracy (81.24%) still shows a ~0.5% gap from the 81.8% reference, indicating that the issue has not been fully resolved. I recommend the authors refer to the official DeiT repository for verified training hyperparameters: https://github.com/facebookresearch/deit/tree/main. Given that this paper targets a top-tier conference like NeurIPS, such discrepancie on such a well known baselines are not acceptable.\n- While the paper claims broad applicability to Swin and MLP-Mixer models, the experimental evaluation for these architectures is very limited. For example, MLP-Mixer is evaluated only on CIFAR-10. If the authors wish to claim general applicability in the abstract and contributions, a more comprehensive evaluation is warranted.\n- Since the use of random impulse filters may bias the Vision Transformer toward more local attention at the expense of global attention, the paper would benefit from including results on ImageNet variants such as ImageNet-V2, ImageNet-R, or ImageNet-Sketch to demonstrate robustness to distribution shifts.\n- Since the proposed method mimics CNN-like locality, which typically leads to faster convergence, an ablation study on the number of training epochs would be insightful. It would be interesting to see whether the structured initialization leads to faster convergence compared to the baselines. For example, instead of training for 300 epochs, train for 100 epochs and compare the results... I would expect to get better results.\n- The impulse filter generation is stochastic, so final performance may depend on the random seed. An evaluation of variance across different seeds would help assess the robustness of the method.\n- The paper claims (line 36) that prior initialization methods are tied to specific architectures, but it does not specify which methods these are or explain how the limitation applies. This point needs further elaboration.\n- The models are referred to as “ViT” even though they are trained using DeiT training recipes. To avoid confusion with other works (e.g., “How to Train Your ViT”), it would be more accurate to refer to these models as “DeiT.”", "was_revised": true, "final_justification": "While I remain somewhat unconvinced by the reasoning provided regarding the discrepancies in the DeiT-Base results, I feel that my main concerns have been addressed. The core idea of the paper is interesting, and the overall results are promising enough. I have increased my score accordingly.", "rebuttal": "We sincerely thank the reviewer for the constructive and thoughtful review. We're encouraged that the reviewer appreciates our method’s simplicity, clarity, and consistent improvements across small and medium-scale datasets. Below, we address each concern raised by the reviewer in detail.\n\n## Weaknesses: ##\n\n### **1. ImageNet-1K Performance discrepancy.**\n\nWe kindly point the reviewer to **Appendix Section F** for a detailed explanation of this performance gap. In a word, different ```timm``` versions result in different training settings such as different augmentations, learning rate scalings, *etc* (summarized in **Table 5**). We will update the main paper with the detailed explanation and the updated results reported in the Appendix. In detail, we will update **Table 2** with the **Table 5** results. We will clearly report that under the training settings of a scaled learning rate of $1e{-}3$, and a repeated augmentation factor of $3.0$, with ***default initialization***, the vanilla ViT-Base achieves an accuracy of $81.24\\%$ on ImageNet-1K dataset, while mimetic initialization achieves $80.56\\%$, and our method achieves $81.83\\%$.\n\n| Scaled LR | Repeated Augment | Default | Mimetic | Ours (impulse) | \n| :---: | :----: | :-----: | :----: | :----: | \n| 1e-3| 3.0 | 81.24 | 80.56, 0.68 $\\downarrow$ | **81.83**, 0.59 $\\uparrow$ |\n\nIn addition, we would like to clarify that the remaining small gap (\\~0.5\\%) between our reported vanilla ViT result and the previous result is attributed to minor differences in hardware settings and implementation library versions.\nNevertheless, both intuitive and quantitative experiments validate that no matter how the training configurations change, our method maintains the best performance with the biggest performance improvements over the vanilla ViT. These results indicate the effectiveness and the robustness of our method under different training configurations.\n\n### **2. Evaluation on other ViT variants.**\n\nWe would like to clarify that we include results on Swin and MLP-Mixer as a proof-of-concept application to demonstrate that our method does not require any structural modifications, and can be easily integrated into different architectures as a plug-in module. Further extensions to various non-ViT models could be an interesting future work.\n\n### **3. More experiments on other larger-scale datasets, such as ImageNet variants.**\n\nWe would like to emphasize that the focus of the paper is to introduce the architectural inductive bias from CNNs to initialize the attention map in Transformers to aid training on limited data settings. Although we have shown that our method maintains good performance on large-scale datasets such as ImageNet-1K, the focus is to validate the preservation of the transformer architectures and the flexibility of our method in learning dynamic global relationships when enough data is given—for example, training on different ImageNet variants—despite introducing the CNN inductive bias during initialization. \n\n### **4. Model convergence rate.**\n\nWe would like to clarify that while impulse filters mimic convolutional locality, they ***do not reuse weights*** like CNNs do—which may explain the fast convergence of CNNs. Interestingly, we did observe a faster convergence when using the mimetic or impulse initialization to replace the vanilla ViT model. We give an example of the training accuracy of ViT-Tiny with different initialization methods on CIFAR-10 during the optimization stage, as shown below: \n\n| | Default | Mimetic | Ours (impulse) | \n| :---: | :---: | :----: | :---: | \n| Epoch 10 | 47.95 | 49.05 | **54.62** |\n| Epoch 50 | 61.51 | 63.62 | **69.51** |\n| Epoch 100 | 76.87 | 77.64 | **81.87** |\n\nWe can clearly see a faster convergence of mimetic and our impulse initialization methods compared to the default initialization used in vanilla ViTs. \nOne intuitive explanation is that the mimetic initialization yields \\~4x the norm values of the default initialization, while our initialization method has \\~2x the norm values of the default initialization. Different norm values will affect the gradient magnitudes and early training dynamics. We will add such a discussion to the revised version.\n\n### **5. Multiple runs with different random seeds and stochastic filters**\n\nWe would like to refer the reviewer to **Appendix G.2** and **Figure 5** for detailed experiments on using different seeds with stochastic filter generations with **5** different runs. We will also update the results in the main paper accordingly. \n\n### **6. Explain the limitation of previous methods: “their effectiveness is often tied to specific model architectures”**\n\nWe would like to clarify that when we discuss the limitations of the previous methods, *“their effectiveness is often tied to specific model architectures”* refers to methods that are “architecture-dependent”—specialized to a certain model and cannot be treated as a “plug-in” module to generalize to other types of architectures. For example, methods like [14, 23, 24, 33] *“directly sampled weights from pretrained large-scale models as initialization for smaller models”* (**Line 75-76**), which means those weights are only effective for the specific models with specific parameter sizes. Unlike these previous methods, our approach is “architecture-agnostic”, does not require any pretrained weights, and can work as a “plug-in” module across different model architectures.\n\n### **7. Naming of ViT and DeiT**\n\nWe would like to clarify that we use conventional ViT architectures in our experiments, except for **Section 5.3**, where we adopt different ViT variants such as Swin Transformers and MLP-Mixer. Specifically, we use the default ViT implementation from the ```timm``` library for most of our experiments. We only follow the training recipe of DeiT for the ImageNet-1K experiments for its clarity and fair comparison to the previous methods.\n\n## **Questions:** ##\n\n### **1. Further explanations for the attention map visualization.**\n\nWe would like to refer to **Section 5.4**, especially **Line 277-280**, for a detailed explanation of non-diagonal peaks in the attention patterns. Specifically, different attention peak patterns shown in **Figure 3** correspond to different impulse patterns that are applied to different attention heads. We would also point the reviewer to **Section 3**, where we theoretically show that as long as the initial filters span the spatial filter space, they are functionally equivalent to distinct spatial filters in CNNs in terms of their representation ability. \nFrom these perspectives, each attention head should ideally serve as an independent basis vector, and different heads should be orthogonal to one another to have sufficient representation ability to their counterparts in CNNs.\n\nFrom **Figure 3**, we clearly see that our initialization generates different attention heads initialized with different patterns (better representation ability) while default and mimetic methods only initialize attention heads in the same way (no sufficient representation ability compared to spatial filters in CNNs). \nOur structured initialization ensures that the model starts with diverse, localized spatial priors while preserving the flexibility to learn global dependencies through training.\n\n### **2. Clarification of attention map visualization and how the proposed method still enables attending global features during training**\n\nWe would like to clarify that the attention map shown in **Figure 3** and **Appendix Figure 6** are the attention maps during initialization—before training. Therefore, the attention maps show clear and distinct patterns for different initialization methods. Since our method is a “plug-in” module that only changes the initialization method, no architectural modifications to the transformer structure are made. Therefore, training such ViTs still allows the model to update the attention weights $Q$, $K$, $V$, and enables the model to learn global, dynamic dependencies through optimization.\n\nNonetheless, we have observed that although the attention patterns differ at the initialization stage, the attention map of ViTs using our initialization method gradually yields similar patterns as the conventional ViTs with default initialization. The final heat map of different images also validates that our method does not prevent the model from attending to global features.\n\nDue to the rebuttal limit, we cannot show such figures here. We will update the revised version with attention maps before, during, and after training, as well as attention heat maps of different inference images to better support our argument.\n\n### **3. Clarification on using pretrained models**\n\nWe would like to clarify that we ***did not use any pretrained models*** for our methods except for **Section 5.5**, where the pretrained models are explicitly evaluated. For all other smaller-scale experiments, we only follow the training configurations from [33] for a fair comparison, including batch size, learning rate, number of epochs, *etc*. Although [33] initializes the model weights from large pretrained models, we ***did not adopt any pretraining step for our comparisons***. In contrast, we apply default, mimetic, and our impulse initialization methods to ViT-Tiny models from scratch. We will make this implementation detail clearer in the revised version."}]

Author rebuttal:

Meta-review: The paper proposes a structured initialization scheme for vision transformers, approximating impulse filters. This initialization substantially improves the performance in small-data regimes over standard initialization and also over some recent work. 

The results are fairly solid, with substantial improvements on several datasets and architectures.

The main concern is about the DeiT baseline on Imagenet-1k performing much worse than the original implementation. This has been discussed and the authors provided updated numbers with updated data augmentation and learning rate schedule - the proposed initialization still helps, although much less. This does not really affect the main conclusions of the paper, but investigating the proposed method at larger scale would certainly be a valuable follow-up.

The reviewers are all positive about the paper, the results are solid, so I recommend acceptance.
Decision: Accept (poster)