Structured Initialization for Attention in Vision Transformers

## Abstract

Abstract The training of vision transformer (ViT) networks on small-scale datasets poses a significant challenge.
By contrast, convolutional neural networks (CNNs) have an architectural inductive bias enabling them to perform well on such problems.
In this paper, we argue that the architectural bias inherent to CNNs can be reinterpreted as an initialization bias within ViT.
This insight is significant as it empowers ViTs to perform equally well on small-scale problems while maintaining their flexibility for large-scale applications.
Our inspiration for this “structured” initialization stems from our empirical observation that random impulse filters can achieve comparable performance to learned filters within CNNs.
Our approach achieves state-of-the-art performance for data-efficient ViT learning across numerous benchmarks including CIFAR-10, CIFAR-100, and SVHN.

###### Keywords:

## 1 Introduction

Figure: Figure 1: Illustration comparing conventional generative initialization with structured initialization (ours) strategy for the weights $\mathbf{Q}$ and $\mathbf{K}$ of the self-attention in transformers. Conventional generative initialization involves sampling parameters $\mathbf{Q}$ and $\mathbf{K}$ from certain distributions, such as Gaussian or Uniform, resulting in unstructured initial attention maps. In contrast, our structured initialization strategy imposes constraints on the initial structure of the attention maps, specifically requiring them to be random impulse filters. The initialization of parameters $\mathbf{Q}$ and $\mathbf{K}$ is computed based on this requirement on attention maps. Note that in both attention maps and random impulse filters, the pink cells indicate ones, while the gray cells represent zeros.
Refer to caption: extracted/2404.01139v1/init_method.png

Vision Transformer (ViT) networks have shown significant promise when exposed to large amounts of training data.
Their performance, however, is diminished when applied to small-scale datasets where convolutional neural networks (CNNs) still outperform ViTs by a significant margin [, ].
To address this drawback, ViT [] has employed pre-training on larger-scale datasets such as ImageNet [], JFT-300M [], *etc*.
However, this approach has a fundamental limitation.
One motivation for training visual learning algorithms, such as CNNs, on smaller-scale datasets is the ability to quickly and cost-effectively evaluate their suitability for larger-scale learning tasks.
Requiring an algorithm to pre-train on a large-scale dataset before training on a smaller-scale dataset essentially undermines this purpose.
Overcoming this limitation in modern ViTs will make them more accessible and potentially spark new waves of innovation and exploration within the vision community.

Recent advances in CNNs, most notably ConvMixers [], have become increasingly similar in architecture to ViTs.
Specifically, they split multi-channel convolution into depth-wise and channel-wise convolution.
Cazenavette *et al*. [] recently demonstrated that random depth-wise (spatial mixing) convolution filter weights can achieve comparable performance to learned weights in ConvMixer and ResNet [] frameworks.
The only parameters that need to be learned are those associated with channel-wise convolution, commonly referred to as channel mixing in the transformer literature [].
This result is particularly intriguing, suggesting that the utility of CNNs lies largely in their convolution structure rather than the exact weights of the filters.
ViTs still utilize channel-wise convolution for channel mixing but substitute depth-wise convolution with multi-headed attention for spatial mixing.
However, unlike depth-wise convolution, multi-headed attention lacks a predefined structure and instead learns spatial relations from data, which makes it challenging to learn on small-scale datasets.

Expanding upon this insight, we propose to reinterpret the architectural bias found in CNNs into an initialization bias within ViTs.
Specifically, we provide a theoretical explanation for the effectiveness of random channel-wise convolution filters in ConvMixer.
Furthermore, we demonstrate that utilizing impulse filters yields similarly impressive performance without the need for learning.
While the theoretical equivalence between impulse filters and random filters may appear trivial in ConvMixer, the utilization of impulse filters in ViTs differs substantially from random filters as it inherently builds the softmax structure.
An impulse filter contains only ones and zeros, with each row (matching the softmax dimension) having precisely a single non-zero entry, which can be effectively modeled as a softmax attention matrix.
Based on these observations, we propose that the convolutional inductive bias inherent in CNNs through their architecture can be realized through a structured initialization of the attention maps within ViT, specifically initializing the attention maps within ViT with convolutional matrices of impulse filters.

Traditional initialization strategies typically follow a generative approach, where parameters are sampled from identically independent distributions.
Many works [, ] focus on exploring the suitable distribution for this purpose.
However, the unique architecture of ViT emphasizes the importance not only of the parameters in $\mathbf{Q}$ and $\mathbf{K}$ but also of the attention maps they formalized.
In contrast to conventional generative approaches, our structured initialization imposes specific structural requirements on the attention maps, controlling $\mathbf{Q}$ and $\mathbf{K}$ without directly constraining trainable parameters (see Fig. 1).
Additionally, our strategy involves designing different initializations for different attention heads, as the attention map of each head resembles a distinct impulse filter.
To the best of our knowledge, we are the first to adopt this innovative approach, which embeds some of the architectural principles of CNNs as an initialization bias within ViTs.

Our paper makes the following contributions:

- •
We provide a theoretical explanation for the effectiveness of random spatial convolution filters, attributing their success to the redundancy in embeddings.
Specifically, we argue that the learning of spatial convolution filters can be attributed to channel mixing weights as long as the redundancy condition is satisfied.
- •
We propose to use convolutional structured impulse initialization on the attention maps in ViT.
Our approach embeds this CNN architectural inductive bias as a structured initialization while preserving the flexibility of transformers in learning attention maps.
- •
We demonstrate state-of-the-art performance for data-efficient ViT learning across various benchmarks including CIFAR-10, CIFAR-100 [], and SVHN [].
Our approach outperforms the recent mimetic approach [] and offers a deeper understanding of the ViT initialization.

## 2 Related Work

Convolution as attention.
Since their introduction [, ], the relationship between transformers and CNNs has been a topic of immense interest to the vision community.
Andreoli [] studied the structural similarities between attention and convolution, bridging them into a unified framework.
Building on this, Cordonnier *et al*. [] demonstrated that self-attention layers can express any convolutional layers through a careful theoretical construction.
However, while these studies highlighted the functional equivalence between self-attention in ViTs and convolutional spatial mixing in CNNs, they did not delve into how the inductive bias of ViTs could be adapted or enhanced through this theoretical connection.
Moreover, the reliance on relative positional encoding, which is not widely used in current mainstream ViT implementations, limited the practical impact of these insights.
In contrast, our work offers a simpler insight: random impulse filter convolution can be effectively approximated by softmax self-attention.

Bias through architecture.
Many efforts have aimed to incorporate convolutional inductive bias into ViTs through architectural modifications.
Dai *et al*. [] proposed to combine convolution and self-attention by mixing the convolutional self-attention layers.
Both Pan *et al*. [] and Li *et al*. [] introduced hybrid models wherein the output of each layer is a summation of convolution and self-attention.
Wu *et al*. [] explored using convolution for token projections within self-attention, while Yuan *et al*. [] demonstrated promising results by inserting a depthwise convolution before the self-attention map as an alternate strategy for injecting inductive bias.
d’Ascoli *et al*. [] introduced gated positional self-attention (GPSA) to imply a soft convolution inductive bias.
However, these techniques have a fundamental limitation—they aim to introduce the inductive bias of convolution through architectural choices.
Our approach, on the other hand, stands out by preserving the architectural flexibility of ViT through a novel CNN-inspired initialization strategy.
Such an approach offers several advantages as it: (i) exhibits data efficiency on small-scale datasets, (ii) retains the architectural freedom to be seamlessly applied to larger-scale datasets, and (iii) gives an alternate theoretical perspective on how the inductive bias of convolution can be applied within transformers.

Bias through initialization.
To date, the exploration of applying inductive bias through initialization within a transformer is limited.
Zhang *et al*. [] posited that the benefit of pre-trained models in ViTs can be interpreted as a more effective strategy for initialization.
Trockman *et al*. [, ] recently investigated the empirical distributions of self-attention weights, learned from large-scale datasets, and proposed a mimetic initialization strategy.
While this approach lies between structured and generative initialization, it relies on the simple structure derived from pre-trained large models.
A key difference in our approach is that it does not require offline knowledge of pre-trained networks (mimetic or empirical).
Instead, our strategy is intrinsically connected to the inductive bias of convolution, without the need to incorporate convolution as an architectural choice.

Figure: Figure 2: Illustration of why random spatial convolution filters are effective. Patch embeddings $\mathbf{X}\,{\in}\,\mathbb{R}^{N{\times}D}$ are typically rank-deficient and can be approximately decomposed to $k$ basis. Meanwhile, a linear combination of $f^{2}$ linearly independent filters $\mathbf{h}$ can express any arbitrary filter in the filter space $\mathbb{R}^{f{\times}f}$. Based on these two observations, we derive the inequality $D\,{\geq}\,kf^{2}$ in 1.

## 3 Preliminaries: Why Random Filters Work?

Cazenavette *et al*. [] recently demonstrated that randomly initialized convolution filters, in networks such as ConvMixer and ResNet, work remarkably well when solely learning the channel mixing parameters.
However, they failed to offer any insights into the underlying reasons.
In this section, we provide a brief theoretical intuition.
This finding is significant as it establishes a conceptual link between the architecture of ConvMixer and the initialization of ViT, offering a deeper understanding of desired properties for spatial mixing matrices.For clarity and simplicity, we have omitted activations (*e.g*., GeLU, ReLU, *etc*.), bias, batch normalization, and skip connections in our formulation.

###### Remark 1

Let us define the patch embeddings or intermediate layer outputs in ConvMixer as $\mathbf{X}\,{=}\,[\mathbf{x}_{1},\mathbf{x}_{2},\dots,\mathbf{x}_{D}]$, where $D$ is the number of channels and $N$ is the number of pixels in the vectorized patch $\mathbf{x}\,{\in}\,\mathbb{R}^{N}$.
An interesting observation is the rank (stable rank, defined as $\sum\sigma^{2}/\sigma_{max}^{2}$) of $\mathbf{X}$ is consistently much smaller than the minimum dimension $\mbox{min}(N,D)$ of $\mathbf{X}$, which indicates a significant amount of redundancy in $\mathbf{X}$.

Let us define a 2D convolution filter as $\mathbf{h}\,{\in}\,\mathbb{R}^{f{\times}f}$.
In general, this kernel can be represented as a convolutional matrix $\mathbf{H}\,{\in}\,\mathbb{R}^{N{\times}N}$, such that $\mathbf{h}\,{*}\,\mathbf{x}\,{=}\,\mathbf{H}\mathbf{x}$, where $*$ is the convolutional operator.
Let $\mathbf{w}\,{=}\,[w_{1},w_{2},\dots,w_{D}]^{T}\,{\in}\,\mathbb{R}^{D{\times}1}$ be the channel mixing weights for one output channel and $\mathbf{H}_{1},\mathbf{H}_{2},\dots,\mathbf{H}_{D}$ are the corresponding spatial convolution filters for each channel.
Therefore, the result $\mathbf{y}\,{\in}\,\mathbb{R}^{N}$ after spatial and channel mixing can be represented as,

$$ $\mathbf{y}=\sum_{i=1}^{D}w_{i}\mathbf{H}_{i}\mathbf{x}_{i},$ (1) $$

With Remark 1, we can suppose the rank of $\mathbf{X}\,{\approx}\,\mathbf{Z}\mathbf{A}$ is $k$, where $\mathbf{Z}\,{=}\,[\mathbf{z}_{1},\ldots,z_{k}]$ and $k\,{\ll}\,D$, as illustrated in Fig. 2.
We then obtain

$$ $\displaystyle\mathbf{y}\approx\sum_{i=1}^{D}\sum_{j=1}^{k}w_{i}a_{ji}\mathbf{H }_{i}\mathbf{z}_{j}=\sum_{j=1}^{k}\tilde{\mathbf{H}}_{j}\mathbf{z}_{j},$ (2) $$

where $a_{ji}$ refers to the row $j$, column $i$ element of $\mathbf{A}$, and $\tilde{\mathbf{H}}_{j}\,{=}\,\sum_{i=1}^{D}w_{i}\,a_{ji}\,\mathbf{H}_{i}$.

Remember that a linear combination of $f^{2}$ linearly independent filters $\mathbf{h}$ can express any arbitrary filter in filter space $\mathbb{R}^{f{\times}f}$, where $\mathbf{h}$ serves as the basis.
Consequently, any desired $\tilde{\mathbf{H}}_{1},\tilde{\mathbf{H}}_{2},\dots,\tilde{\mathbf{H}}_{D}$ can be achieved by only learning the channel mixing weights $\mathbf{w}$.
Therefore, we obtain the following proposition.

###### Proposition 1

In a ConvMixer block composed of a spatial mixing layer and a channel mixing layer, suppose $D$ is the number of channels, $k$ is the rank of input $\mathbf{X}$, and $f{\times}f$ is the number of convolution filters basis, then any possible output for $f{\times}f$ filters can be achieved by only learning the channel mixing weights, as long as

$$ $D\geq kf^{2}.$ (3) $$

Although this derivation simplifies by omitting activations, batch normalizations, and skip connections, it offers an intuition on how solely learning channel mixing can be sufficient for achieving reasonably good performance.
Furthermore, it reveals that the convolution filters do not necessarily need to be strictly unique from each other.
As long as we have $f^{2}$ linearly independent convolution filters, each with at least $k$ copies, such as a random impulse filter, the model should still perform well.
Additionally, this configuration can be viewed as having $f^{2}$ heads, with each head having an embedding dimension of $k$ in multi-head self-attention.

###### Remark 1

###### Proposition 1

## 4 Method

ConvMixer and ViT share most components in their architectures.
The gap in their performance on small-scale datasets stems from their architectural choices regarding spatial mixing matrix.
Although depthwise convolution (ConvMixer) and multi-head self-attention (ViT) may appear distinct at first glance, their underlying goal remains the same: to identify spatial patterns indicated by the spatial mixing matrix.
The spatial mixing step can be expressed as

$$ $\mathbf{x}\leftarrow\mathbf{M}\mathbf{x},$ (4) $$

where $\mathbf{X}\,{=}\,[\mathbf{x}_{1},\ldots,\mathbf{x}_{D}]$, $\mathbf{x}\,{\in}\,\mathbb{R}^{N}$ are the patch embeddings or intermediate outputs, and $\mathbf{M}$ is the $N{\times}N$ spatial mixing matrix.

In ConvMixer, there exists a strong assumption that the pattern only appears in the local area defined by the convolutional kernel size.
Therefore, the spatial mixing matrix $\mathbf{M}$ can be viewed as the convolutional matrix $\mathbf{H}$ in Eq. 1.
In particular, for each channel $\mathbf{x}_{1},\ldots,\mathbf{x}_{D}$, the corresponding spatial mixing matrix $\mathbf{M}_{1},\mathbf{M}_{2},\dots,\mathbf{M}_{D}$ is $\mathbf{H}_{1},\mathbf{H}_{2},\dots,\mathbf{H}_{D}$.

In ViT, this assumption is noticeably weaker.
The pattern is learned from data, based on the similarity of projected input.
Therefore, the matrix $\mathbf{M}$ for multi-head self-attention can be expressed as follows:

$$ $\mathbf{M}_{i}^{\hphantom{T}}=\mbox{softmax}(\mathbf{X}\mathbf{Q}_{i}^{ \hphantom{T}}\mathbf{K}_{i}^{T}\mathbf{X}^{T}),$ (5) $$

where $\mathbf{Q}_{i}^{\hphantom{T}},\mathbf{K}_{i}^{\hphantom{T}}\,{\in}\,\mathbb{R}
^{D{\times}K}$ denote the attention weight matrices, $i\,{=}\,1,\ldots,h$, and $K$ represents the feature dimension in each head, typically set to $D/h$, with $h$ being the number of heads.
In summary, for channels $\mathbf{x}_{1},\ldots,\mathbf{x}_{D}$, the spatial mixing matrices are $\mathbf{M}_{1},\mathbf{M}_{2},\dots,\mathbf{M}_{h}$ where each $\mathbf{M}_{i}$ is shared by $D/h$ channels.
It is similar to our findings in ConvMixer, except that there are in total $h$ unique spatial mixing matrices and each has $D/h$ copies.
Recall our discussion in 1, where we noted that the filter matrices $\mathbf{H}_{1},\mathbf{H}_{2},\dots,\mathbf{H}_{D}$ need not to be unique from each other.
Leveraging this insight, we propose to initialize the attention map for each head in ViT with a convolutional filter matrix $\mathbf{H}$.
Our approach integrates the architectural bias inherent in CNNs into the initialization of attention maps within ViT.
For clarity and brevity, the following discussions will focus only on one head of multi-head self-attention.

### 4.1 Structured Convolutional Initialization

Based on our observation in ConvMixer and ViT, we propose to use a structured convolutional initialization strategy to
incorporate prior knowledge into the initialization of the attention map, imposing a convolutional structure on it:

$$ $\mathbf{M}_{\text{init}}^{\hphantom{T}}=\mbox{softmax}(\mathbf{X}\mathbf{Q}_{ \text{init}}^{\hphantom{T}}\mathbf{K}_{\text{init}}^{T}\mathbf{X}^{T})\approx \mathbf{H}.$ (6) $$

Why using impulse filters?
Usually, random convolution filters contain both positive and negative values, while the output of the softmax function is always positive.
One straightforward option is to use random positive convolution filters with a normalized sum of one, following the property of softmax.
However, this approach often proves inefficient as the patterns may be too complicated for a softmax function to handle.
Tarzanagh *et al*. [] found that the softmax attention map functions as a feature selection mechanism, and typically tends to select a single related feature.
In convolution filters, this softmax attention map can be viewed as an impulse filter.
According to 1, there exists no distinction in the choices of different filters, as long as these filters form the basis for the $f^{2}$ space.
In conclusion, when initializing a softmax attention map, the most straightforward and suitable choice is random impulse convolution filters.

Pseudo input.
The advantage of self-attention is that its spatial mixing map is learned from data.
However, during the initialization phase, there is no prior information about the input.
To address this problem, we use absolute sinusoidal positional encoding $\mathbf{P}$ as pseudo input, replacing the original $\mathbf{X}$.
In addition, we explore alternative pseudo inputs such as random inputs sampled from Gaussian or Uniform distribution, a combination of random inputs and positional encoding, *etc*.
The ablation study of different pseudo inputs is presented in Sec. 5.4.
While some of these choices may yield promising results, the simplest and most reasonable approach remains the utilization of positional encoding.

With these two adaptions, Eq. 6 becomes

$$ $\mathbf{M}_{\text{init}}^{\hphantom{T}}=\mbox{softmax}(\mathbf{P}\mathbf{Q}_{ \text{init}}^{\hphantom{T}}\mathbf{K}_{\text{init}}^{T}\mathbf{P}^{T})\approx \mathbf{H}_{\text{impulse}}^{\hphantom{T}}.$ (7) $$

### 4.2 Solving 𝐐 init superscript subscript 𝐐 init absent \mathbf{Q}_{\text{init}}^{\hphantom{T}} bold_Q start_POSTSUBSCRIPT init end_POSTSUBSCRIPT start_POSTSUPERSCRIPT end_POSTSUPERSCRIPT and 𝐊 init subscript 𝐊 init \mathbf{K}_{\text{init}} bold_K start_POSTSUBSCRIPT init end_POSTSUBSCRIPT

There exist numerous approaches to solve Eq. 7 for $\mathbf{Q}_{\text{init}}^{\hphantom{T}}$ and $\mathbf{K}_{\text{init}}^{\hphantom{T}}$ with known $\mathbf{H}_{\text{impulse}}^{\hphantom{T}}$ and $\mathbf{P}$.
In mimetic initialization [], $\mathbf{Q}_{\text{init}}^{\hphantom{T}}\mathbf{K}_{\text{init}}^{T}$ is initialized, and SVD is utilized to solve $\mathbf{Q}_{\text{init}}^{\hphantom{T}}$ and $\mathbf{K}_{\text{init}}^{\hphantom{T}}$.
While a similar SVD-based approach could be employed in our scenario—despite we intend to initialize the softmax attention map, it is found to be ineffective due to the large error resulting from the pseudo-inverse of $\mathbf{P}$ and low-rank approximation.
Consequently, we opt not to pursue an analytical solution but rather employ a simple optimization to obtain $\mathbf{Q}_{\text{init}}$ and $\mathbf{K}_{\text{init}}$.
This approach also addresses concerns regarding scale and layer normalization in the attention mechanism.

Figure: Algorithm 1 Convolutional structured impulse initialization for ViT

The pseudo code for our initialization strategy is shown in Algorithm 1.
In the first step, we compute the attention map $\mathbf{H}_{\text{impulse}}^{\hphantom{T}}$ based on the 2D impulse convolution matrix.
The pseudo input $\tilde{\mathbf{X}}$ is then computed through the absolute positional encoding $\mathbf{P}$.
Note that the pseudo input $\tilde{\mathbf{X}}$ remains constant throughout the entire optimization process without requiring re-sampling.
Additionally, the constant scale $\sigma$, and any normalization techniques such as layer normalization or batch normalization remain consistent with those utilized in ViT.

To optimize $\mathbf{Q}_{\text{init}}^{\hphantom{T}}$ and $\mathbf{K}_{\text{init}}^{\hphantom{T}}$, our objective function is defined as

$$ $\displaystyle\operatorname*{arg\,min}_{\mathbf{Q}_{\text{init}}^{\hphantom{T}} ,\mathbf{K}_{\text{init}}^{\hphantom{T}}}\frac{1}{N^{2}}\left\|\mathbf{H}_{ \text{impulse}}^{\hphantom{T}}-\mbox{softmax}\left(\sigma\tilde{\mathbf{X}} \mathbf{Q}_{\text{init}}^{\hphantom{T}}\mathbf{K}_{\text{init}}^{T}\tilde{ \mathbf{X}}^{T}\right)\right\|^{2}_{F},$ (8) $$

where $\tilde{\mathbf{X}}$ is the normalized pseudo input, and $\mathbf{Q}_{\text{init}}^{\hphantom{T}}$ and $\mathbf{K}_{\text{init}}^{\hphantom{T}}$ can be random initialized before optimization.
We then optimize for $max_iter\,{=}\,10,000$ epochs using Adam optimizer [] with a learning rate of $1e^{-4}$ using mean squared error (MSE) loss.
It is worth noting that this optimization does not count as a pre-training step since no real data is involved.
Rather, our optimization algorithm serves as a surrogate for the SVD solver, converging in just a few seconds ($\sim$5s).

## 5 Experiments and Analysis

### 5.1 Settings

Dataset.
We evaluate our structured initialization strategy on the small-scale datasets CIFAR-10, CIFAR-100 [], SVHN [].
Additionally, we test our model on a large-scale ImageNet-1K [] dataset.
Furthermore, in validating our theory on ConvMixer, we conduct all related experiments on CIFAR-10.

Models.
Our experiments primarily focus on the tiny ViT model, namely ViT-T [].
Instead of using the classification token and a learnable positional encoding as defined in ViT, we use the average global pooling and a sinusoidal absolute positional encoding.
In general, these small tweaks will not compromise the performance of ViTs.
On the contrary, as shown in Tab. 1, these two modifications lead to improved performance on the CIFAR-10 dataset.
Henceforth, all the following experiments use this configuration.
The default architecture of ViT-T includes a depth of 12, an embedding dimension of 192, and 3 heads.

Table: Table 1: Classification accuracy(%) of ViT-T with various basic settings on CIFAR-10.

Training.
We utilize the PyTorch Image Models (timm) [] to train all ViT models.
We employ a simple random augmentation strategy from [] for data augmentation.
Our models were trained with a batch size of 512 using the AdamW [] optimizer, with a learning rate of $10^{-3}$ and weight decay set to 0.01, for 200 epochs.
Note that all experiments were conducted on the Tesla V100 SXM3 with 32GB memory.

Initialization.
Considering that the number of heads in ViTs is typically small, we utilized both $3{\times}3$ (Imp.-3) and $5{\times}5$ (Imp.-5) filters for our convolutional structured impulse initialization method.
We compare our method with Pytorch default initialization (Kaiming Uniform []), timm default initialization (Trunc Normal), and mimetic initialization (Mimetic []).

### 5.2 Results Across Datasets

In Tab. 2, we present the results of five different methods across four datasets.
For ImageNet-1K, we follow the training settings defined in ConvMixer [], training all models for 300 epochs.
Our proposed methods, both Imp.-3 and Imp.-5, demonstrate comparable—if not superior—performance compared to mimetic initialization.
Particularly on smaller-scale datasets like CIFAR-10, CIFAR-100, and SVHN, known to pose challenges for ViT models, our method consistently exhibits $2\%$ to $4\%$ improvement compared to Trunc Normal.
Notably, our method maintains to perform well on large-scale datasets like ImageNet-1K, which shows that our structured initialization keeps the flexibility of the attention map even when learning from large-scale data.

### 5.3 Larger Models

Although our method demonstrates impressive performance when training ViT-T on small-scale datasets, the model ViT-T only has 3 heads, which falls short of the requirements defined in 1.
To better showcase the advantage of using our initialization strategy, we increase the number of heads to 8 in ViT-T, denoted as ViT-T/h8.
In addition to the experiments with ViT-T, we also tested our method on the small ViT model (ViT-S).
The configuration of ViT-S includes an embedding dimension of 384, a depth of 12, and 6 heads, denoted as ViT-S/h6.
Furthermore, we increase the number of heads to 16 and denote this model as ViT-S/h16.
The results on CIFAR-100 are shown in Tab. 3.

Table: Table 2: Classification accuracy(%) of ViT-T using different initialization methods on CIFAR-10, CIFAR-100, SVHN and ImageNet-1K. Red number indicates accuracy decrease, and green number indicates an increase in accuracy. Note that we compare the performance to the Trunc Normal initialization method (shaded in gray).

Table: Table 3: Classification accuracy(%) of ViT-T/h3, ViT-T/h8, ViT-S/h6 and ViT-S/h16 using different initialization methods on CIFAR-100. Red number indicates accuracy decrease, and green number indicates an increase in accuracy. Note that we compare the performance to the Trunc Normal initialization method (shaded in gray).

Table: Table 4: Classification accuracy(%) of different pseudo input on CIFAR-10. Green shaded row indicates the best choice of pseudo inputs when achieving the best average accuracy. “PE” denotes positional encoding, and “G” represents random sampling from the Gaussian distribution.

As the model size increases, particularly with a higher number of heads, our initialization method demonstrates improved and more stable performance, bringing a larger gap between other initialization methods.
This performance increase proves our theory (see 1) regarding the expressibility of spatial mixing matrix: more heads provide more linearly independent filters.
For instance, when the number of heads is 3, as in ViT-T/h3, each layer contains only 3 unique “filters” with each filter having $192\,{/}\,3\,{=}\,64$ copies.
While the number of copies is sufficient, having only 3 unique “filters” is inadequate for forming the filter basis, even for a $3\,{\times}\,3$ random impulse filter.

As we increase the number of heads, we observe an adequate improvement in the performance of our method.
However, maintaining a constant embedding dimension while increasing the number of heads leads to fewer copies per head.
While this may not present a significant issue in ConvMxier as long as the number of copies exceeds the rank of the inputs, a notable challenge arises with multi-head attention: the dimensionality of $\mathbf{Q}$ and $\mathbf{K}$ will decrease to $D\,{/}\,h$ as the number of heads $h$ increases.
Consequently, the rank of $\mathbf{Q}$ and $\mathbf{K}$ diminishes considerably, making it more challenging for the low-rank approximation $\mathbf{Q}\mathbf{K}^{T}$ to learn an effective attention map.

This phenomenon may explain why the Kaiming Uniform and Trunc Normal methods occasionally exhibit inferior performance as the number of heads increases.
For the mimetic initialization, the situation is potentially more problematic, as it utilizes SVD to solve for a low-rank $\mathbf{Q}$ and $\mathbf{K}$.
As the number of heads increases, resulting in a lower rank, the approximation error grows, further deviating the actual $\mathbf{Q}\mathbf{K}^{T}$ from the anticipated value.
In contrast, our initialization strategy employs an iterative optimization method, which helps mitigate errors arising from low-rank approximations.
Consequently, our method benefits more when applied with a larger number of heads.

### 5.4 Pseudo Input

It is important to note that no real data is involved in our initialization strategy, distinguishing it from pre-training methods.
Incorporating real data into the initialization will unnecessarily complicate the optimization of attention parameters, thereby increasing computational complexity.
Therefore, we use pseudo input to optimize attention parameters $\mathbf{Q}_{\text{init}}^{\hphantom{T}}$ and $\mathbf{K}_{\text{init}}^{\hphantom{T}}$ using gradient descent.
Among various options for pseudo input, sinusoidal absolute positional encoding (PE) proves to be a good choice considering that (1) it remains independent of data, and (2) it inherently embeds the spatial information.
Nonetheless, alternative choices for pseudo inputs are worth exploring.

Here, we test different pseudo inputs, including random inputs sampled from Gaussian or Uniform distributions.
As discussed in Sec. 4.2, these pseudo inputs are sampled once and kept fixed during the optimization process.
We show this ablation results in Tab. 4.
Note that we only include results for Gaussian distributions (G) that sampled from zero mean and a standard deviation of 0.5, truncated at [-2, 2].
Results for Uniform distributions are included in the supplementary material.
Apart from using the same pseudo input for all network layers, we also explored using different pseudo inputs for individual layers.

During the training of the ViT model, the inputs to different network layers differ in that the input to the first layer is a patch embedding of the image along with a positional encoding, whereas the input to the following layers is the output feature embedding obtained from the previous layer.
To accommodate these differences in network layers, we explore using different pseudo inputs for different layers.
Notably, we treat the first layer separately from the following layers.
Additionally, we explore two configurations regarding the attention parameters: one where the same optimized parameters are utilized across the entire network, and another where these parameters are optimized separately for each layer. These configurations are labeled as “same” and “different” for $\mathbf{Q}_{\text{init}}^{\hphantom{T}}$ and $\mathbf{K}_{\text{init}}^{\hphantom{T}}$.
Consequently, we test different pseudo inputs in eight different configurations with ViT-T/h3 and ViT-T/h8.

From Tab. 4, it is evident that using PE as pseudo input yields the best performance.
Switching from all PEs to PE+G results in a 1%-3% performance drop, while using random inputs alone leads to the poorest performance, with accuracy dropping by 4%.
Additionally, substituting PE with alternative pseudo inputs results in worse performance compared to modifications in the first layer, since there are 11 “following layers” in ViT-T.

Figure: Figure 3: Visualization of attention maps in ViT-T using our impulse initialization method, mimetic [], and random [] initializations. Red boxes highlight zoomed-in details of the $48{\times}48$ upper left corner in attention maps. White boxes indicate the main diagonal blocks of the zoomed-in attention maps. Our structured initialization method offers off-diagonal attention peaks aligned with the impulse structures, whereas mimetic initialization primarily strengthens the main diagonal of the attention map. Random initialization shows little to no patterns.
Refer to caption: extracted/2404.01139v1/vis_attn_04.png

### 5.5 Attention Maps

To show the effectiveness of our initialization on constraining the structure of attention maps, we show the averaged attention maps across CIFAR-10 training data using ViT-T/h3 at 1st, 6th, and 12th layers in Fig. 3.
The attention map showed here does not have an exact pattern as seen in impulse filters since we use pseudo inputs to optimize the initial attention parameters.
Nevertheless, there exists a clear pattern of convolutional structures.
As the network layer becomes deeper, this convolutional structure becomes less visible.

Table: Table 5: Classification accuracy(%) of ConvMixer (depth 8) with different filter sizes, embedding dimensions on CIFAR-10.

### 5.6 Comparing ViT with ConvMixer

To validate our findings discussed in Sec. 3 regarding the effectiveness of random filters, we train ConvMixer [] models with an embedding dimension of 256, a depth of 8, and a patch size of 2 on the CIFAR-10 dataset, using filter size of 3, 5, and 8.
We follow the same configurations as defined in Sec. 5.1, except for setting the learning rate to 0.01, and the number of epochs to 100.
Additionally, we tested with a version of ConvMixer with an embedding dimension of 512.
The results are shown in Tab. 5.

We tested on the end-to-end trained ConvMixer along with three different initialization methods: random, impulse, and box.
Please note that the three initialization methods only initialize the spatial convolution filters without training.
Specifically, the box filters use all ones, serving as an average pooling function.

In general, random and impulse initialization achieve comparable accuracy compared to the end-to-end trained model, while box initialization exhibits inferior performance.
This discrepancy can be attributed to the deficient rank of box filters, as they lack $f^{2}$ linearly independent filters, unlike random and impulse initialization, which can form the basis of the filter space.

For ConvMixers with the same embedding dimension, the performance gap between trained filters and random or impulse widens as the kernel size increases.
As we discussed in Sec. 5.3, as the kernel size increases, more unique filters are needed to form the basis of filter space.
Consequently, each head (unique filter) has fewer copies, making it difficult to match the rank of inputs, thus failing to meet the condition in Eq. 3.
When the embedding dimension doubles, the performance gap between trained filters and random or impulse filters diminishes with the same kernel size.
However, models with larger kernel sizes still tend to have a bigger gap due to an insufficient number of copies for each unique filter.

Our method is motivated by the similarity between ViT and ConvMixer.
To show this connection, we train ViTs with similar configurations to ConvMixer as described in Sec. 5.6.
Specifically, we train ViTs of a depth of 8 with embedding dimensions of 256 and 512.
The number of heads is from 4 to 32.
Results of different initialization methods are shown in Tab. 6.

Our impulse initialization methods demonstrate superior performance across nearly all configurations.
Especially, our method achieves a top accuracy of $90.67\%$ and $91.87\%$ with an embedding dimension of 256 and 512, respectively, significantly outperforming other initialization methods.
Moreover, our method achieves results on par with end-to-end trained ConvMixers ($91.76\%$ and $92.82\%$) of equal depth and embedding dimensions.
In addition, these results with different numbers of heads validate our discussions in Sec. 5.3 regarding the impact of the number of heads on the performance.

Table: Table 6: Classification accuracy(%) of ViT (depth 8) with a different number of heads, embedding dimensions on CIFAR-10.

## 6 Limitations

(1) Pseudo input.
Although positional encoding is a suitable choice for pseudo input considering its simplicity and its data-independent nature, there may exist even simpler choices for the pseudo input.
Since PE does not account for network depth, the actual attention maps shown in Fig. 3 appear more blurred as the network depth increases.

(2) Pre-optimization step.
The iterative optimization method for solving initial attention parameters is simple and fast, surpassing SVD in mitigating the low-rank approximation errors.
However, leveraging the nonlinearity of the softmax function during initialization may cause network weights to enter a gradient plateau, hindering training in subsequent tasks such as classification.

(3) Hard constraints.
Our 1 is based on the presumption that the filters can at least form the basis of filter space, a characteristic inherent in CNNs.
However, in ViTs, the limited number of heads may be inadequate to span the filter space of a small kernel.
Finding better adaptions in this scenario remains a challenge.

(4) Value initialization.
Our method does not consider the initialization for the value weights $\mathbf{V}$ in ViT.

## 7 Conclusion

In this paper, we propose a new ViT initialization strategy—convolutional structured impulse initialization—to address the problem that ViTs are difficult to train on small-scale datasets.
Our initialization requires no off-line knowledge of pre-trained models on large-scale datasets (mimetic or empirical).
Our strategy is instead inspired by the architectural inductive bias of convolution, without the need for an architectural modification for ViT.
Unlike traditional generative initialization strategies focusing on the distribution of parameters, our method constrains the structure of attention maps within ViT.
After a careful study of why random spatial convolution filters work, we opt to initialize self-attention maps as random impulse convolution filters, which reinterpret the architectural inductive bias in CNNs as an initialization bias and preserve the architectural flexibility of transformers.
We validate our methods across various datasets, achieving state-of-the-art performance for ViT trained on small-scale datasets.

## Appendix 0.A Convolutional Represetation Matrix

In Sec. 3, we interchangeably use the terms convolution filter $\mathbf{h}$ and convolution matrix $\mathbf{H}$.
Additionally, we represent the impulse filter as a convolutional matrix.
Here, we offer a detailed explanation of the relationship between the convolutional filters and the convolutional matrices.

Let us define a 2D convolution filter as $\mathbf{h}\,{\in}\,\mathbb{R}^{f{\times}f}$ with elements

$$ $\mathbf{h}=\left(\begin{matrix}h_{11}&\cdots&h_{1f}\\ \vdots&\ddots&\vdots\\ h_{f1}&\cdots&h_{ff}\end{matrix}\right).$ (9) $$

When $\mathbf{h}$ is convolved with an image $\mathbf{x}\,{\in}\,\mathbb{R}^{H{\times}W}$, this convolution operation is equivalent to a matrix multiplication

$$ $\mbox{vec}(\mathbf{h}\,{*}\,\mathbf{x})\,{=}\,\mathbf{H}\,\mbox{vec}(\mathbf{x }),$ (10) $$

where $\mathbf{H}$ is composed from the elements in $\mathbf{h}$ and zeros in the following format:

$$ $\mathbf{H}=\left(\begin{matrix}\mathbf{F_{1}}&\mathbf{F_{2}}&\cdots&\mathbf{F_ {f}}&\mathbf{0}&\mathbf{0}&\cdots&\mathbf{0}\\ \mathbf{0}&\mathbf{F_{1}}&\mathbf{F_{2}}&\cdots&\mathbf{F_{f}}&\mathbf{0}& \cdots&\mathbf{0}\\ \vdots&\ddots&\ddots&\ddots&\ddots&\ddots&\ddots&\vdots\\ \mathbf{0}&\cdots&\mathbf{0}&\mathbf{F_{1}}&\mathbf{F_{2}}&\cdots&\mathbf{F_{f }}&\mathbf{0}\\ \mathbf{0}&\cdots&\mathbf{0}&\mathbf{0}&\mathbf{F_{1}}&\mathbf{F_{2}}&\cdots& \mathbf{F_{f}}\end{matrix}\right),$ (11) $$

where

$$ $\mathbf{F_{i}}=\left(\begin{matrix}h_{i1}&h_{i2}&\cdots&h_{if}&0&0&\cdots&0\\ 0&h_{i1}&h_{i2}&\cdots&h_{if}&0&\cdots&0\\ \vdots&\ddots&\ddots&\ddots&\ddots&\ddots&\ddots&\vdots\\ 0&\cdots&0&h_{i1}&h_{i2}&\cdots&h_{if}&0\\ 0&\cdots&0&0&h_{i1}&h_{i2}&\cdots&h_{if}\end{matrix}\right),$ (12) $$

for $i\,{=}\,1,2,\,{\dots}\,,f$.
$\mathbf{F_{i}}$s are circulant matrices and $\mathbf{H}$ is a block circulant matrix with circulant block (BCCB).
Note that convolutions may employ various padding strategies, but the circulant structure remains consistent.
Here, we show the convolution matrix without any padding as an example.

## Appendix 0.B Additional Results

Table: Table 7: Classification accuracy(%) of different pseudo input on CIFAR-10. Green shaded row indicates the best choice of pseudo inputs when achieving the best average accuracy. “PE” denotes positional encoding, and “G” represents random sampling from the Gaussian distribution. “Imp.-3” represents a $3{times}3$ impulse filter, and “Imp.-5” denotes a $5{\times}5$ impulse filter.

Figure: Figure 4: Visualization of attention maps in ViT-T using our impulse initialization method, mimetic [], and random [] initializations. Red boxes highlight zoomed-in details of the $48{\times}48$ upper left corner in attention maps. White boxes indicate the main diagonal blocks of the zoomed-in attention maps. Our structured initialization method offers off-diagonal attention peaks aligned with the impulse structures, whereas mimetic initialization primarily strengthens the main diagonal of the attention map. Random initialization shows little to no patterns.
Refer to caption: extracted/2404.01139v1/supp_attn_03.png

Figure: Figure 5: Visualization of attention maps in ViT-T using our impulse initialization method, mimetic [], and random [] initializations. Red boxes highlight zoomed-in details of the $48{\times}48$ upper left corner in attention maps. White boxes indicate the main diagonal blocks of the zoomed-in attention maps. Our structured initialization method offers off-diagonal attention peaks aligned with the impulse structures, whereas mimetic initialization primarily strengthens the main diagonal of the attention map. Random initialization shows little to no patterns.
Refer to caption: extracted/2404.01139v1/supp_attn_04.png

Table: Table 8: Classification accuracy(%) of ViT (depth 8) with a different number of heads, embedding dimensions on CIFAR-10.

### 0.B.1 Pseudo Input

We offer additional results for the ablation study on pseudo input with random inputs sampled from a Uniform distribution in Tab. 7.
The trend is aligned with the findings in the main paper for random inputs sampled from a Gaussian distribution.
However, the performance of using random inputs sampled from a Uniform distribution is worse than those using a Gaussian distribution, since the real input data is more likely to follow a Gaussian distribution.

### 0.B.2 Relationship between Head Numbers and Embedding Dimension

In the main paper, we have discussed the relationship between the number of heads and the embedding dimensions.
We have provided the results with embedding dimensions of 256 and 512 for a ViT with depth 8.
Here we show additional results with embedding dimensions of 64 and 128 in Tab. 8.
The trend is consistent with the discussions in the main paper.

### 0.B.3 Attention Maps

Here we provide additional visualization of the attention maps for all 12 layers in Fig. 4 and Fig. 5.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Why not apply structured initialization to the value (V) component of self-attention? Additionally, how are the feed-forward network (FFN) layers, normalization layers, and projection layers initialized in the proposed method?
soundness: 3
strengths: 1.Theoretical Foundation: The structured initialization method is based on solid theoretical analysis rather than just empirical results, providing a strong rationale for its effectiveness.

2.Performance Improvements: The method consistently shows significant performance improvements over conventional ViT initialization methods in small-scale datasets, which is a notable achievement.
summary: This paper addresses the challenge of applying Vision Transformers (ViTs) to new domains with small datasets, where Convolutional Neural Networks (CNNs) typically excel due to their inherent architectural inductive bias. The authors propose a novel approach that reinterprets CNN's architectural bias as an initialization bias for ViTs, termed "structured initialization." Unlike traditional ViT initialization methods that rely on empirical results or attention weight distributions, this method is theoretically grounded and constructs structured attention maps. The paper demonstrates that this structured initialization enables ViTs to achieve performance comparable to CNNs on small-scale datasets while retaining the flexibility to perform well on larger-scale applications. The proposed method shows significant improvements over conventional ViT initialization across several small-scale benchmarks, including CIFAR-10, CIFAR-100, and SVHN, and maintains competitive performance on large-scale datasets like ImageNet-1K.
weaknesses: 1. In terms of innovation, the Transformer architecture was initially designed to minimize inductive bias. The author's attempt to incorporate structural biases from CNNs into the Transformer seems to go against the original intent of the Transformer design, which could be seen as a step backward for the evolution of Transformer models.

2. The variety of experimental backbones is somewhat limited. It would be beneficial to conduct experiments with DeiT or Swin-Transformer to compare results. Furthermore, aside from classification tasks, it would be interesting to test the method on detection or segmentation tasks to further evaluate its versatility and effectiveness.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: -  Does the structured initialization limit the model's ability to learn better representation? It would be better to provide some representation-level analysis using metrics like Centered Kernel Alignment (CKA) similarity [1].

[1] Kornblith, Simon et al. “Similarity of Neural Network Representations Revisited.” ICML 2019.
soundness: 3
strengths: - The paper is easy to follow.
- As much work has been trying to introduce convolutional design into the ViT model, this paper provides an interesting viewpoint that initializing the attention map as CNNs can also help to introduce the inductive bias and subsequentially improve the performance of trained ViT on small-scale datasets.
- A theoretical explanation is provided to show the connection between the structural initialization in ViT and inductive bias in CNNs.
- Some special designs like more heads and various initialization conv kernel sizes are adopted.
summary: The paper presents an approach to initialize ViT through structured initialization of attention maps. By incorporating CNN-like inductive biases during initialization, it aims to combine the local spatial processing capabilities with the global relationship learning of attention mechanisms and take advantage of CNNs' inductive bias. Experimental results on several small-scale datasets validate its effectiveness.
weaknesses: - The fundamental approach of forcing attention maps to mimic convolutional kernels seems to contradict the core advantage of attention mechanisms, as their advantage is to learn flexible, dynamic global relationships. It would be better to justify why structured initialization is preferred over simply incorporating convolutional blocks into the architecture, which would be a more straightforward solution. 
- It would be better to provide more analysis of why this approach is better compared to well-established solutions: 
  - Transfer learning from large-scale pre-trained ViTs
  - Hybrid architectures combining convolution and attention
- The optimization process required for initializing attention maps introduces additional computational overhead during training, and one needs to further choose the optimizer for initialization and conv kernel size (as well as other hyperparameters for different model sizes), which makes it impractical.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. Why Impulse Filters? The reasoning behind choosing impulse filters (instead of other structured filters) for initializing attention maps could be explained in more detail. Are impulse filters the best possible choice, or could other filter types provide better generalization?

2. How does this structured initialization affect the deeper layers of ViTs after finetuning? Figure 3 is quite interesting. Do the constraints imposed by impulse filters affect the deeper layers’ ability to fine-tune long-range dependencies? The paper does not discuss the long-term effects of this initialization on the network’s convergence.
soundness: 2
strengths: Structured architecture achieves good performance across both small and large datasets, which demonstrates its scalability and flexibility.
summary: This paper introduces a novel method for improving the performance of ViTs when trained on small-scale datasets by incorporating structured initialization. The authors identify that ViTs struggle with small datasets compared to CNNs, which benefit from inherent inductive biases. By reinterpreting the architectural bias in CNNs as an initialization bias for ViTs, the authors propose a "structured initialization" method that results in structured attention maps for ViTs. The key contribution lies in the use of random convolutional impulse filters to guide the initialization process. The method is theoretically justified and empirically validated across serveral benchmarks. The paper demonstrates that structured initialization yields performance improvements on small datasets without compromising ViT’s flexibility on larger datasets.
weaknesses: 1. The core argument of the method is that the convolutional structure can be transferred to the attention mechanism in transformers by initializing the attention maps with random impulse filters. However, this analogy between convolutional layers in CNNs and the attention mechanism in ViTs may be overly simplistic. CNNs' convolutional filters are spatially local and fixed in structure, while attention in ViTs is meant to capture long-range dependencies and is more flexible. This difference is crucial, and the method does not seem to fully address how imposing a rigid, convolution-like structure at initialization aligns with the flexibility that the attention mechanism needs. The convolution structure might limit the model's ability to learn long-range dependencies that are essential to the transformer. The claim that random impulse filters can replace learned convolutional filters is somewhat true for CNNs under certain conditions (like ConvMixers), but applying this to ViTs is more challenging. The attention mechanism is a more complex and dynamic operation compared to convolutions, and it’s unclear if the same approximation can hold. In practice, imposing a convolution-like structure might hinder the attention mechanism's ability to adapt during training.

2. The paper proposes that impulse filters, combined with the softmax operation, can initialize the attention maps. The softmax function ensures that all outputs are non-negative, which is a crucial difference from convolutions, which can have both positive and negative values. Random convolutional filters may contain both positive and negative values, while softmax output does not. This inconsistency could cause issues. The authors acknowledge this (stating the filters must be positive), but they do not provide a deep exploration of how this might affect the quality or flexibility of the learned attention maps. Relying on impulse filters could reduce the model's expressivity, especially if the initialized filters are too rigid and only positive-valued patterns are learned initially.

3. The authors propose an iterative optimization process to solve for the initial values of $Q_{init}$ and $K_{init}$ such that the resulting attention maps resemble impulse convolution filters. The optimization is based on a pseudo-input, which is generated from positional encodings rather than actual data. This could introduce an unwanted bias into the model's initial learning process. While using positional encoding as pseudo-input is an interesting idea, the paper does not adequately explore how different choices of pseudo-inputs affect the results or whether using actual training data for initialization would be a better alternative.

4. The optimization process described is more computationally expensive (up to 10,000 iterations using Adam) compared with traditional initialization methods. This added complexity raises the question of whether the benefits of structured initialization outweigh the cost, especially given that the improvements on large datasets are marginal. There is no discussion of the computational cost vs. benefit of this method compared to standard initialization techniques.

5. The use of random impulse convolution filters assumes that locality is always important, but this assumption may not hold in tasks where global context is critical. In CNNs, locality is useful because of the hierarchical structure of learned features. However, in transformers, the attention mechanism is specifically designed to handle long-range dependencies. By forcing the model to start with local dependencies (via impulse filters), the authors may inadvertently restrict the model's ability to learn global features early in training, leading to potential issues in tasks where global context is key from the beginning.

6. The method relies on several hyperparameters, including filter size (3x3 or 5x5) and the number of iterations for optimization. However, the choice of these hyperparameters is not adequately justified or explored. The method's performance is likely sensitive to these parameters, but there is no thorough analysis of how variations in filter size or optimization parameters affect results. Given the complexity of the initialization process, these aspects should have been investigated in detail to ensure the method’s robustness.

Meta-review: This paper proposes structured initialization to improve the performance of Vision Transformers (ViTs) on small-scale datasets. The approach reinterprets architectural bias from CNN as an initialization bias for ViTs. This method enables ViTs to achieve competitive performance with CNNs on small datasets, such as CIFAR-10, CIFAR-100, and SVHN, while maintaining flexibility and strong performance on larger datasets like ImageNet-1K. The paper provides both theoretical justification and empirical validation, demonstrating improvements over conventional ViT initialization methods.

After the review process, all reviewers unanimously voted for rejection, citing several concerns, including limited experiments, marginal improvements on large datasets, and the generalizability of the approach to other datasets.

Decision: Reject