IBiT: Utilizing Inductive Biases to Create a More Data Efficient Attention Mechanism

## Abstract

Abstract In recent years, Transformer-based architectures have become the dominant method for Computer Vision applications. While Transformers are explainable and scale well with dataset size (Dosovitskiy et al., 2020 ) , they lack the inductive biases of Convolutional Neural Networks (LeCun et al., 1989 ) . While these biases may be learned on large datasets, we show that introducing these inductive biases through learned masks allow Vision Transformers to learn on much smaller datasets without Knowledge Distillation. These Transformers, which we call Inductively Biased Image Transformers (IBiT), are significantly more accurate on small datasets, while retaining the explainability Transformers.

## 1 Introduction

Vision Transformers (Dosovitskiy et al., 2020) have become established as the state-of-the-art Computer Vision backbone. Such self-attention (Vaswani et al., 2017) based networks have also shown excellent pre-training performance on datasets like ImageNet (Deng et al., 2009). Unfortunately, Vision Transformers require large amounts of data (14M-300M images) during pre-training to outperform Convolutional Neural Networks (Dosovitskiy et al., 2020). In more specialized domains where transfer learning on large image recognition datasets is not possible, and dataset size is often quite small (Zhai et al., 2020), Convolutional Neural Networks (CNNs), more specifically ResNets (He et al., 2015), are still the dominant model architecture.

This is because CNNs possess inductive biases, namely translational equivariance and locality (Dosovitskiy et al., 2020), which allow such models to achieve better results with smaller amounts of data.

Using this intuition, we propose Inductively Biased Image Transformers, or IBiTs, a model that can approximate the inductive biases of a CNN, allowing it to learn better on small datasets. This is possible through the application of learnable masks to the attention layers during self-attention (Vaswani et al., 2017), allowing IBiTs to mimic a CNN’s biases with a Transformer architecture. Using a technique called Rank Approximation, we significantly reduce the parameters required to implement these learnable masks by using low rank approximations.

When all of these methods are implemented, the final model is able to outperform comparable Transformer-based methods (Touvron et al., 2021) by two percentage points when trained solely on ImageNet, achieving state-of-the-art image transformers performance while retaining explainability and scalability.

## 2 Related Work

Vision transformers were first introduced by Dosovitskiy et al. (2020) and pretrained Vision Transformers have seen use in a wide variety of applications.

DeiTs were the first models to explore generalizing Transformers for smaller datasets using knowledge distillation. Knowledge distillation uses strong teacher models to introduce inductive biases into Vision Transformers. Although we also introduce inductive biases, our method, which uses learnable masks, does not use strong teacher models which often need large amounts of data and computational power to train.

Another method to induce inductive biases in Transformers models involves the use of learned relative positional encodings to make self-attention layers have the capacity to model CNN layers (Cordonnier et al., 2020). However, this method does not perform as well as Vision Transformers. It also requires the number of heads in the self-attention layer to be equivalent to the square of the filter size being approximated (eg. A transformer would need 9 heads to approximate a convolutional layer with a filter size of 3x3).

To improve the performance of relative positional encodings, further research involved using a weighted sum between relative positional encodings and normal self-attention (d’Ascoli et al., 2022). This method outperformed DeiTs by 0.9 percent. However, ConViTs, as the models are called, slightly modify the architecture of DeiTs, using four heads instead of the three normally used by DeiTs.

Our method, IBiT, does not use relative positional encodings, instead using learnable masks. By using this new approach, IBiTs achieve improved performance with fewer parameters and faster performance. IBiTs do not require any underlying assumptions surrounding model architecture like ConViTs do, meaning it is easy to modify existing models to incorporate our methods, by just converting a normal self-attention layer into a learned mask self attention layer with the exact same structure. Our method also performs significantly better than both these approaches of introducing inductive biases into Transformers, while retaining the explainability of attention-based models.

## 3 Model Architecture

### 3.1 Approximating Convolution With Self-Attention

To approximate convolution with self-attention, we begin with convolution with a single filter and a single channel.

$$ $Y_{i,j}=\sum_{k=0}^{f}\sum_{l=0}^{f}X_{(i+k),(j+l)}*W_{k,l}$ $$

In this equation, $f$ represents the filter size, $W$ represent the filter weights, $X$ is a 2-D input image, and $Y$ is the result of the convolution operation.

The equation for a single head of self-attention is as follows:

$$ $Y_{i}=\sum_{m=0}^{size}X_{m}*W_{i,m}$ $$

where $X$ is a 1-D representation created by flattening the image, $size$ is the length of the 1-D representation of the image, and $W$ is the attention map generated by self attention.

By re-indexing $size$ in terms of $height$ and $width$ and re-indexing $i$ in terms of $i$ and $j$ in the 2-D image, we can create the following equation, which is equivalent to self-attention for a single head.

$$ $Y_{i*width+j}=\sum_{m=0}^{height}\sum_{n=0}^{width}X_{m*width+n}*W_{i*width+j,m*width+n}$ $$

Setting the row $W_{i*width+j}$ to equal $W_{k,l}$ where $k=m\in\{0,1,...,f\}$ and $l=n\in\{0,1,...,f\}$, and setting all other entries in the matrix $W$ to zero, the equation simplifies to the equation below.

$$ $Y_{i*width+j}=\sum_{k=0}^{f}\sum_{l=0}^{f}X_{(i+k)*width+(j+l)}*W_{k,l}$ $$

Figure: Figure 1: Visual Representation of Convolution through Matrix Multiplication
Refer to caption: /html/2509.22719/assets/Images/ConvMatrix.png

This equation is equivalent to the equation for convolution, where the output is flattened into a 1-D representation. For a visual representation of the process, see Figure 1.

### 3.2 Low Rank Approximation

To introduce inductive biases into self-attention the weights of the attention map at row $i*width+j$ should be equal to $W_{k,l}$ where $k=m\in\{0,1,...,f\}$ and $l=n\in\{0,1,...,f\}$.

This means that the attention map has a rank of $height*width$. In order for the queries and keys to reproduce an attention map with inductive biases, both the queries or the keys need to have a rank of $height*width$. Since the maximum rank of a matrix is the minimum of the number of rows or columns, and the sequence length of the unrolled image is $height*width$, the embedding size of the matrix must be equal to or larger than the sequence length for inductive biases to arise in the self-attention layer.

In practice, this is rarely the case. Increasing the embedding size increases the computational cost, and most common architectures have significantly larger sequence lengths than embedding sizes. Dedicating such a large proportion of the network to represent inductive biases would also reduce the performance of the model.

To mitigate these problems, we introduce rank approximation. Due to the sparsity of the attention map, by rolling each row back by its row number, we are able to reduce the rank of the attention map to the square of the filter size (i.e., a $3\times 3$ filter could be represented by a matrix with rank $9$).

Since inductive biases do not depend on the content of the image, we use two separate learnable matrices called sub-masks. These matrices have a shape of $(height*width)\times mask_fidelity$, where mask fidelity is the square of the filter size. Through a dot product, these sub-masks, A and B respectively, become the inductive bias matrix.

$$ $W^{inductive}=A\bullet B^{T}$ $$

We then undo the rank-reducing rolling operation and multiply this inductive bias matrix and the attention map element-wise, essentially treating the unrolled inductive bias matrix as a mask on the attention map.

This ensures that our inductive biases are parameter-efficient and persistent regardless of image content.

### 3.3 Mask Training

We initially set the inductive biases to an inductively biased attention map. Instead of training our masks to initially represent a Convolutional kernel, we use the same rank approximation methodology to approximate a Gaussian kernel.

Figure: Figure 2: Convolutional Inductive biases versus Gaussian Kernel Inductive Biases
Refer to caption: /html/2509.22719/assets/Images/FiltervsRBF.png

To do this, the initial weight matrices are trained to approximate the unrolled attention map using the algorithm below.

Figure: Algorithm 1 Mask Weights Training Procedure

Figure: Figure 3: Visual Representation of LMSA Layer
Refer to caption: /html/2509.22719/assets/Images/RankApproximation.png

The algorithm for the full Learned Mask Self Attention layer is described in the page below, while Figure 3 presents a visual representation of the Learned Mask Self Attention process.

Figure: Algorithm 2 LMSA Layer

## 4 Experiments

### 4.1 Experimental Setup

Our model architecture is the same as the DeiT model architecture, with the same number of heads per layer, and the same number of layers. We replace all Multi-Head Self Attention layers with Learned Masked Self Attention. We also remove the CutMix data augmentation technique (Yun et al., 2019), because it hinders the learning of inductive biases. Finally, we remove stochastic depth on the feed-forward layer, since it improves ViT performance. We also compare our results to ConViTs, which have one more head than both DeiTs, and our model. We detail the hyper-parameters and other model information in the table below. We train our models on ImageNet-1k for 300 epochs, and use the reported results on ImageNet-1k for other models.

**Table 1: Training Configurations**
| Model Type | # of Layers | # of Heads | D${}_{\mbox{model}}$ | Learning Rate | # of Params |
| --- | --- | --- | --- | --- | --- |
| IBiT (Ours) | 12 | 3 | 192 | 0.001 | 6M |
| DeiT | 12 | 3 | 192 | 0.001 | 6M |
| ConViT | 12 | 4 | 192 | 0.001 | 6M |

For our scaling experiments, we compare DeiTs and ConViTs on varying percentages of the ImageNet dataset to show dataset sample efficiency.

**Table 2: Scaling Configurations**
| Model Type | # of Layers | # of Heads | D${}_{\mbox{model}}$ | Learning Rate | # of Params |
| --- | --- | --- | --- | --- | --- |
| IBiT (Ours) | 12 | 3 | 192 | 0.001 | 6M |
| DeiT | 12 | 3 | 192 | 0.001 | 6M |

To show explainability, we use Rollout Attention to highlight which pixels the model is using to classify the input (Abnar & Zuidema, 2020).

### 4.2 Results

Here we present the results of our experiments. Our model, the IBiT, outperforms both DeiTs and ConViTs significantly.

**Table 3: Training Results**
| Model Type | Accuracy | # of Params |
| --- | --- | --- |
| IBiT (Ours) | 74.2 | 6M |
| DeiT | 72.2 | 6M |
| DeiT (Our Reproduction) | 71.5 | 6M |
| ConViT | 73.1 | 6M |

Figure: Figure 4: Training and Scaling Curves for IBiT and DeiT

### 4.3 Explainability

Here are some representative attention maps between the CLS token and the image, using Rollout Attention (Abnar & Zuidema, 2020). More attention maps can be found in Appendix B.

Figure: Figure 5: Representative Attention Maps using Rollout Attention
Refer to caption: /html/2509.22719/assets/Images/Explainablity.png

The model typically attends to the subject of the image. Through attention maps, we show that the explainability of Transformers is preserved, despite the use of Learnable Masks.

## 5 Ablation Results

Here we present the results for our different ablation configurations.

**Table 4: Ablation Configurations**
| Configuration | Learnable Masks | CutMix | Accuracy |
| --- | --- | --- | --- |
| A (IBiT) | ✓ | ✗ | 74.2 |
| B | ✓ | ✓ | 72.2 |
| C (DeiT) | ✗ | ✓ | 71.5 |

for our ablation, we ablate on the Learnable Mask Self Attention Layer, and on CutMix. We find that the use of CutMix degrades model performance significantly. By mixing different portions of the image, CutMix hinders the formation of inductive biases in our model, leading to reduced performance.

## 6 Discussion

### 6.1 Learnable Masks

Here we show the learnable masks as they evolve throughout the training process. You can see how later layers have less of the initial inductive biases, while earlier layers learn multiple strong inductive biases quickly.

Figure: Figure 6: Learnable Mask Visualization for Trained IBiT Network
Refer to caption: /html/2509.22719/assets/Images/LMSAHeads.png

This motivates the idea that learned masks may not even be needed in later layers, though we leave further architectural modifications to future work.

### 6.2 Gradient Biasing

One reason we believe models using learnable masks perform better is the property of gradient biasing.

Since we set the mask to mimic the inductive biases of a Gaussian kernel, and the mask is applied through element-wise multiplication, the gradient update on the attention mask can be calculated as $W^{mask}\frac{\partial J}{\partial W^{attention}}$

This means that at each weight of the mask, the gradient, $\frac{\partial J}{\partial W^{attention}}$ is scaled by the mask weight. When this mask represents a gaussian kernel, this means that values close to each are scaled to have a larger gradient, and hence the model learns to use these values better. These inductive biases are visible in the learnable mask even during late stages of training (Figure 5 and Appendix C) highlighting the effect of gradient biasing.

## 7 Conclusion

Clearly, using learnable masks preserves the explainability of regular Transformers, while being significantly more accurate on ImageNet. These models are significantly more sample efficient compared to other other Transformer based models. We hope the interesting new properties these models possess and further work on parameter efficiency and more efficient learnable masks may lead to significant advancements in the field of explainable, data-efficient computer vision.

## 8 Reproducibility

To ensure that our results are reproducible, we have published our trained models and code on GitHub. We have also put all model hyperparameter configurations for all the model training runs in the appendices.

## References

- Abnar & Zuidema (2020)
Samira Abnar and Willem Zuidema.
Quantifying attention flow in transformers, 2020.
URL [https://arxiv.org/abs/2005.00928](https://arxiv.org/abs/2005.00928).
- Cordonnier et al. (2020)
Jean-Baptiste Cordonnier, Andreas Loukas, and Martin Jaggi.
On the relationship between self-attention and convolutional layers, 2020.
URL [https://arxiv.org/abs/1911.03584](https://arxiv.org/abs/1911.03584).
- Deng et al. (2009)
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.
Imagenet: A large-scale hierarchical image database.
In *2009 IEEE Conference on Computer Vision and Pattern Recognition*, pp. 248–255, 2009.
doi: 10.1109/CVPR.2009.5206848.
- Dosovitskiy et al. (2020)
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.
An image is worth 16x16 words: Transformers for image recognition at scale.
*CoRR*, abs/2010.11929, 2020.
URL [https://arxiv.org/abs/2010.11929](https://arxiv.org/abs/2010.11929).
- d’Ascoli et al. (2022)
Stéphane d’Ascoli, Hugo Touvron, Matthew L Leavitt, Ari S Morcos, Giulio Biroli, and Levent Sagun.
Convit: improving vision transformers with soft convolutional inductive biases*.
*Journal of Statistical Mechanics: Theory and Experiment*, 2022(11):114005, November 2022.
ISSN 1742-5468.
doi: 10.1088/1742-5468/ac9830.
URL [http://dx.doi.org/10.1088/1742-5468/ac9830](http://dx.doi.org/10.1088/1742-5468/ac9830).
- He et al. (2015)
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Deep residual learning for image recognition.
*CoRR*, abs/1512.03385, 2015.
URL [http://arxiv.org/abs/1512.03385](http://arxiv.org/abs/1512.03385).
- LeCun et al. (1989)
Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel.
Backpropagation applied to handwritten zip code recognition.
*Neural Computation*, 1(4):541–551, 1989.
doi: 10.1162/neco.1989.1.4.541.
- Touvron et al. (2021)
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.
Training data-efficient image transformers & distillation through attention, 2021.
URL [https://arxiv.org/abs/2012.12877](https://arxiv.org/abs/2012.12877).
- Vaswani et al. (2017)
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.
Attention is all you need.
*CoRR*, abs/1706.03762, 2017.
URL [http://arxiv.org/abs/1706.03762](http://arxiv.org/abs/1706.03762).
- Yun et al. (2019)
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo.
Cutmix: Regularization strategy to train strong classifiers with localizable features, 2019.
URL [https://arxiv.org/abs/1905.04899](https://arxiv.org/abs/1905.04899).
- Zhai et al. (2020)
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, Lucas Beyer, Olivier Bachem, Michael Tschannen, Marcin Michalski, Olivier Bousquet, Sylvain Gelly, and Neil Houlsby.
A large-scale study of representation learning with the visual task adaptation benchmark, 2020.
URL [https://arxiv.org/abs/1910.04867](https://arxiv.org/abs/1910.04867).

## Appendix

## Appendix A More Explainability Visualizations

Figure: Figure 7: More Representative Attention Maps using Rollout Attention
Refer to caption: /html/2509.22719/assets/Images/ExplainabilityAppendix.png

## Appendix B Mask Visualizations Through Learning

Figure: Figure 8: Learnable Mask Visualization during Training of IBiT Network
Refer to caption: /html/2509.22719/assets/Images/MasksAppendix.png

## Appendix C Hyperparameter Configurations for Trained Models

**Table 5: Training Configurations**
| Model Type | IBiT (Ours) | DeiT (Our Reproduction) |
| --- | --- | --- |
| # of Layers | 12 | 12 |
| # of Heads | 4 | 4 |
| D${}_{\mbox{model}}$ | 192 | 192 |
| Learning Rate | 0.001 | 0.001 |
| Weight Decay | 0.005 | 0.005 |
| Label Smoothing Rate | 0.1 | 0.1 |
| Dropout Rate | 0.0 | 0.0 |
| Drop Path Rate | 0.1 | 0.1 |
| Learning Rate Scheduler | Cosine-Warmup | Cosine-Warmup |
| # of Params | 6M | 6M |
| CutMix $\alpha$ | N/A | 1.0 |
| MixUp $\alpha$ | N/A | 0.8 |

## Peer Reviews

confidence: 4
contribution: 1
initial_rating: 0
presentation: 2
questions: - Is there a reason for not comparing the proposed method with the methods in [1–8]?

- Where do the authors think this method can be applied, and what are the limitations of using distillation or other approaches for small datasets like [10]?

[10] Vision Transformer for Small-Size Datasets
soundness: 1
strengths: - The paper is easy to follow.
- The results show improvement over the DeiT baseline on a well-known dataset such as ImageNet.
- The proposed method is simple to apply and can therefore be easily adopted by the community.
summary: To improve the sample efficiency of vision transformers and enable them to perform effectively on small datasets, the authors propose adding a learned masks to the transformer to incorporate CNN-like inductive bias (spatial inductive bias). The method demonstrates positive results on ImageNet, outperforming the DeiT baseline while using fewer parameters and achieving faster performance.
weaknesses: - The paper omits a significant amount of important related work. Several examples include Swin Transformer[1,2], Conv-based ViTs [3,4,5], and other works that incorporate inductive bias into vision transformers (for instance  MEGA [6], 2D-SSM [7], MaxViT [8], MixFFN and others). These works should be used both as baselines and to better differentiate the proposed method.
- There is no analysis of latency, FLOPs, or memory usage, during either training or inference.
- Further analyses are missing, such as evaluating robustness (Imagenet-A, Imagenet-E, and others) and beyond-classification performance (segmentation, generation).

___

**References:**

[1] Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. ICCV 2021

[2] Swin Transformer V2: Scaling Up Capacity and Resolution. CVPR 2022

[3] CMT: Convolutional Neural Networks Meet Vision Transformer. CVPR 2022

[4] Early Convolutions Help Transformers See Better. NeurIPS 2021

[5] CoAtNet: Marrying Convolution and Attention for All Data Sizes. NeurIPS 2021

[6] Mega: Moving Average Equipped Gated Attention. ICLR 2023.

[7] 2D-ssm a general spatial layer for visual transformers. ICLR 2024

[8]  MaxViT: Multi-Axis Vision Transformer. ECCV 2022

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 1
questions: The presentation of this paper should be largely improved.
soundness: 2
strengths: - The authors mathematically derive that convolution operations can be exactly implemented using specific sparse structures in the attention weights, thereby establishing a mathematical connection between self-attention and convolution.
- The attempt to introduce convolution’s translational equivariance and locality into ViT, with the goal of improving performance on small datasets, is considered meaningful.
- Experimental results show that on the ImageNet dataset, IBiT outperforms DeiT and ConViT, which provides some evidence for the effectiveness of the approach.
summary: This paper proposes an Inductively Biased Image Transformer (IBiT) framework to improve data efficiency and small-dataset performance of ViT by introducing convolution-like inductive biases into the self-attention mechanism. Specifically, IBiT includes two core technical ideas: Learnable Mask and Low-Rank Approximation. The Learnable Mask module applies trainable, convolution-inspired locality constraints directly onto the attention maps, enabling the ViT to mimic CNN’s translational equivariance and locality without the use of knowledge distillation or teacher models. The Low-Rank Approximation technique exploits the sparsity of local attention patterns to represent the inductive bias mask using two sub-matrices, significantly reducing parameter count and computational cost. 

The experimental evaluation in this paper assessed the performance of the proposed IBiT on ImageNet-1K, comparing it to various state-of-the-art Transformer-based methods. The results indicate that IBiT consistently outperformed baselines with the same parameter count while maintaining better scaling behavior on smaller subsets of the dataset.
weaknesses: - Although the mathematical derivation in Sec. 3.1 is valid; there remain key differences from actual convolution. For example, attention weights are input-dependent, while convolution kernel parameters are independent of the input. In self-attention, the weights depend on the dot products between queries and keys, so different inputs will produce different attention distributions. The proposed method merely multiplies the attention weights by a fixed-shape mask; while the mask enforces locality, the attention values are still content-dependent, and thus, its translational equivariance may be less stable than that of convolution.
- Convolution not only enforces locality, but also weight sharing and a fixed linear combination pattern, both of which are important for feature stability. The proposed Learnable Mask employs a rolling mechanism to maintain consistency in the mask pattern. However, it still multiplies the mask pattern with attention weights that may vary significantly across positions, thereby ignoring the inherent variability of the weights themselves.
- One of ViT’s advantages over CNNs is the ability to model global dependencies. Applying a strong locality mask may suppress interactions between distant tokens, which could negatively affect tasks requiring full scene semantics.
- The experiments mainly compare against DeiT and ConViT, lacking a comprehensive comparison with other locality-enhanced Transformers (e.g., Swin Transformer).
- All experiments are conducted on classification tasks; there is no evaluation on detection or segmentation downstream tasks, leaving it unclear whether the proposed inductive bias would also be effective for tasks requiring precise spatial localization.

---

confidence: 4
contribution: 1
initial_rating: 0
presentation: 1
questions: * Why do you think of $X_m$ as flattening of the image? In transformers it is consist of the patches of the input images, or tokens in LLMs and not the pixels themselves.
* When you talking on inductive bias in CNNs with respect to those of attention – do you mean that CNNs are inherently handling spatial consistency? While it might be that transformers are not dedicated to? If so, then I think that this claim should be clarified and exemplified – since there is a positional encoding – so the spatial information is somehow embedded in the transformer learning scheme.
* What should I learn from tables 1 and 2? It is a configuration of the training, you can just state it, why do you place it in table? Moreover, the values are almost identical.

Im sorry, but this paper is far from being  up to academic standard, especially for top-tier conference. The level of writing, depth of analysis, thorough comparisons, mathematical correctness are far from the required. I tried read it few times but I think I couldn’t catch the conceptual novelty here, both because the writing level and the mathematic confusing notations. Even if there is conceptual contribution hidden here, this paper should be dramatically revised.
soundness: 1
strengths: * Better understanding of the process mechanism of attention heads is important, and handling low-data regime might be beneficial in some cases where data is hard to harvest.
* The importance maps of your method look nice, however beside the first two images, I cannot understand what the attention map represents, on which object does it focuses and why.
summary: The authors are trying to mimic inductive bias in CNNS and plant it into self-attention mechanisms. They propose using learnable masking and low rank approximation to learn this mask, and they claim it should benefit in low data regime.
weaknesses: * Poor level of writing which is not up to standard, to elaborate on some:
  * The abstract and the introduction sections are far from being comprehensive, they are too short and not informative enough.
  * It is not common to embed citations in the abstract.
  * The figures – for example Fig. 1 and Fig. 4 are too detailed, visualizations are better in clarifying or convey ideas, not for detailed explanations.
  * Unbalanced layout, the Figs are took over the space in the paper, where in the extreme case in page 5 – Figure 3 and Algorithm 2, are covering the entire page (beside 2 sentences in the middle).
  * Uncommon notations – it is not common to denote variable as a full name like width, height and size. Using of notations without explaining them like the full dot (which I can get that it is multiplication I guess) but it is better to declare prior to the usage.
  * There is no period at the end of sentences of almost all captions of figures.
  * The comparisons are not convincing, with low margin and against too few exemplars.
  * Overuse of capital letters – see for example Lines 12 and 17 in the abstract.
  * The equations are not numbered.
  * Lack of flow and tension in the paper, for example the separation for paragraphs is not enhancing flow and sometimes seems random.
* The mathematical notations are confusing and very hard to follow – The symbol * is not what you want to use for multiplication, it is better to leave it without any symbol for multiplication. Moreover, the indexing is super hard to follow.
* The experimental section is too poor. There is too few comparisons with only DeiT and ConViT only on ImageNet. Moreover, there seems to be no improve, or only marginal when using lower portions of ImageNet, whereas this is the claim to fame of the paper.

---

confidence: 4
contribution: 1
initial_rating: 0
presentation: 1
questions: No more questions. Please reconsider this topic and write a proper paper.
soundness: 1
strengths: Introducing inductive bias to the ViTs is a good topic to study.
summary: This paper introduces a learned mask in self-attention to better capture the inductive bias that is lacking in the conventional ViT models. The authors compared their method on the ImageNet-1K dataset, showing good performance compared to other ViTs.
weaknesses: - The motivation of the paper is that introducing the inductive bias to the ViTs can improve ViT training on small-scale datasets. However, only one experiment on ImageNet-1K is shown in the paper, which is not considered to be “small-scale”. The experiment does not correspond to the motivation of the paper.
- The paper lacks a significant amount of related work discussion. There are many methods that do weight selection, weight mimetic, introducing CNN structure, etc. These methods are very related to the topic explored in this paper; the authors should properly cite this research and give reasonable discussions.
- The paper does not provide a solid, thorough explanation of the proposed method. No valid theoretical proofs or thorough quantitative proofs.
- The paper is not well-organized or polished. There are many unnecessary parts in the paper that only add to the pages. For example, algorithm 2 and figure 3 are redundant, training curves are not explained, just randomly shown in the paper, etc. There exist many typos, unexplained figures, and unreasonable arguments.


Decision: Reject