Title: IBiT: Utilizing Inductive Biases to Create a More Data Efficient Attention Mechanism
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Computer Vision, Transformers

Abstract: In recent years, Transformer-based architectures have become the dominant method for Computer Vision applications. While Transformers are explainable and scale well with dataset size, they lack the inductive biases of Convolutional Neural Networks. While these biases may be learned on large datasets, we show that introducing these inductive biases through learned masks allow Vision Transformers to learn on much smaller datasets without Knowledge Distillation. These Transformers, which we call Inductively Biased Image Transformers (IBiT), are significantly more accurate on small datasets, while retaining the explainability and scalability of Transformers.

TL;DR: We used inductive biases that are created through learned masks to create a model that performs significantly better on smaller datasets.