Sparse Attention Mechanisms in Vision Transformers: from Spatiotemporal Transformers to Instance-Dependent Attention Patterns
Keywords: Computer Vision
Deep Learning
Machine Learning
Pattern Recognition

Deep learning (DL) algorithms have produced superior results for a variety of vision, language, and speech processing tasks, but have proven to be power and compute-hungry. In the past, hardware advancements have met the growing need for compute. Yet in recent years the end of Dennard scaling and Moore's law makes hardware advancements unlikely to continue at the same fast pace. Hence innovative algorithmic solutions are needed to meet DL algorithms' ever-growing demand for compute. In this dissertation we investigated methods using sparsity and conditional computation to meet this compute demand. The two main components of this contribution involve applying structure and dynamic prediction of sparsity patterns by machine learning algorithms. The hypothesis was that structured and dynamic sparse computation improves latency and memory footprint compared to existing approaches with equal algorithm effectiveness. To show this, we built on the state of the art in sparse computations. This dissertation fills in previously unexplored areas of applying structured sparsity to videos, and of predicting per-instance sparsity that is unstructured. The purpose was to investigate using structured and per-example sparsity to design performant and effective deep learning algorithms. Existing literature had investigated sparsity in computer vision, but only in the context of either convolutional neural networks (CNNs) or Transformers applied to image processing. Given the increased computational complexity, this raised the question of how sparsity fares when applied to video processing. We investigated this question in work presented at the CVPR 2021 conference called Sparse Spatiotemporal Transformers for Video Object Segmentation (SSTVOS). SSTVOS looks at the highly computationally demanding problem of applying Transformers to video by using sparse attention blocks. The sparse attention blocks in SSTVOS use a fixed sparsity pattern that takes advantage of a priori knowledge of which visual features to attend to. However, using fixed sparsity patterns limits us to problems where we have this a priori knowledge. Therefore we further executed a research direction to learn sparsity patterns that are performant (low latency and memory usage) and effective (superior predictive power) in a method called Sparsifiner. Sparsifiner predicts per-example sparsity patterns as a more flexible generalization of the fixed sparsity patterns in SSTVOS. With the expectation that a machine learning model learns to predict sparsity patterns superior to any fixed pattern designed by hand, the idea for Sparsifiner was born. Sparsifiner improves over existing sparsity methods in Vision Transformers~(ViTs) by predicting per-instance sparsity patterns with a computationally-efficient connectivity predictor module. The method serves as empirical evidence of the effectiveness of per-instance sparsity patterns compared to fixed sparsity patterns. Sparsifiner shows a Pareto-optimal trade-off between FLOPs and top-1 accuracy on ImageNet when compared to contemporary structured-sparse methods. Together, both SSTVOS and Sparsifiner constitute a basis for future work in the direction of structured and dynamic sparsity in deep learning.