ICML 2025

Steerable Transformers for Volumetric Data
Soumyabrata Kundu 1 Risi Kondor 1 2
Abstract
We introduce Steerable Transformers, an exten-
sion of the Vision Transformer that is equivari-
ant to the action of the Special Euclidean group
SE(d). We propose an steerable self-attention
mechanism that operates on features extracted
by steerable convolutions. Our experiments in
both two and three dimensions show augmenting
steerable convolutional networks with steerable
transformer leads to improved performance.
1. Introduction
Transformers have emerged as the preferred architecture
for natural language processing tasks, with recent powerful
models like Chat-GPT employing this framework. Their
relatively straightforward design, coupled with remarkable
success, has led to their widespread adoption across various
domains, including image classification (Dosovitskiy et al.,
2021), object detection (Carion et al., 2020), and graph
based problems (Dwivedi & Bresson, 2020).
The self-
attention mechanism (Bahdanau et al., 2014) employed in
transformer architectures has proven to be crucial for cap-
turing relationships between different parts of the input se-
quence. Dosovitskiy et al. (2021) introduced transformers
as an alternative to traditional convolutional architectures
for vision tasks.
Unlike convolutional neural networks
(CNNs), which focus on local neighborhoods, transform-
ers excel at capturing relations across different parts of the
input.
Equivariant neural network architectures have gained sig-
nificant popularity in recent years due to their inherent abil-
ity to comprehend the underlying symmetries of problems,
making them highly effective tools for real world tasks. For
instance, achieving equivariance to the permutation group
Sn is crucial in graph based problems, since the structure
1Department of Statistics, University of Chicago, Chicago,
USA 2Department of Computer Science, University of Chicago,
Chicago,
USA. Correspondence to:
Soumyabrata Kundu
<soumyabratak...