Title: FLARE: Fast Low-rank Attention Routing Engine
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
Keywords: attention mechanisms, low-rank approximation, PDE surrogate modeling, scientific machine learning, point clouds, unstructured meshes, efficient transformers, additive manufacturing, 3D geometry

Abstract: The quadratic complexity of self-attention limits its applicability and scalability on large unstructured meshes.
We introduce Fast Low-rank Attention Routing Engine (FLARE), a linear complexity self-attention mechanism that routes attention through fixed-length latent sequences.
Each attention head performs global communication among $N$ tokens by projecting the input sequence onto a fixed length latent sequence of $M \ll N$ tokens using learnable query tokens.
Routing information through this bottleneck constrains the interaction to have rank at most $M$: the resulting operation is equivalent to an attention matrix of rank at most $M$, realized at $\mathcal{O}(NM)$ cost.
This formulation allows users to flexibly adjust the effective rank of attention --and thus the balance between accuracy and efficiency-- by tuning $M$ to their resource and performance requirements.
FLARE not only scales to unprecedented problem sizes, but also delivers superior accuracy compared to state-of-the-art neural PDE surrogates across diverse benchmarks.
We also introduce a new additive manufacturing dataset to spur further research.

TL;DR: FLARE is an efficient transformer that leverages the low-rank structure of attention to balance accuracy and efficiency, scaling to extremely large token counts on a single GPU.