FLARE: Fast Low-rank Attention Routing Engine
Authors: name: Vedant Puri, name: Aditya Joglekar, name: Kevin Ferguson, name: Yu-hsuan Chen, name: Yongjie Jessica Zhang, name: Levent Burak Kara

The quadratic complexity of self-attention limits its applicability and
scalability on large unstructured meshes. We introduce Fast Low-rank Attention
Routing Engine (FLARE), a linear complexity self-attention mechanism that
routes attention through fixed-length latent sequences. Each attention head
performs global communication among N tokens by projecting the input sequence
onto a fixed length latent sequence of M ll N tokens using learnable query
tokens. By routing attention through a bottleneck sequence, FLARE learns a
low-rank form of attention that can be applied at O(NM) cost. FLARE not only
scales to unprecedented problem sizes, but also delivers superior accuracy
compared to state-of-the-art neural PDE surrogates across diverse benchmarks.
We also release a new additive manufacturing dataset to spur further research.
Our code is available at https://github.com/vpuri3/FLARE.py.