FLARE: Fast Low-rank Attention Routing Engine
Authors: name: Vedant Puri; name: Aditya Joglekar; name: Kevin Ferguson; name: Yu-hsuan Chen; name: Yongjie Jessica Zhang; name: Levent Burak Kara
Published: 2025-08-18 03:00:55

Abstract: The quadratic complexity of self-attention limits its applicability and
scalability on large unstructured meshes. We introduce Fast Low-rank Attention
Routing Engine (FLARE), a linear complexity self-attention mechanism that
routes attention through fixed-length latent sequences. Each attention head
performs global communication among N tokens by projecting the input sequence
onto a fixed length latent sequence of M ll N tokens using learnable query
tokens. By routing attention through a bottleneck sequence, FLARE learns a
low-rank form of attention that can be applied at O(NM) cost. FLARE not only
scales to unprecedented problem sizes, but also delivers superior accuracy
compared to state-of-the-art neural PDE surrogates across diverse benchmarks.
We also release a new additive manufacturing dataset to spur further research.
Our code is available at https://github.com/vpuri3/FLARE.py.