Don't Pay Attention
Authors: name: Mohammad Hammoud; name: Devang Acharya
Published: 2025-06-12 21:11:06

Abstract: The Transformer has become the de facto standard for large language models
and a wide range of downstream tasks across various domains. Despite its
numerous advantages like inherent training parallelism, the Transformer still
faces key challenges due to its inability to effectively process sequences
beyond a fixed context window and the quadratic complexity of its attention
mechanism. These challenges have renewed interest in RNN-like architectures,
which offer linear scaling with sequence length and improved handling of
long-range dependencies, albeit with limited parallelism due to their
inherently recurrent nature. In this paper, we propose Avey, a new neural
foundational architecture that breaks away from both attention and recurrence.
Avey comprises a ranker and an autoregressive neural processor, which
collaboratively identify and contextualize only the most relevant tokens for
any given token, regardless of their positions in the sequence. Specifically,
Avey decouples sequence length from context width, thus enabling effective
processing of arbitrarily long sequences. Experimental results show that Avey
compares favorably to the Transformer across a variety of standard short-range
NLP benchmarks, while notably excelling at capturing long-range dependencies.