Breaking the Attention Bottleneck
Authors: name: Kalle Hilsenbek

Attention-based transformers have become the standard architecture in many
deep learning fields, primarily due to their ability to model long-range
dependencies and handle variable-length input sequences. However, the attention
mechanism with its quadratic complexity is a significant bottleneck in the
transformer architecture. This algorithm is only uni-directional in the decoder
and converges to a static pattern in over-parametrized decoder-only models. I
address this issue by developing a generative function as attention or
activation replacement. It still has the auto-regressive character by comparing
each token with the previous one. In my test setting with nanoGPT this yields a
smaller loss while having a smaller model. The loss further drops by
incorporating an average context vector. This concept of attention replacement
is distributed under the GNU AGPL v3 license at
https://gitlab.com/Bachstelze/causal_generation.