In 2024 the Bernstein Network collaborates with World Wide
Neuro.

Please find all abstracts sorted by sessions below. In addition,
abstracts are also available on World Wide Neuro. There, you have
the opportunity to search for similar abstracts. Poster presenters
can furthermore upload additional materials such as video or audio
tracks.

https://www.world-wide.org/bernstein-24/

Bernstein Conference 2024

Recurrent Attention Network

Yannis Bendi-Ouis ¹, Xavier Hinaut ¹

1. Mnemosyne, Inria, Bordeaux, France

With the recent success of Transformers in the field of automatic natural language processing [1], it has become clear that we are able to create complex, high-performance neural network architectures that can extract, combine and transform information to meet a specific objective, such as predicting the next word in a text.

However, to function, Transformers need to receive a context of all N elements of the past sequence (N = 2k to 32k tokens) at each time step, causing the number of computations to grow quadratically (N^2) with the length of the sequence [1].

This "perfect" memory of a context of thousands of elements is equivalent in neuroscience to a
biologically unrealistic working memory. In order to be more biologically plausible, we propose to
drastically reduce the size of this context (allowing more local and less costly computations), while
introducing recurrences allowing the use of a longer-term working memory.

An efficient and biologically plausible way to have recurrences is to place oneself in the Reservoir
Computing paradigm [2], where the recurrent connections are random and the output layer serves as a
readout trained by linear regression (or equivalence). For computational neuroscience, Reservoir
Computing is a very interesting approach for modeling high-dimensional non-linear computations with
low computational complexity.

Unlike other approaches that seek to modify the calculations behind attention (RWKV [5], RetNet [6]), we propose to combine these two models so as to benefit from both the attentional computations of Transformers and the working memory of Reservoir Computing.

In the same way that dendritic computations [4] show us that neurons are capable of more complex computations, we believe that it may be interesting to replace the neurons (tanh unit) present in a Reservoir with more complex units, such as the attention blocks present in Transformers, even for a short period of time.
We are evaluating two new architectures on a set of sequential tasks (e.g. SequentialMnist): one containing units equivalent to the transformers' attention blocks (fig A), and the other (fig C) decomposing the attention block into two types of unit (fig B): a unit based on the QKV (Query-Key-Value) mechanism and another unit of "memories" [3] (Feed-Forward), thus making it possible to model different brain structures (cortex, hippocampus).

<figure><img src="image_1.png" /><figcaption>Figure 1: Different Architecture Attempt of Recurrent Attention Network</figcaption></figure>

## References

1. Attention is all you need (https://arxiv.org/abs/1706.03762), 10.48550/arXiv.1706.03762 (http://dx.doi.org/10.48550/arXiv.1706.03762)

2. The "echo state" approach to analysing and training recurrent neural networks (https://www.ai.rug.nl/minds/uploads/EchoStatesTechRep.pdf)

3. Transformer Feed-Forward Layers Are Key-Value Memories (https://aclanthology.org/2021.emnlp-main.446.pdf), 10.48550/arXiv.2012.14913 (http://dx.doi.org/10.48550/arXiv.2012.14913)

4. Dendritic computation (https://pubmed.ncbi.nlm.nih.gov/16033324/), 10.1146/annurev.neuro.28.061604.135703 (http://dx.doi.org/10.1146/annurev.neuro.28.061604.135703)

5. RWKV: Reinventing RNNs for the Transformer Era (https://arxiv.org/pdf/2305.13048), 10.48550/arXiv.2305.13048 (http://dx.doi.org/10.48550/arXiv.2305.13048)

6. Retentive Network: A Successor to Transformer for Large Language Models (https://arxiv.org/abs/2307.08621), 10.48550/arXiv.2307.08621 (http://dx.doi.org/10.48550/arXiv.2307.08621)

Copyright: © (2024) Bendi-Ouis Y, Hinaut X

Citation: Bendi-Ouis Y, Hinaut X (2024) Recurrent Attention Network. Bernstein Conference 2024. doi: 10.12751/nncn.bc2024.314 (http://doi.org/10.12751/nncn.bc2024.314)