# Echo State Transformer

Yannis Bendi-Ouis, Xavier Hinaut

► To cite this version:

Yannis Bendi-Ouis, Xavier Hinaut. Echo State Transformer. Journée de l'EDMI 2025 - Journée annuelle de l'écode doctorale de mathématiques et d'informatique, Apr 2025, Talence (Bordeaux), France. hal-04991876

HAL Id: hal-04991876

https://hal.science/hal-04991876v1

Submitted on 14 Mar 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Echo State Transformer

Yannis Bendi-Ouis¹,²,³, Xavier Hinaut¹,²,³

¹Inria Center of Bordeaux University, Bordeaux, France

²LaBRI, Bordeaux Univ., Bordeaux INP, CNRS UMR 5800, France

³Bordeaux Univ., CNRS, IMN, UMR 5293, Bordeaux, France

This work is supported by Inria AEx BrainGPT project.
Thanks to Experimental Inria Cluster Plafrim.

Inspired by Reservoir Computing
we add Memory Units to Transformer
to reduce their complexity from O(n²) to O(1).

<figure><img src="image_2.png" /></figure>

## References

[1] Vaswani et al., Attention is all you need.

[2] Geva et al., Transformer feed-forward layers are key-value memories.

[3] Jaeger, The "echo state" approach to analysing and training recurrent neural networks-with an erratum note.

[4] Kowsher, M., & Xu, J. Reservoir Transformer at Infinite Horizon: the Lyapunov Time and the Butterfly Effect.

[5] Kowsher, M., Khan, A. R., & Xu, J. Changes by Butterflies: Farsighted Forecasting with Group Reservoir Transformer.

[6] Jaegle et al., Perceiver: General perception with iterative attention.