# Tutorial on Sustainable Deep Learning for Time-series: Reservoir Computing

Andrea Ceni, Peter Ford Dominey, Claudio Gallicchio, Xavier Hinaut, Gianluca Milano

## ► To cite this version:

Andrea Ceni, Peter Ford Dominey, Claudio Gallicchio, Xavier Hinaut, Gianluca Milano. Tutorial on Sustainable Deep Learning for Time-series: Reservoir Computing. ECML-PKDD 2024 - European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, Sep 2024, Vilnius, Lithuania. hal-04908999

HAL Id: hal-04908999

https://inria.hal.science/hal-04908999v1

Submitted on 23 Jan 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Tutorial Proposal on
# Sustainable Deep Learning for Time-series:
# Reservoir Computing

Andrea Ceni, Peter Ford Dominey, Claudio Gallicchio, Xavier Hinaut, and Gianluca Milano

**Title:** Sustainable Deep Learning for Time-series: Reservoir Computing

## Abstract

Reservoir Computing (RC) is a promising approach for time-series forecasting that offers efficient and sustainable deep learning alternatives to traditional neural networks. This tutorial provides an introduction to sustainable deep learning with RC, including its theoretical foundations, practical implementation, and advantages over other approaches. The tutorial starts with an overview of the RC framework, including the Echo State Network (ESN) and Liquid State Machine (LSM) models. The fundamental principles of these models are explored, including their unique network structure, the learning mechanisms, and the fundamentals of RC mathematical background. The tutorial then proceeds to explore the strong links with computational neuroscience, and the exciting recent advances in neuromorphic hardware implementations of reservoirs. Finally, the tutorial covers practical implementation aspects, existing software libraries, as well as successful application examples in several domains including automotive, avionics, health, and robotics.

In addition to providing practical guidance, this tutorial also emphasizes the importance of sustainability in deep learning. The environmental impact of training deep neural networks is substantial, and RC offers a more energy-efficient approach that is particularly well-suited for time-series forecasting. By using RC, researchers can reduce the carbon footprint of their research while achieving state-of-the-art results.

Overall, this tutorial provides a comprehensive introduction to sustainable deep learning with RC for time-series forecasting, and it is suitable for both beginners and advanced researchers looking to explore this exciting area of machine learning.
## Description of the tutorial

Reservoir Computing (RC) is a paradigm for designing recurrent neural models for sequential data processing, distinctly featured by efficient training algorithms. The approach is rooted in the analysis of the dynamical components of the neural architecture under the perspective of dynamical systems theory, and proposes to use fixed (possibly randomized) connections under asymptotic stability conditions. The resulting reservoir dynamics provide a rich encoding of the external input signals, enabling to reduce the learning problem to a simple regression performed by a linear model. The approach stands out in the context of Machine Learning and Deep Learning research for its distinctive and striking advantage in terms of computational efficiency, ensuring fast and sustainable training algorithms for time-series AI applications. Apart from the undoubtedly computational advantage, RC is appealing for a number of reasons, including its neurobiological plausibility, its amenability to implementations in unconventional neuromorphic hardware (e.g., in photonics), and the fact that it leads to clean mathematical analysis of the architectural biases of recurrent models. Intriguingly, RC gives a refreshed perspective over diverse topics in Deep Learning research, including learning beyond Backpropagation, stable neural architectures, the lottery ticket hypothesis, tiny ML, and neural architecture search.

The tutorial will offer a comprehensive view of the field, from basics to the most recent advances on the machine learning side, the starting and current trends of RC in Computational Neuroscience, the recent developments in neuromorphic hardware, and will survey current libraries of RC with a detailed introduction to ReservoirPy with Python Notebooks available online.

This tutorial is targeted to both researchers and practitioners that are interested in setting up fastly-trained recurrent neural networks for structured data. It will open their mind to a computing paradigm different from the most well-known ones, which offers state-of-the-art performances on several applications. Such a paradigm should be considered as its own and to implement mixtures of approaches in order to tackle seriously the environmental challenges. It is also recognized in neuroscience as an important principle to interpret and model brain computations. Basic concepts on Machine Learning and Deep Neural Networks are suggested prerequisites.

## Relevance to the ECML community

This tutorial will provide an intriguing and broad overview on the emerging topic of efficiently trainable neural networks architectures, ranging from the enabling mathematical theory and the available software libraries, to computational neuroscience evidences and neuromorphic implementations in unconventional physical substrates. In this sense, the diverse backgrounds of the presenters, along with the designed format for the tutorial (featuring relatively short presentations), will be key factors in providing a stimulating and effective introduction to the subject.

The focus on energy expenditure, environmental and economic impact related to the development of deep learning algorithms is becoming increasingly important in the European and global landscape. This tutorial directly addresses the ECML PKDD community's increasing focus on sustainable AI solutions, presenting Reservoir Computing as an renewed approach to meet the efficiency challenges in deep learning. We thereby believe
that the issues addressed in this tutorial are extremely relevant to the ECML PKDD community.

# Outline of the tutorial

This tutorial is structured in chapters, according to the following organization. Each chapter will be structured into sub-sections, each designed to fit within the allocated time while ensuring a comprehensive coverage of the topic. Note that a 30 minutes break is expected between Chapters 3 and 4.

* **Chapter 1: Introduction** (10 minutes)

Preliminaries on Deep Learning for sequential data and Recurrent Neural Networks (RNNs); green AI and the problem of designing sustainable Deep Learning algorithms.

* **Chapter 2: Basics** (30 minutes)

Randomization in Deep Neural Networks; Echo State Networks, Liquid State Machines, and deep architectures; quality of dynamics and network topologies; training algorithms; Reservoir Computing for graphs.

* **Chapter 3: Mathematical foundations** (40 minutes)

Recurrent Neural Networks as input-driven dynamical systems. Stability of recurrent neural networks; Memory-nonlinearity trade-off; Edge-of-chaos dynamics.

* **Chapter 4: Computational Neuroscience of Reservoir Computing** (40 minutes)

Temporal processing hierarchy in the brain; relations between reservoir connectivity and temporal hierarchical processing; simulation of human narrative processing using word embeddings and reservoirs; effects of network topology on narrative processing temporal hierarchy.

* **Chapter 5: Neuromorphic Hardware implementations** (40 minutes)

Principles of neural networks in unconventional and neuromorphic hardware; Physical substrates for neuromorphic reservoirs: photonics, memristors, spintronics, and mechanical systems.

* **Chapter 6: Software libraries and applications** (40 minutes)

Python frameworks for Reservoir Computing; hyperparameter exploration; building complex architectures; example applications (e.g., sound classification with implicit segmentation) and tricks of the trade. Short hands-on lab demonstration (ReservoirPy): Implementing Echo State Networks in Python for time-series prediction and classification.

* **Chapter 7: Quo vadis?** (10 minutes)

Conclusions and outlook.
# Instructors

All instructors will be physically present at the conference. Each chapter of the tutorial will be presented by one of the instructors, as detailed below.

## Andrea Ceni

**Affiliation:** Department of Computer Science, University of Pisa, Italy;

**email:** andrea.ceni@di.unipi.it

**Presenter for:** Chapter 3 (Mathematical Foundations)

**Short bio:** Andrea Ceni is a Research Fellow at the Department of Computer Science of the University of Pisa, Italy. He has been a Postdoctoral Research Associate at CEMPS, University of Exeter. He received the Ph.D. degree from the Department of Computer Science of the University of Exeter, UK, in 2021, and the M.Sc. degree in Mathematics cum laude from Università degli Studi di Firenze, Italy, in 2017. His research interests include recurrent neural networks, deep learning, computational neuroscience, chaos theory, and complex systems. He has co-organized the 1st workshop on Deep Learning meets Neuromorphic Hardware, co-located with ECML-PKDD 2023, the tutorial on "Sustainable Deep Learning for Time-series: Reservoir Computing" at ECML-PKDD 2023, and the special sessions on Reservoir Computing at IJCNN 2023 and IJCNN 2024.

## Peter Ford Dominey

**Affiliation:** Centre National de la Recherche Scientifique (CNRS), Paris, France; and INSERM U1093 Cognition, Action et Plasticité Sensomotrice Dijon, France.

**email:** Peter-Ford.Dominey@u-bourgogne.fr

**Presenter for:** Chapter 4 (Computational Neuroscience of Reservoir Computing)

**Short bio:** Peter Ford Dominey is a CNRS Research Director at INSERM U1093 in Dijon, France. He completed the BA at Cornell University in 1984 in cognitive psychology and artificial intelligence. From 1984 to 1986 he was a Software Engineer at Data General, and from 1986 to 1993 he was a Systems Engineer at NASA/JPL/CalTech. In 1989 and 1993 respectively he obtained the M.Sc. and Ph.D. in computer science from the University of Southern California, developing neural network models of sensorimotor sequence learning, including the first instances of reservoir computing. In 1997 he joined the CNRS at the Institut des Sciences Cognitives in Lyon France and initiated the Robot Cognition Laboratory. In Dijon since 2019 at INSERM U1093, his research interests include understanding and simulating the neurophysiology of cognitive sequence processing, action and language, and their application to robot cognition and language processing, and human-robot cooperation.

## Claudio Gallicchio

**Affiliation:** Department of Computer Science, University of Pisa, Italy;

**email:** claudio.gallicchio@unipi.it

**Presenter for:** Chapters 1 (Introduction), 2 (Basics), 6 (Quo vadis?)

**Short bio:** Claudio Gallicchio is a Tenured-track Assistant Professor of Machine Learning at the Department of Computer Science of the University of Pisa, Italy. He received his PhD from the University of Pisa, where he focused on Recurrent and Reservoir Computing models and theory for structured data. His research is based on the fusion of concepts from Deep Learning, Recurrent Neural Networks and Randomized Neural Systems. He is the founder and former chair of the IEEE CIS Task Force on Reservoir Computing, co-founder and vice-chair of the IEEE Task Force on Randomization-based Neural Networks and
Learning Systems. He is a member of the IEEE CIS Data Mining and Big Data Analytics Technical Committee, of the IEEE CIS Task Force on Deep Learning, and of the IEEE Task Force on Learning for structured data. He (co-)authored several publications at major ML conferences, including ICLR, ICML, AAAI, and IJCNN/WCCI. He has (co-)organized several events (tutorials, special sessions, and workshops) at major ML international conferences. In particular, he has co-organized the Ambient AI Workshop co-located with the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP) 2023, the 1st and the 2nd International Workshop on Machine Learning for Irregular Time series co-located with the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML-PKDD) in 2021 and 2023, the 1st workshop on Deep Learning meets Neuromorphic Hardware co-located with the ECML-PKDD 2023, the Deep Learning in Unconventional Neuromorphic Hardware workshop within the IJCNN 2021 conference, the International Workshop on Reservoir Computing in 2019 within the ICANN conference, and the Machine Learning and Data Mining workshop within the International Conference of the Italian Association for Artificial Intelligence since 2016. He has a long experience in organizing special sessions at international conferences, having organized 10+ of them in recent years. He serves as a member of several program committees of conferences and workshops in ML and AI (including NeurIPS, ICML, AISTATS, ICLR, IJCAI-PRICAI, ECML-PKDD, ICONIP, IJCNN/WCCI). He is an associate editor of IEEE Transactions on Neural Networks and Learning Systems (TNNLS). He is the coordinator of the project "NEURONE: extremely efficient NEUromorphic Reservoir computing in Nanowire network hardware", funded by the Italian ministry of Research.

**Xavier Hinaut**

**Affiliation:** Inria, Bordeaux, France;

**email:** xavier.hinaut@inria.fr

**Presenter for:** Chapter 6 (Software libraries and applications)

**Short bio:** Xavier Hinaut is Research Scientist in Bio-inspired Machine Learning and Computational Neuroscience at Inria, Bordeaux, France since 2016. He received a MSc and Engineering degree from Compiègne Technology University (UTC), FR in 2008, a MSc in Cognitive Science & AI from EPHE, FR in 2019, then his PhD of Lyon University, FR in 2013. He is a member (Vice Chair) of IEEE CIS Task Force on Reservoir Computing. His work is at the frontier of neurosciences, machine learning, robotics and linguistics: from the modeling of human sentence processing to the analysis of birdsongs and their neural correlates. He both uses reservoirs for machine learning (e.g. birdsong classification) and models (e.g. sensorimotor models of how birds learn to sing). He manages the "DeepPool" ANR project on human sentence modeling with Deep Reservoirs architectures and the Inria Exploratory Action "BrainGPT" on Reservoir Transformers. He leads ReservoirPy development: the most up-to-date Python library for Reservoir Computing: https://github.com/reservoirpy/reservoirpy
He is also involved in public outreach, notably by organizing hackathons from which fun projects with reservoirs came out: https://github.com/HugoChateauLaurent/remi

**Gianluca Milano**

**Affiliation:** INRIM, Torino, Italy;

**email:** g.milano@inrim.it

**Presenter for:** Chapter 5 (Neuromorphic Hardware implementations)

**Short bio:** Gianluca Milano is currently a permanent researcher at the Italian National Institute of Metrological Research (INRIM). He received a Ph.D. in Physics cum laude from
Politecnico di Torino, Italy, in collaboration with the Italian Institute of Technology (IIT). His main research interests and activities focus on: i) the investigation of electronic and ionic transport properties and physicochemical phenomena in nanodevices and low dimensional systems; and ii) memristive devices and architectures for memory and neuromorphic applications, from material synthesis to device characterization, modeling, and implementation of unconventional and brain-inspired computing paradigms in neuromorphic architectures. He is coordinator of the european project EMPIR MEMQuD (https://memqud.inrim.it/) that involves 15 european partners, including Universities, research centers and industries, that focus on the development of memristive devices working in the quantum regime for quantum and neuromorphic applications. He is member of the Editorial advisory Board for the journal APL Machine Learning. For his work on in-materia implementation of reservoir computing in self-organizing networks of nano objects he has received the NEST prize for Nanoscience 2021. He has co-organized the 1st workshop on Deep Learning meets Neuromorphic Hardware, co-located with ECML-PKDD 2023, and the tutorial on "Sustainable Deep Learning for Time-series: Reservoir Computing" at ECML-PKDD 2023.

Previous Venues

Similar tutorials/seminars, covering a part of the topics of the proposed one for ECML, have been previously given in the following venues.

[ECML-PKDD 2023] "Sustainable Deep Learning for Time-series: Reservoir Computing", tutorial of the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML-PKDD), 22 September 2023. Speakers: A. Ceni, C. Gallicchio, X. Hinaut, G. Milano, and A. Morrison. This represents the first edition of this tutorial, which was held in conjunction with the ML4ITS 2023 workshop. Duration 1.5 hours. Audience size: around 60 people. Webpage with the tutorial material: https://sites.google.com/view/reservoir-computing-tf/activities/ecmlpkdd-2023-tutorial-on-reservoir-computing. The second edition in 2024 will cover new advancements especially in the areas of neuromorphic implementations, computational neuroscience links, mathematical investigations, architectural construction, and real-world use cases.

[AAAI 2021] "Deep Randomized Neural Networks", Tutorial of the 35th AAAI Conference on Artificial Intelligence (AAAI), 2-9 February 2021, Virtual Conference. Speaker: Claudio Gallicchio. Audience size: around 50 people. Duration: 1.5 hours. Conference program link and tutorial material: https://aaai.org/Conferences/AAAI-21/aaai21tutorials/#mq3 and https://sites.google.com/site/cgallicch/resources/tutorial_DRNN

[IJCNN 2021] "Reservoir Computing: Randomized Recurrent Neural Networks", Tutorial of the International Joint Conference on Neural Networks (IJCNN), 18-22 July 2021, Virtual Conference. Speaker: Claudio Gallicchio. Audience size: around 50 people. Duration: 2 hours. Conference program link: https://www.ijcnn.org/tutorials Tutorial recording on YouTube: https://www.youtube.com/watch?v=XJg7VdN7g-0

[NMP2021 - 11th Optoelectronics and Photonics Summer School] "Neuromorphic Computing for Neural Networks", 20-26 June 2021, Monte Bondone, Italy. Speaker: Claudio
Gallicchio. Audience size: around 100 people. Duration: 2 hours. Program of the school: https://event.unitn.it/nmp2021/.

Technical equipment

The tutorial requires standard conference equipment, including a video projector, microphones, and usb-c or hdmi cables for slides presentation. Based on our previous experience and the growing interest in sustainable deep learning, we anticipate an audience size of approximately 60-80 attendees