Locally Connected Echo State Networks for Time Series Forecasting
Authors: Filip Matzner
František Mráz
Venue: ICLR 2025 Poster (2025)
Keywords: Time Series Analysis
Time Series Forecasting
TSF
Recurrent Neural Networks
RNN
Regression
Echo State Networks
ESN
Primary area: learning on time series and dynamical systems
TL;DR: Improved locally connected ESN method comparable with state-of-the-art on real-world time series datasets.

Abstract: Echo State Networks (ESNs) are a class of recurrent neural networks in which only a small readout regression layer is trained, while the weights of the recurrent network, termed the reservoir, are randomly assigned and remain fixed. Our work introduces the Locally Connected ESN (LCESN), a novel ESN variant with a locally connected reservoir, forced memory, and a weight adaptation strategy. LCESN significantly reduces the asymptotic time and space complexities compared to the conventional ESN, enabling substantially larger networks. LCESN also improves the memory properties of ESNs without affecting network stability. We evaluate LCESN's performance on the NARMA10 benchmark task and compare it to state-of-the-art models on nine real-world datasets. Despite the simplicity of our model and its one-shot training approach, LCESN achieves competitive results, even surpassing several state-of-the-art models. LCESN introduces a fresh approach to real-world time series forecasting and demonstrates that large, well-tuned random recurrent networks can rival complex gradient-trained feedforward models. We provide our GPU-based implementation of LCESN as an open-source library.

Published as a conference paper at ICLR 2025
LOCALLY CONNECTED ECHO STATE NETWORKS FOR TIME SERIES FORECASTING
Filip Matzner
František Mráz
(filip. mattner, grantisek.nng mraz @mff cuni, cz
Charles University, Faculty of Mathematics and Physies
ABSTRACT
Echo State Networks (ESNs) are a class of recurrent neural networks which only small readout regression layer is trained, while the weights of the recurrent network, termed the reservoir, are randomly assigned and remain fixed, Our work introduces the Locally Connected ESN (LCESN), novel ESN variant with a locally connected reservoir, forced memory, and a weight adaptation strategy. LCESN significantly reduces the asymptotic time and space complexities compared to the conventional ESN, enabling substantially larger networks. LCESN also improves the memory properties of ESNs without affecting network stability. We evaluate LCESN's performance the NARMA 10 benchmark task and compare it to state- of-the-art models on nine real-world datasets. Despite the simplicity of our model and its one-shot training approach, LCESN achieves competitive results, even surpassing several state-of-the-ar arr models. LCESN introduces a fresh approach to real-world time series forecasting and demonstrates that large, well-tuned random recurrent networks can rival complex gradient-trained feedforward models. We provide our GPU-based implementation of LCESN as an open-source library.
INTRODUCTION
The field of time series forecasting (TSF) is currently dominated by feedforward, gradient- based neural networks, often based on the Transformer architecture introduced by Vaswani al. (2017). Recurrent neural networks (RNNs) lag behind feedforward models to the point where they are only rarely compared in state-of-the-art papers (Nie et al.) 2023). Despite being a natural fit for processing and predicting time series data, RNNs tend to be challenging to train (Pascanu et al. 2013).
Multiple approaches have been developed to avoid gradient pitfalls and facilitate the training of recurrent networks, with Long Short-Term Memory (LSTM) networks (Hochreiter and Schmidhuber 1997) being one the most popular methods. A lesser-known approach is the Echo State Network (ESN) introduced by Jaeger (200T), which completely avoids gradient-basee training by using large random recurrent network called a reservoir that remains untrained. The only trained component in ESN is small linear readout layer computed via linear regression. The basic idea is that the random reservoir generates wide variety of complex combinations of input data, making the extraction of almost any desired output a simple task.
The main advantages of ESNs over -rrainentttaieed models are resistance to overfitting, a fast (one-shot) training procedure, and simple architecture (Lukoševičius 2012 Jaeger 200T). Since the ESN is not trained via gradient descent, it can incorporate non non- con-differentiblle components and formulas with unstable gradients. ESNs have been successfully used for sequence prediction and sqquence-to-sqqeeee tasks, but on large real-world datasets, they have been overshadowed by modern large-scale gradient-based networks. Over the years, many authors have suggested improvements in reservoir topology and neuron design. In contrast, Matzner (2022) demonstrated that even a basic ESN can surpass those improvements when its hyperparameters are properly tuned. Unfortunately, even optimized ESNs have struggled to compete with ofate-of-the-ant models.
We have identified the following two limitations of conventional ESN approaches. The first is the quadratic time and space complexity of each ESN step, which, in practical terms, limits the network size to hundreds or a few thousand neurons. The second is that reservoirs with long memory make the network unstable and oversensitive hyperparameter perturbations.

---

Published as a conference paper at ICLR 2025
1.1 RESEARCH OBJECTIVE
Our goal is to address the limitations of ESNs and make them competitive with state-of-the-art models on real-world datasets. Rather than aiming for top rankings in every benchmark, we aim create baseline recurrent model with a distinct architecture significantly different from the industry standard. Furthermore, we deliberately choose to provide a model that can be built from scratch on consumer hardware in a reasonable time.
2 RELATED WORK
Feedforward neural networks (multilayer perceptrons: ML.Ps and recurrent neural networks (RNNs) have both been used in time series forecasting (TSF) (Zhang et al. 1998 Lipton et al., 2015) and are traditionally trained using gradient-based methods. However, as the context length and network size increase, the gradient magnitude can decrease or increase uncontrollably. This effect is known as the vanishing and exploding gradient, and it complicates the training process for deep models (Pas- canu et al. (2013). To overcome these issues, Long Short-Term Memory (LSTM) (Hochreiter and Schmidhuber 1997) and its variants, such as Random Connectivity LSTM (RCLSTM) (Hua et al. 2019), have been developed to facilitate long-tern information retention. Despite their advantages, these models are known for their computational complexity. Jaeger (2001) introduced the Echo State Network (ESN) designed to mitigate the high computational costs and the issues of vanishing and exploding gradients. ESNs are described in detail in Section 3.1
Following the tremendous success of Transformer-baseed models in natural language process- ing (Vaswani et al.] (017). Transformer- based models have been successfully adapted for TSF iTransformer (Liu al.] (024) applies independent attention across variates to build time series representations, which are combined using a feedforward network. PatchTST (Nie et al., (202)) leverages self-supervised pre-trane mre-training to represent fixed -enghh patches, enabling compressed input processing and allowing efficient training while incorporating longer history.
Multiple recent methods have explored time series decomposition, such as the Fourier transform employed by FEDFormer (Zhou et al.) 2022) and TimesNet (Wu et al.. 2023). While FEDFormer combines a Transformer with seasonal -trend decomposition method, TimesNet transforms the original I-D input into a 2-D representation processed by 2-D kernels. AutoFormer (Wu et al. 2021 introduces a novel decomposition architecture featuring an autocorrelation mechanism that discovers dependencies at the subseries level.
Notably, Zeng et al. (2023) introduced a simple univariate one-layer linear model DLinear that surprisingly outperformed the state-of-the-art transformer models by large margin. RLinear (Li ct al. 2023) extended the linear approach of DLinear with RevIn (Kim et al.) 2022). model-agnostic
www.liaattoonddennmmaaizizii method capable of eliminating the distribution shift within time series.
Finally, the recent introduction of TSMixer (Chen et al., 2023) marks a significant advancement, surpassing the aforementioned models. TSMixer alternates between time-mixing and feature-mixing MIPs, akin to the transposition processes of I'Transformer. However, it uses MI.Ps instead of transformers with complex attention mechanisms.
3 METHODS
3.1 ECHO STATE NETWORKS
basic Echo State Network with neurons, nia inputs, and nout outputs consists of reservoir represented by a connectivity matrix W e R=*" an input weight matrix W E Raxmin a readout weight matrix Wout Xn and a feedback weight matrix Wib gaxnous (see Figure II. Neuron activations at time t are denoted by a(t) E R" the input values by u(t) E R the output values by ,=)t E Roo and the target by yt) E Re The following recurrence represents one discrete step of an ESN, transitioning it to the next time point:
2(t) Wa(t 1) + W., u(t) + W Wroz(t - 1) + pb.
a(t) = (1 - 7))((t - 1) + tan)(
(1)
x(t) Woura(t),
2

---

Published as a conference paper at ICLR 2025
U1
W out
teacher-fordmm
prediction
Wb
feedback
Figure 1: Overview of Echo State Network.
where z(t) E I represents the pre-activation potentials before being processed by the tanh activation function, & is the leakage parameter, and lb E IR denotes constant bias shared by neurons.
The initial activation a(0) is set to zero. The reservoir weights W are generated from the uniform distribution U(pres dress /rres + rres), the input weights for the i-th input W from U(-o in In). and the feedback weights for the j-th output W), from the uniform distribution U(-ojo: 76)) (.), for
hyperparameters Pres E IR, Fres, oin o/ E RR.. Note that we specify the parameters of uniform distribution using its mean l and spread Ø, rather than its borders, smooth the fitness space for hyperparameter optimization. After feeding the training data into the network, the output matrix Wout is trained using ridge regression to minimize the squared error between the predicted sequence att) and the target sequence y(t) (more details in Appendix C). The network output is undefined until the training procedure sets out. Until then, the feedback connections are driven by the target sequence yt)) Using the target sequence instead of the network's own output is known teacher forcing (Jaeger 2001 Lukoševičius 2012).
ESNs represent dynamical system whose behavior can range from stable, unchanging state to a chaotic white noise generator, depending on their configuration (e-g., Tres). Between these two regimes lies the so-called edge of chaos. which, according to some authors. is the regime where the network performs best (Bertschinger and Natschläger 2004 Matzner Boedeckee et al.) 2011]. In order to avoid the chaotic regime, the network has to exhibit the echo state property (also called fading memory), meaning that its state depends only on a finite history of its inputs (Jaeger 2001). One of the metries used to measure chaoticity is called the Lyapunov exponent, which can be estimated by introducing a tiny perturbation into the reservoir and measuring how much the perturbed and unperturbed networks diverge. Negative Lyapunov values denote order, positive values denote chaos, and zero indicates the edge of chaos. For more details on its properties and the estimation algorithm, see (proot (2003).
3.2 HYPERPARAMETER OPTIMIZ.ATION
We use the hyperparameter tuning framework by Matzner (2022). based on the Covariance Matrix Adaptation Evolution Strategy (CMA-ES) (Hansen and Ostermeier 200T). For more details on CMA-ES, see Hansen (016), and consult Matzner (2022) for use ESN hyperparameter tuning.
The optimized hyperparameters include the reservoir scaling parameters dres, Pres, the bias term μab the input scaling parameters ofnn for each input, the feedback scaling parameters 071 for each output (only for sequence- to-eueee---eeencce tasks), and the ridge regression regularization coefficient X2 RR.. In total, this amounts to + nin + Tout hyperparameters, where 7in and nowt are the numbers inputs and outputs, respectively, and depend on the task. For details on optimizer settings and parameter limits, please refer to Appendix D
3.3 LOCAL TOPOLOGY
Conventional ESNs introduced by Jaeger (2001) use a fully connected reservoir, making the state transition computationally expensive in both time and memory. Some authors have proposed reservoir topologies with lower time complexity, such as ring or chain topologies (Rodan and Tino (01T), but these do not provide the same performance as conventional fully connected sparse ESNs (Matzner
3

---

Published as a conference paper at ICLR 2025
Figure 2: Demonstration of local topology grid of 4x6 neurons and 3x3 kemnll. Two non- overlapping kernels are visualized; one wraps around the top edge.
Analysis of reservoirs designed using the HyperNEAT evolutionary algorithm (Stanley al. 2009) suggested that placing neurons on grid and connecting each neuron only within its neighborhood may be beneficial for increasing the network's 's stability (Matzner (017). A similar idea has also been proposed under the term small-world connectivity (Watts and Strogatz 1998). where, in addition to neighborhood connections, neurons occasionally form longer connections, thus reducing the network's 's diameter. Small-world reservoirs have also been studied, e.g., in the fields of biologically plausible networks and neuromorphic hardware, where they provide better wiring cost than fully connected networks (Kawai et al. 2017 Suárez et al., 2021 Daniels et al. (022).
2022).
We propose local topology that reduces the asymptotic time complexity of the ESN step without compromising accuracy. This allows for larger reservoirs at the same computational cost. Neurons are placed on torus-shaped grid, limiting the maximum distance between them, similar to small- world connectivity. Each neuron is connected to all neurons within its K x K neighborhood (see Figure 2 Adopting the notation of convolutional neural networks, we term the weights for this neighborhood kernel. However, contrary to convolutional layers, each neuron has its own kernel, and no weights are shared. The weights are stored in tensor W E IR N x where NxM represents the reservoir dimensions (i.e., N rows and M columns, totaling NM = neurons).
With fully connected reservoir, the weights were stored in a larger matrix W R.NXMXNXM Therefore, assuming a naive implementation of matrix multiplication, this technique reduces both the time and space complexity of the ESN step from D(NM)) ONN K2). Furthermore, the proposed topology is designed with GPUs in mind, allowing efficient (coalesced) access to GPU memory and avoiding the overhead of generic sparse matrix computation. According to our measurements, GPU implementation provides up to 15x speedup compared to a fully connected GPU for a network of size 80 x 100 and 7 x kernel, For details on
reservoir on a consumer-gee GPU implementation and comparison with other methods, please refer Appendix F
It may be tempting to use the same kernel for all neurons and turn the local topology into an even more efficient "recurrent convolutional network". Unfortunately, as presented in Appendix E.3 using convolutional kernels does not provide the same performance as the proposed topology.
3.4 FORCED MEMORY
The stability of ESNs is intrinsically linked their memory capacity. In conventional ESN, the network's s entire memory is implicitly encoded within its state, propagating sequentially from one time step to the next. However, since the reservoir is composed of randomly initialized weights, it does not differentiate between useful information and noise present in the data. As the memory capacity of the network increases, its sensitivity both signal and noise grows accordingly. Eventually, this heightened sensitivity can lead to the network being overwhelmed by noise, pushing it into chaotic state. To prevent the network from crossing the edge of chaos, its memory capacity must be limited, which unfortunately restricts the network's 's ability to retain longer context.
Many authors have proposed methods to improve memory capacity (MC). To give a few examples, Holzmann and Hauser (2009) added trainable delays between the reservoir and the output layer, Echo Memory-AAgmentee Network (EMAN) by Ma et al. (202I) employs a complex low-learning weight attention mechanism, and Distance- Settacce-eelyed-ee Nelwyed-Networks (DDN) by lacob and Dambre (2024) rely physics-inspee distance- based delays. Our approach is more straightforward. It simply gives each neuron lookback connection its own historical state random fixed time horizon. Despite its
4

---

Published as a conference paper at ICLR 2025
mm(-1)
a(--5)
af:-4)
a(t-3)
a(t-2)
a(t-1)
vtt)
Figure 3: An example of forced memory with maximum delay И = 5. The vector v(t), combining memory m(t 1) with last state a(t 1), is used instead of a(t 1) during state transition.
simplicity, this method avoids the need to propagate the entire memory through every step and helps the network to remain in the stable regime.
Formally, in memory forcing, the activation a(t); of neuron at time / directly combined with its historical activation a(t h2); The delay h2 and the combination coefficient (eme ) randomly chosen for each neuron separately and remain fixed. The recurrence equations are modified as follows, with Figure illustrating this process.
v(l) Wmem ml( 1) + (1 Wmemm) a(l 1),
z(t) Wo(t) + (tn + Ws 1) +bb
a(t) tanh(=(t)),
x(t) = Woura(t),
m(t): h4); Vi
where denotes elementwise multiplication, m(t) R" is the memory vector with each element equal to a historical activation, Wmem E R" represents the preference of the historical state over the current state, and u(t) R denotes the state combined with its memory. The coefficients Wmem for the affine combination are generated from a uniform distribution U(-1, The delays h; are initialized random integers from uniform distribution UO0 H 1). where H denotes the maximum allowed delay (set to 100 in our experiments). Note that we have omitted the leakage coefficient y because its role is fulfilled by the memory vector; fact, leakage can be simulated by setting h; = 1 and (Wmem);
3.5 PREDICTION METHODOLOGY
To be comparable to the state-of-the-an feedforward models TSF benchmarks, we have used the same sainning/aalidationneesii split and the same data normalization technique (Nie et al. 2023). However, the use of recurrent networks fundamentally differs from that of feedforward models. Feedforward models expect input of fixed length Tin and predict output of fixed length Toutt- The dataset is created by extracting patches of length Lin + Toout from the original time series and feeding them the model independently. Having fixed input and output length has few benefits, such as the ability to process many patches as single GPU batch and the flexibility to feed random patches the model until training convergence. However, it also comes with few downsides. For instance, since the patches are independent, it is often necessary to encode the position of the patch within the whole sequence (e.g.. the day, week, and year in the case of real-world datasets). Additionally, separate model must be trained for each prediction horizon Loutt
RNNs, on the other hand, conventionally predict the immediately following time point, and the user can repeat this step many times as desired to predict the whole horizon. Unlike feedforward models, ESNs inherently track their position the input sequence, removing the need for explicit position encoding due to their potentially unlimited context length. In fact, we do not enhance the input data in any way; we pass them to the network in their raw form. However, a natural downside of recurrent models with long memory is the necessity of feeding the entire time series up the moment of prediction (or at least a substantial part of it). Fortunately, in the case of ESNs, training and prediction rather fast, and we do not need to propagate any gradient through the whole sequence.
5

---

Published as a conference paper at ICLR 2025
washout data teacher forcing
training data teacher forcing
validation data
teacher forcing
uf
U2
63
U5
U6
U7
uB
training Waut lin. reg.
u9
u10
u12
ina
ut4
clone
prediction data self-feeding
XI
Figure 4: Overview of the prediction procedure.
Ultimately, for a fair comparison with the state of the art, we need to evaluate our model on all l Lout 1 subsequences of the testing data while running solely a self-feeding loop (without the ground truth data), where l is the length of the test data. The testing procedure is demonstrated in Figure 4 and works as follows. The network is fed with the washout sequence, and its outputs and states are ignored. Afterwards, the network is fed with the training data (teacher forced) up to the beginning of the first validation subsequence. Next, the output weights are trained via linear regression (as described in Section 3.1 The network is now ready to be used for a self-feeding prediction loop.
A temporary clone of the network is created and driven by its own output for Lout steps. That is, at time 4, uses its own output x=1 - 1) as input. Its outputs are recorded and compared with the target sequence. The clone is destroyed, and the original network is fed with the next time point (one step with teacher forcing). This process of cloning, prediction, and updating is repeated until the cloned network reaches the end of the validation sequence.
An important topic is the output weights (Wout) adaptation strategy when feeding the validation data with teacher forcing. Let us list few options ordered by their computational and memory complexities and their notation used throughout the paper.
1. LCESN: Keep the output weights fixed after the initial training.
2. L.CES-LLMS: Update the weights via a simple signal filter. There are many approaches addressing the online adaptation of ESNs, such the popular Recursive Least Squares method called FORCE (Sussillo and Abbott 2009), more computationally effective pagproppagation--eerrreatiooooo (Steil 2004), and many others (Lukoševičius) 2012). The evaluation of all existing methods is beyond the scope of this paper, and we chose Normal- ized Least Mean Squares (NLMS) (Haykin 2002) as a minimalist baseline.
3. LCESN-LRION 1000: Inherit the filtering from LCESN-LMS, and recalculate the full linear regression after every 100 steps. This option balances speed and accuracy, and the results will later show that a shorter interval would bring about negligible effect.
4. LCESN-LRE Recompute the linear regression before every self- self-fceding prediction loop. This option requires impractical computational resources, but represents an upper bound on the accuracy of online learning.
Note that all of these methods use the same byperparameter-oppiiizee network and only differ in the evaluation procedure. Therefore, there is no need to repeat the hyperparameter optimization, which is the most time-consuming part of the process. In Section we will present practical measurements on real hardware.
DATASETS
We use the NARMA10 dataset, widely used in ESN literature, along with nine real-world datasets commonly utilized in state-of-the-aat sequence forecasting studies. From longest to shortest, these datasets are ETTml, ETTm2 (Electricity Transformer Temperature), Weather, Solar Energy, Electric- ity, Traffic, ETThI, ETTh2, and Exchange. We focus only on the multivariate case, i.c., predicting all available features. For more detailed description of the datasets, please refer to Appendix A

---

Published as a conference paper at ICLR 2025
10
state size
10
20x25
28x36
40x50
10
80x100
10 11
113x141
MSE
12
10"
14
10 15
10"
17
11x11
15x15
19x19
kernel
Figure 5: Comparison of various network and kernel sizes NARMA10 benchmark tasks. Each violin represents the evaluation of 100 unique random sequences predicted by a network selected as the best of five optimization runs.
EXPERIMENT
First, we test the basic LCESN without forced memory and weight adaptation the NARMAI0 benchmark to demonstrate the effect of the local topology. Second, we evaluate the performance of LCESN in practical terms of consumed time and memory resources Third, we evaluate the effect of forced memory on ETTm datasets and compare LCESN with the conventional ESN. Finally, compare the four full-featured LCESN variants presented in Section 3.3 with state-of-the-arr models in time series forecasting on real-world datasets.
consumer geradd GPU.
To enable building model from scratch consumer hardware, limit each hyperparameter optimization run to 2000 evaluations, ensuring it fits within a 24-hour deadline on an older NVIDIA GTX 2080 Ti GPU (released in 2018) and single core of average desktop CPU. For a fair comparison, we use the same hyperparameter optimization technique for baseline ESN and for the proposed LCESN models. Further technical details the experimental setup are provided in Appendix D
6 RESULTS AND DISCUSSION
6.1 EFFECT OF LOCAL TOPOLOGY
We evaluated grid of network sizes from 1,000 to 16,000 neurons and kernel sizes from 3x3 to 19x19 on the widely used NARMA 10 benchmark task. The model used the vanilla LCESN without forced memory and weight adaptation (i.e., a conventional ESN with local topology). As demonstrated in Figure when the network size increases, the eπтог on NARMA drops by orders of magnitude. Furthermore, the kernel size plays much less significant role, with 7x7 kernel representing the best accuracy while still providing acceptable computational demands. An important observation that increasing the kernel size further, and thus approaching the fully connected conventional ESN, has not provided any benefit. In fact, the results with the 19x19 kernel are statistically significantly worse (p 0.05) compared the 7x7 kernel for the two largest networks.
The results suggest that on the NARMA10 benchmark task, the widely used network sizes of 100 to 1,000 neurons simply too small, and with proper hyperparameters, the performance depends mostly on the size of the network. Needless to say, the enror on NARMA10 is below an interesting threshold, and it should longer be used to compare state-of-the-arr results in the Echo State
7

---

Published as a conference paper at ICLR 2025
600
madel
7000
model
500
Icesn
max memory [MIB]
600D
Icesn
esn
Icesn-Ir100
5000
Icesn-Ir100
time (
4000
300
3000
2000
100
1000
0
0
20x25
28x36
40x50
57x70
80x100
20x25
28x36
40x50
57x70
80x100
state size
state size
Figure 6: Wall-clock time and memory resources required to evaluate the entire ETTh1 dataset using the models presented in Section 3.5 with 7x7 kernel and known hyperparameters. The measurements encapsulate all the evaluation phases: generating the network, the washout period, linear regression training, and test prediction. The ESN model represents the conventional ESN with a full connectivity matrix; LCESN reduces it to local topology and adds forced memory; and LCESN- LR100 additionally performs regression retraining every 100 teacher forced steps.
Network literature. However, it remains convenient, hard-to-overfit benchmark for checking basic properties ESNs and validating new ideas for peaueence-t-sequenne tasks.
Note that the network size increases, the improvement gradually diminishes. One possible explanation that the size of the training data (12,000 steps) insufficient for the largest network (16,000 neurons). However, further increasing the size of the training set has not significantly improved the results (see Appendix E.I for details). For comparing various state sizes and alternative topologies, such convolution, please refer to Appendices E.2 and B3
6.2 BENCHMARKING
The effect of the lower time complexity of the local topology is demonstrated in Figure 6 Notably, in the case large networks, the gain from the faster LCESN step fully compensates for the time needed to recalculate the linear regression every 100 steps (LCESN-LR 100). However, the recalculation consumes a lot of memory resources; therefore, it must be used with caution on large networks and datasets. The measurements were performed on NVIDIA GTX 1080 Ti GPU (released in 2017) supported by single core of an Intel i7-4770 CPU. Training and evaluation took under 40 seconds for 40x50 network.
Building the network from scratch also requires careful tuning of its hyperparameters (Matzner 2022).
In all our experiments and for all tested models, the optimizer is limited to 2,000 full evaluation cycles. Each cycle consists of generating new network, performing regression training, and predicting the validation sequence. the case of the ETThl dataset, this procedure took less than four hours. Each of our models is selected as the best from five hyperparameter optimization runs (on the validation set). Therefore, it took 20 hours total to train the ETThl model.
For more detailed decomposition and comparison with other models, please refer to Appendix G
6.3 EFFECT OF FORCED MEMORY
In this experiment, compared conventional ESN without forced memory and LCESN with increasing forced memory horizons. None the tested models uses weight adaptation, meaning that the LCESN without forced memory denotes ESN with local topology. The hyperparameters for all models, including the baseline ESN, are optimized using the same procedure described in Section 3.2
Figure 7 shows the validation errors of five separate hyperparameter optimization dataset. Disabling forced memory or setting its maximum horizon to less than 50 steps increases noise, hindering the hyperparameter optimizer's ability to estimate the gradient and leading substantially
the ETTml
8

---

Published as a conference paper at ICLR 2025
0.7
MSE
0.6
0.5
ESN memory
LCESN H-25
LCESN H+200
memory
H+50
M=100
Figure 7: Comparison of the final MSE reached by the hyperparameter optimization on the validation sequence for five separate runs on the ETTml dataset. H denotes the forced memory horizon limit. The reservoir size is 40x50 with 7x7 kernel.
Lyapunov
-0.25
-0.50
-0.75
-1.00
Figure 8: Lyapunov exponent versus forced memory horizon for LCESN runs evaluated in Figure 7
worse results with higher variance. Local topology is only supposed to reduce computational costs, increase the accuracy. Therefore, unsurprisingly, the ESN baseline has performed similarly to LCESN with short memory horizons. As side note, forced memory and local topology do not appear to interact on the ETTml task (see Appendix [or further details).
Figure g demonstrates the relation of forced memory and the network dynamics. Limiting the forced memory horizon clearly pushes the network toward the edge of chaos. In contrast, longer forced memory horizons allow the network to stay in stable regime while still benefiting from long memory, The right choice of maximum memory horizon depends on the dataset hand. According Figure [ for the ETTml dataset, 50-step horizon limit would likely suffice, but other datasets may require longer memory. Therefore, we will use a memory horizon limit of 100 steps for the rest of this work.
6.4 REAL-WORLD DATASETS
We adopted the results of third-party models from Nie et al. (2023) and Liu et al. (2024), except for the results the more recent TSMixer, which are obtained directly from its original paper Chen et al.. (2023). Furthermore, Liu et al. (2024) only provide results for prediction length of 96 time points, whereas Nie et al. (2023) provide stronger basclines by selecting the best results from six different prediction lengths for the Transformer-based models. When both baselines available, we opt for the stronger results from Nie et al. (2023). TSMixer was not tested on the Solar Energy and Exchange datasets its original paper, so those results are omitted from the comparison table, and TSMixer not considered in their ranking. However, since it was the overall best-ranked model, it is possible that it would also perform well the omitted datasets.
The comparison with existing third-party models and conventional ESN is provided in Table For complete results with all prediction horizons and for relative ranking between models, see Appendix All four introduced LCESN variants improved upon the conventional ESN on all datasets, highlighting the effect of forced memory. The three weight adaptation methods implemented by LCESN-LMS. LCESN-LRION and LCESN-LRI I notably improve the results of the basic LCESN. Therefore, if the data frequency and computational resources allow, as in the case of the evaluated real-world datasets, regular retraining can yield better performance. LCESN-LRI. however, did not significantly improve upon the results of -CESN-LRR00, so its high computational demands have not proven to be justifiable on the tested datasets. According the relative ranking across all datasets and horizons (Appendix B.2), LCESN-LRI clinched the second place and LCESN-LEE 100 the third place among all the evaluated models.

---

Published as a conference paper at ICLR 2025
Table 1: The results of the LCESN approach on real-world datasets compared the results of other
authors. Every number is the average over four prediction horizons 96, 192. 336, and 720. The best
result is red and the second best is blue. The four LCESN variants and the ESN are not compared to each other. The reservoir size for ESN and LCESNs is 40x50 with a 7x7 kernel. Datasets are sorted from longest (top) to shortest (bottom).
Models
ESN
LCESN
LCSSN-LMS LCESN LCESN LR1
TSMiter
Dliseer
PuchTST
iTunsfomee FEDioemer
Timeshett
RLisez
Aunofomeer
2022
Metric
MSSE MAF
MSE MAR MMS
MAB
MMSE
MAB
MSS
MAR
MSE
MSE MSE MAR
MAIIMSE
MSB
MAE
MAE, MSS MAR
ETTml
0:527
YL412 3534
0.303
10362
0.312
0,359
0,300
[0.47 0.375ju.40 0.417 0.00|o.. 0,110 j0332 0:422 (:400 406 0.444 0.408 0.515 0,495
ETTm2
|0,550.0
11201
0.30
16.50
0.255
0.126
20.20
0.323/o 6.401 j0:28:
0:128fu.25
0.332
j0.292 0.33-0.3333...00
Weather
[0.255
0.215
21222
0,279
11230
0.222
16,21
0.264
10.23
0.264
10.24 0.204p..25 0.317ju
0.20/ju.25
0,238
10.310 237 (0225 286 0.272 291 0.335 0,31
Solar
10225 0.384
(1220
0.272 10202
0.268
10.301
0.223
0.201
0.214
N/A NIA Ja 50,50 10.500000000 200
9.262
10301 0.3: 9 359 o 0.555 0.711
Electricity
0.351
0.329 1119T
0.308
[0.154
0.214
0.185
0.293
[0.100 6:377/u 0.30
0.20o
0.211
(0.20 0.321
0.218 0.25
Traffic
|1005 0:53 j1882
a 407 justs
0442
|0,00)
0.392
0.782
0.392
[0.00 0.20:]
0,282
3.3 6.62 n 0666 0.364
ETThl j0.550 (.811
p.s 0,500
|a.500
0.518
0.541
0.521
|0.412
ossoju o4siffus54
0:448
15.128 4:4 90-458 455
446 0:434 0.473 0.477
[0,559 0.529
(.429993333435 j0435
11372
464 11a72
9419
|0,37
0421
0.311
0,422
[0,55
0.s0 o511|u3s o.injuu33 0.406
j0.358
0.43 (0-414
0.37% 0,398
Exchange|j... (0.703 0.575 ju899
j0718
0s02
Ju866
0.524
0.699
0.566
NA
NIA
oarajo qumil le) 0.483
jo.518
0.429
p.416 443 p. 8: 0.615 0.539
10
LCESN variants were most successful on the ETTm1, ETTm2, Weather, and Solar Energy datasets, even clinching first place in several cases. One of the reasons for the success those particular datasets might be their length, as they represent the four longest of the nine tested series, and the difference in dataset length is substantial. Each of the four listed datasets has more than 52,000 time points, while each of the remaining five has fewer than 27,000 time points. This suggests that to fully utilize the potential of LCESN, tasks need to provide sufficient amount of training data. For more details on dataset properties, see Appendix A
We used a medium-sized model of size 40x50 to reduce the experiment duration. Whether smaller or larger network could further improve the results remains an open question.
7 CONCLUSION
In recent years, recurrent neural networks (RNNs) have been overshadowed by gradient-trained feedforward architectures, to the point that modern time series forecasting papers often do not even include them in comparisons due to their lack of recent successes.
To revitalize research in this area, we revisited the Echo State Network (ESN) approach - an older, often overlooked yet promising method - and addressed two of its limitations. First, introducing a lo- cal topology allows for much larger networks at the same computational costs. Second, incorporating forced memory facilitates longer time dependencies and enhances network stability.
The resulting Locally Connected Echo State Network (LCESN) is simple and free of gradient descent. It is fast to train and infer after its hyperparameters While not designed to surpass every existing method, it demonstrates competitive accuracy with state-of-the-ars models, even outperforming them on some of the longest of the tested datasets.
set, and can predict arbitrary horizons.
Our model can be built from scratch on consumer-grade hardware in under 24 hours, with most of that time devoted to hyperparameter optimization area offering significant room for future improvements. Once hyperparameters are known, training and inference typically take seconds or minutes, depending on the dataset size.
We have released our C+/CUDA implementation of LCESN under a permissive license at our GitHub repository (https: /gubbht..b..... com/Floopder /echo-chooeee seehcchttaeeeeee wetkksss,s together with logs, network checkpoints, and fixed random seeds to ensure reproducibility.
10

---

Published as a conference paper at ICLR 2025
8 ACKNOWLEDGEMENTS
The work was supported by the grant NVV-2025-260822 Computational resources were provided by the e-INFRA CZ project (ID:90254), supported by the Ministry of Education, Youth and Sports of the Czech Republic.
REFERENCES
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez. ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In International Conference Learning Representations, 2023.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In Proceedings of the 30th International Conference International Conference Machine Learning Volume 28, ICML' 13, page 1310-1318. JMLR.org 2013.
Sepp Hochreiter and Jürgen Schmidhuber. Long Short-Term Memory. Neural Computation, 9(8): 1735-1780, 11 1997. ISSN 0899-7667. doi: 10. 1622neeo..... 199t)/ecc...1...99.. 1997.998.. 177155..
Herbert Jaeger. The "echo state" approach to analysing and training recurrent neural networks-with an erratum note'. Bonn, Germany: German National Research Center for Information Technology GMD Technical Report, 148, 01 2001.
Mantas Lukoševičius. A Practical Guide to Applying Echo State Networks, pages 659-686. Springer Berlin Heidelberg, Berlin, Heidelberg, 2012. ISBN 978-3-642-35289-8 doi: 10.1007/ 978-3-642- 688-3--4-23529--88 36.
Filip Matzner. Hyperparameter tuning in echo state networks. In GECCO '22: Proceedings of the Genetic and Evolutionary Computation Conference. page 404-412, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 978-1-4503-9233-22 doi: 10.1 (0.1455522229935587211
Guoqiang Zhang, B. Eddy Patuwo, and Michael Y. Hu. Forecasting with artificial neural networks:
The state of the art. International Journal of Forecasting, (4(1):35-62 1998. ISSN 0169-2070. doi: 10. 10.100//S016 (006000009-0700777000047777
Zachary C. Lipton, John Berkowitz, and Charles Elkan. A critical review of recurrent neural networks for sequence learning. arXiv e-prints, 2015. doi: 0005550/rXxii1156.00000000
Yuxiu Hua, Zhifeng Zhao, Rongpeng Li, Xianfu Chen, Zhiming Liu, and Honggang Zhang. Deep learning with long short-term memory for time series prediction. IEEE Communications Magazine, 57(6): 11--119, 2019. doi: 10.1 109/CCMM.2099 1800155.
Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. iTransformer: Inverted Transformers Are Effective for Time Series Forecasting. arXiv preprint arXiv:2310.06625, 2024.
Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. FEDformer: Frequency enhanced decomposed transformer for long-term series forecasting. In Proc. 39th International Conference on Machine Learning (ICML 2022), volume 162 of Proceedings of Machine Learning Research, pages 27268-27286. PMLR, 2022.
Haixu Wu, Tengge Hu, Yong Liu. Hang Zhou, Jianmin Wang, and Mingsheng Long, Timesnet: Temporal 2D-variation modeling for general time series analysis. In International Conference on Learning Representations, 2023.
Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with Cnto-Correlation for long-term series forecasting. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems, pages 22419 22430, 2021.
11

---

Published as a conference paper at ICLR 2025
Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? Proceedings of the AAAI Conference on Artificial Intelligence, 37:11121-11128, 06 2023. doi: 10.1609/aaii.v33177 v3719.26317.
Zhe Li, Shiyi Qi, Yiduo Li, and Zenglin Xu. Revisiting long-term time series forecasting: An investigation on linear mapping. arXiv e-prints, abs/2305. 10721, 2023.
Taesung Kim, Jinhee Kim, Yunwon Tae, Cheonbok Park, Jang-Ho Choi, and Jaegul Choo. Re- versible instance normalization for accurate time-series forecasting against distribution shift. In International Conference on Learning Representations, 2022.
Si-An Chen, Chun-Liang Li, Sercan O Arik, Nathanael Christian Yoder, and Tomas Plister. TSMixer: An alI-MLP architecture for time series forecasting. Transactions on Machine Learning Research, September 2023. ISSN 2835-8856.
Nils Bertschinger and Thomas Natschläger. Real-time computation the edge of chaos in recurrent neural networks. Neural computation, 16(7):1413-1496, 2004.
Filip Matzner. Neuroevolution on the edge of chaos. In GECCO '17 Proceedings of the Genetic and Evolutionary Computation Conference, pages 465-472, New York, NY, USA, 2017. ACM. ISBN 978-1-4503-4995 doi: 10. 10.1145/3071 0055//307117.30711555 178.3071292.
Joschka Boedecker, Oliver Obst, Joseph Lizier, Norbert Mayer, and Minoru Asada. Information processing echo state networks at the edge of chaos. Theory in biosciences Theorie den Biowissenschaften, 131:205-13, 12 2011.
doi: 10.10007/1200---11-11444
J.C. Sprott. Chaos and Time-series Analysis. Oxford University Press, 2003. ISBN 9780198508403. Excerpt available online 2025-01-25:
Nikolaus Hansen and Andreas Ostermeier. Completely derandomized self-adaptation in evolution strategies. Evolutionary computation, 9(2)1599-15,, 2001.
doi: 10.1162//003365561)55011000099
Nikolaus Hansen. The CMA evolution strategy: tutorial. ArXiv, abs/1604.00772, 2016.
Rodan and Peter Tino. Minimum complexity echo state network. IEEE Transactions on Neural Networks, 22(1):131-144, 2011. doi: 10.1 N0.1109/TNN.200..00000
Kenneth O. Stanley, David B. D' D'Ambosi,, and Jason Gauci. Hypercube- Based Encoding for Evolving Large-Scale Neural Networks. Artificial Life, 15(2):185-213 04 2009. ISSN 1064-5462. doi: 10. 22.1166/aat2220915522.2111.
Duncan Watts and Steven Strogatz. Collective dynamics of 'soall-worldd networks. Nature, 393:440-442, 1998.
Yuji Kawai, Tatsuya Tokuno, Jihoon Park, and Minoru Asada. Echo in small-world reservoir: Time- series prediction using an economical recurrent neural network. In 2017 Joint IEEE International Conference on Development and Learning and Epigenetic Robotics (ICDL-EpiRob), pages 126- 131, 2017. doi: 10. 10.1109/DEVLRN..2 2017. 2017.8397777.
8329797.
Laura E. Suárez, Blake A. Richards, Guillaume Lajoie, and Bratislav Misic. Learning function from structure in neuromorphic networks. Nature Machine Intelligence, 30))771-786, Sep 2021. ISSN 2522-5839. doi: 0011038/422256-02--000111
Ryan Daniels, J.B. Mallinson, Z.E. Heywood, P.J. Bones. M.D. Amoll, and Simon Brown. Reservoir computing with 3D nanowire networks. Neural Networks, 154, 07 2022. doi: n0..0t6/j.neumet.. 2022.07.001.
Georg Holzmann and Helmut Hauser. Echo state networks with filter neurons and a delay and sum readout. Neural networks: the official journal of the International Neural Network Society, 23: 244-56, 08 2009. doi: nww.et/..cnnee2200000. 200004000.
Qianli Ma, Zhenjing Zheng, Wanqing Zhuang, Enhuan Chen, Jia Wei, and Jiabing Wang. Echo memory-augmented network for time series classification. Neural Networks, 133:177-192, 2021. ISSN 0893-6080. doi: 10.10066j..eunet.200 2020. 10.015.
12

---

Published as a conference paper at ICLR 2025
Stefan Iacob and Joni Dambre. Exploiting signal propagation delays to match task memory requirements in reservoir computing. Biomimetics, 9(6), 2024. ISSN 2313-7673. doi:
10.390/biomimmcics9965555
David Sussillo and L. F. Abbott. Generating coherent patterns of activity from chaotic neural networks. Neuron, 63:544-557, 2009.
J.J. Steil. Backprropgation-decorelaaarr online recurrent learning with O(N) complexity. In 2004 IEEE International Joint Conference on Neural Networks, volume 2, pages 843-848, 848, 2004. doi: 10.1 20000/UCNN.200..388800
Simon Haykin. Adaptive filter theory. Prentice Hall, Upper Saddle River, NI, 4th edition, 2002.
Tomoyuki Kubota, Kohei Nakajima, and Hirokazu Takahashi. Dynamical anatomy of NARMAIO
benchmark task. ArXiv, abs/1906.04608. 2019.
Haoyi Zhou, Shanghang Zhang. Jieqi Peng. Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang Informer: Beyond efficient transformer for long sequence time-series forecasting. In The Thirty- Fifth AAAI Conference Artificial Intelligence, AAAI 2021, Viriual Conference, volume 35, pages 11106-1114 AAAI Press, 2021.
Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long- and short-term temporal patterns with deep neural networks. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR '18, page 95-104, New York, NY, USA, 2018. Association for Computing Machinery. ISBN 9781450356572. doi: 10.1145/3209978. 3210006.
Claudio Gallicchio and Alessio Micheli. Reservoir topology in deep echo state networks. In Artificial Neural Networks and Machine Learning- - ICANN 2019: Workshop and Special Sessions, pages 62-75, Cham, 2019. Springer International Publishing. ISBN 978-3-030-30493-55
Nathan Bell and Michael Garland. Implementing sparse matrix- multiplication throughput- oriented processors. In Proceedings of the Conference on High Performance Computing Net- working, Storage and Analysis, SC '09, New York, NY, USA, 2009. Association for Computing Machinery. ISBN 9781605587448.
Pavan Yalamanchili, Umar Arshad, Zakiuddin Mohammed, Pradeep Garigipati, Peter Entschev, Brian Kloppenborg. James Malcolm, and John Melonakos. ArrayFire A high performance software library for parallel computing with an easy-to-use API, 2022. URL https : : :ttp::://giiiie http//gttttu.u..o///////tt com/ arrayf www.rreaaaaaraaeeeeeeeeaaa
13

---

Published as a conference paper at ICLR 2025
Table 2: Sizes of the real-world datasets measured in the number of time steps.
ETTm1/2
Weather
Solar
Electricity
Traffic
ETTh1/2
Exchange
Training Validation
34560
36887
36792
18412
12280
8640
5311
11520
5270
5256
2632
1756
2880
760
Test
11520
10539
10512
5260
3508
2880
1517
Features
7
21
136
321
862
7
A DATASETS
A.1 NARMAIO
In Nonlinear Autoregressive Moving Average 10th order (NARMAI0), each element a nonlinear combination of the last 10 imputs and outputs. Formally, the target sequence is defined follows:
(l + 1) = (1) ()) + 0.05y E.- v(t - + 1.5u(L 10)u(() + 0.1,,
where u(t) the random input sequence generated from uniform distribution (0, 0.5) and y(t) is the target sequence, both in time f. It should be mentioned that NARMAIO is a chaotic sequence that can diverge (Kubota al. 2019), which case has be regenerated.
We use a washout period of 1000 steps, training period of 12000 steps, and a testing period of 1000 steps. For evaluation, we generate 100 unique NARMA10 sequences (using different initial random seed).
A.2 REAL-WORLD DATASTETS
We use nine publicly available datasets ETTh1, ETTh2, ETTmI, and ETTm2 proposed by Zhou et al. (202T), weather by Wu al. (202T), and electricity, traffic, solar-energy. and exchange datasets used by Lai et al. (2018). We focus only on the multivariate case, i.e., predicting all the provided features. For brief description of the contents of the datasets, Appendix A. from Liu et al. (2024).
To be comparable to other works, we normalize each covariate in the training set independently to zero mean and unit variance. We apply the normalization coefficients computed on the training set to the rest of the data (validation and testing set). The reported results are based on the normalized data (Chen al. 2023). See Table for the dataset sizes and consult the source code of the original papers (Python) or ours (C++ for more details.
We noticed order-of-magnitude outliers in the weather dataset training data. would be misleading to fit those using the least squares method (Section Cp. Therefore, we preventively clip the normalized training data and the network input to the interval -100; 10] for all the datasets. The validation and testing data are left untouched.
14

---

Published as a conference paper at ICLR 2025
B FULL RESULTS ON REAL-WORLD DATASETS
B.1 DETAILED VIEW
Table 3: The full results of LCESN approach real-world datasets compared to the results of other authors. The best result is red and the second best result is blue. The reservoir size for ESN and LCESNs is 40x50 with a 7x7 kernel. The four LCESN variants and ESN baseline are not compared to each other. Datasets are sorted from longest (top) to shortest (bottom).
Models
LCESN
LCESN-LAS
LCESN-LRIOO LCESN-LRI
TSMiscr
PaacTST
Mlincar
iTrandienner
RRDfeenne
TimesNet [003]
Rlincar
Astifienc AVC
Ours
Ours
Ours
Ours
H0257
20233
Metric
MSE
MARl
MSE
MAR MSI
MAR:
MSE
MAI
MSSE
MAl
MSI
MARI
MSI
MSE
MALI
MSI
MAB
MSI
MAR:]
MISE:
MAL:
MSE
MA
MSE MAR
0.392
0.425
8.297
0.350
0.391
0.341
0.299
0.343
0.225 0.330 322 0.357 0.372
0.39 0.375
0.37
192
0.475
0.43
0.562
0.397
0.311
0.277
0.355
0,322
0.532
0.271
0.337 9.365 0.367 0.385 0.380 0.389 0.377
0,991
0.365
0.355 0.387
0.391 0.380 0,514 0.312
0.495
536
0.55%
0.546
0.425
0.640
9.379
0.402
0.314
0.400
0.308
0.356 0.382 30990
0:410 6.433 0.410
0.413 0.426 0,420
0.435
0.410
0.411 11.22
0.15
0.492
720
0.734
0:656
0.554
0.524
R:442
0.442
0.440
0.433
0.443
0.450
0.419
0.414
454 0.430
0.439
3.474 0.453
0.491 0.439
0
0.458
0.478
0.450
p.467
0,450
0.52 0.550 0.493
Aug 00.5000.577/00
(1412
0.430
0.102
6.347
0.000ooo.4u 0.41n|u.00
0.410
(0.
0.42//
56 0.339
0.120 0.269 0.156
0.259
0.062
0.266
16162
0.256
0.163 0.252
0.175 0.259 00.93 0.290|o.130 0.264
0.259/0.193
0.277
(0.87 0.267 y.182 0.265/a.205 0.203
192
0.292 0.375
0,231
0.315 10223
0.301
0.118
0.209
0211
0.299
0.216
9.290
0.241
0,302
(.284
0.362 0.250
0.309
(
0,319
0.309
11246
0.30e 0.275 0.336
536
(0-402
0.440
1:305
0 166 0.263
0.342
0.322
0.50
0.571
0.19
0.268
0.314
0.345
(0169
0.427
0.311 0.348
( 0.360003-...000
0.321
0.551 1.307
0.342 0.543 0.399
0.630 0.573
0.436
0440
0.420
0.400/0.554
0.522
0.400
0.408
0.400
0.350 0.414
Aog
[0.300
200
0.384
0.332
0:387
1.33
0.324
112022
0.332
(0 1.2 0.303 [0 0.353/
221
0.332
0,333 11260 0.537/0.500 0.377
Wealber
96
00.172
0.220
1.0000000
0.213
10154
0.203
0,10
0.500
0.200
10.146
0.198
(0.96
0.255/o.174
0.214
0.2288 0.3144.37317
(1.12 0.250/0.249 0.329
0.224
0.205
0.259
0.257
0 0.329
0,219
0.261 112400
0.271
0.325 0.370
0.219
1:356
0.20
0.355
0.339
0:377
0:000
908
0.551
0.346
0.369
0.307
0.335
0.325
0.333
0:300
0.323
6.299
0.200
0.336
0.346
0.346.0344
0.345
0.38.
0:358
0.347
0
0.400
0.365
0.359
p.364
0.355 0.426
Aog 10.25 0:296
0.279
0.212
6:33
0.364
0.364
717
9,264
(1
0.280/o.26
0,317/0.256
1.77%
[
0,20 1.272 0.291
f0,335 0.579
0.221 0.355o.355
0.252
10.12
0,370
0.378
0.342.322/..250
0.202
0.330
Solar
0.254
0.360
0.195
0.207
0226
0263
0.264
NON
N7A.
(0.267
0.320
0.390
223
0.300
0
0.255
0.350
0.296
0.318 0035
0.350
0.834
0.022
0.231
0:210
0:24
1220
0.361
NIA
(0.200
9:5 101
0.223
0.319
9494 222
720
0.3004
0.422
0.228
303
9.225
0.266
8:236
0.300
1.225
0.301
NiA
NNA
(n
2.299 0.317 puss
0.413
0.340
0.275
0.337
0.427
0.338
0.337 p.397
0.717
10.200
0,201
0.218
6,301
10.301
NAA
NNA
0.307 30330
0,401
0.363
[0
09229
(0300
0.319 300 0.355 01:13 0:713
9n
10.247
0.337
0121
0.325
9.189
0:301
0.166
0.226
164
0.276
[0.031 0.229
0.270
0.282l
144
0.240
0.30/o.
0.222...200 10201 0.281 fo 195 0.913
0.272
192
0.250
0.346
0.220
0.325
0.190
0.302
0074
0.286
0.172
0.264
0.151
0.246
0.188 0.274
0.196 0.285
01162
0.253
0.1970.311
0.184 0.289 0.201
0.21 0.224
896
[1.268
0 155
0:21
0.306
0:307
01200
0:704
1.161 0.261
(1:304
0.205
0.700 o 10
1.13%
0:335
198 a 100 1::15
720
0.282
0.365
02240
0.33
0.339
0.213
0.320
02122
0313
0.212
0:312
0.197
0,293
0.324
(:265 0,330
0.225
0:31
0
0.233
0.340
0.200 0.320 0.257 0.351 0.236 0:342
Ang
[0.204
0.351
0.329 90199
0.300
0.194
0.254
[0.183
0.293
0.360 0.257
102 0220 0.300
111
1.270
0.207 0.331
0.205 0.200 0.2000000110000
0:326
96
[0.001
0.400
0.76
0.375 p.732
0.372
0.683
0.364
Ja577
0.363
10.376
0.264 (n 463
2.205 (0650 o 306ll
395
0.366
(0.576
0.990
0.32 (0.699 0.6490.3660 0.399 0.331
192
1.024
0.512
0.526
0.389
0.759
0.384
03733
0.355
0.729
0.374
0.397
0.277
0.605
0.206 0.598
0,370
0.276
0.610
0.330
0.617
0.335 0.601 0.607 0:392
1.142
0.110
n 405
0.450
0.991
0.900
0.787
0.700
1.433 0.700
504 lusts
0.111
(16666
1.7 0.008 0 516 n00 100 a. i 0::0
1,041
0.42
0.30m
0,344
0.640
350 0.90 03305
Aog
1,085
0:533/0882
0.407 ja845
0,402
0.767
0.392
[0,701
0.392
[0,000 2.280
00.01 0.304/0624 o 0.
0.283
[0 10.000
0,372
0.000
0.376 0.378 03380
96
00.400
0.471
[1.15
0.110
0.437
0.407
0.31
[0.407
0.431
10.611
0,392
414 0.41003396 0.400
0.405
[o 1.376
0 402 11336 0.435 444
0.551
0.557
1.705
0.613
10.42
0,435
0.481
0.442
0.083
0.483
0.400
0.437
0.432 33200.00
01425
0.636
0 0.9 0.437
0.454 0.550 0255
0.201
1900
0.563
0.4%
(.401
199 a 1s-
0.347
1.12
0.813 2.637
0.591
0.300
0.658
0.710
0.638
0.463
0.472
0 500 08600 0.5te
0.401
0
1.259
0.521 0 0.500
p.440 0.50 0.517
Aug
p.11 0.602
j0516
0,500
0.537
0.51
[0.54]
0.521
[0.422 9.430
(1 40 0.254 jo456
0.52]o..544
0.445
10.328
0.45-|p.458
0.450)..416
04473 L,477
10.344
12533
0.387
0.100
0.964
10.244
0.341
0.387
2077
0.332
0.374 11268 0. (0:00 1966
0,456
0.414
0.374
0.11
0.377
0/412
0.33
0.000
0477
0.
0.340
0.007
0.414
11337
(0.45
0:511
1.470
0.687
0.419
0.41
0.413
0.464
0.363
0.4m
0000004407
0.457 1:415
1677 u 0-129
(0.761
0.64
1554
0.544
p.427
0.465
0.411
0.667
0.435
0.468
0.445
0.477
(n
431
0:446
0u551 0.65
0.227
(1444
[1.412
0-400
0.466
0.440
i 00:00
Awg 00.559 0.529|0.43
01464
0,419
0.378
0.421
[0.38]
0.422
10.355
0.407 6o.59
0.515j0.353
0.400
[0.35
0.434//0414
30.37 0.390 0.443
Exchenge
[0.477
0:447
0:335
0.900
10.30%
0.360
NIA
NIA
[0.008 0.285 lu0e8
(.305
(0 1.448 o 2n8l 107 0.234 in 1u.99 0: 0:217/1119
0.93
192
11.00
0:510
0.54
01642
10606
0.587
0.565
0.531
10525
0.521
NiA N5A
(1.76 (1259 9.17% (.29
0.315 1177
(.271 0.315 o jt 1.5 o 0.000 0.309
336
0.835
0640
176
0.774
0.716
0.647
0.676
NiA
NIA
11,301
1.397 11111
0.4270003111
(0 Booee o 4777 1 100 a 6as jt 3515 0.457 10000 0.534
720
1,00
0.733
0.08
p.90
0,690
0.901
(.900
0.600
N/A
NIA (n 901 0.214 (63399
5.505
1 1199 dod 100 (1064 o 745 3,380 0.714 1447 (9941
Aug
(0.205
0.577/
(1661
0.666
0.514
|0.000
0.566
N/A
[I 0.404 (1:20/00.544
0.414j0
(o 51 01400000416 a 643 0:377 iisuu 0:539
(996
0996
5016
2/16
4/36
3/96
8236
2/36
2016
2/96
|24/
20/20] 5/16 3/%6 22/6
1/66
1.96 1996 | Jont 0:16 (196 4/95
J0:6 0916
15

---

Published as a conference paper at ICLR 2025
B.2 RELATIVE RANKING
Table Relative ranking models and datasets presented in Table Value at row and j-th column represents the number of datasets and horizons on which model outperformed model
Models are sorted from best (top) to worst (bottom).
LCESN-LRIOO
LCESN-LMS
LCESN-LRI
iTransformer
FEDformer
Autoformer
TSMixer
PatchTST
TimesNet
LCESY
DLinear
RLinear
ESN
SUM
TSMixer
28
27
27
23
24
0
26
28
26
26
27
27
26
315
LCESN-LR1
36
34
26
19
0
24
24
19
23
25
26
21
281
LCESN-LRI00
35
33
27
0
8
24
24
19
22
25
26
21
268
iTransformer
34
22
16
17
17
22
27
0
24
26
35
23
265
PatchTST
31
22
14
11
12
0
28
13
24
27
33
27
244
LCESN-LMS
33
30
0
8
8
22
24
19
22
22
25
21
235
FEDformer
27
14
14
13
12
23
11
0
19
34
22
211
TimesNet
29
21
14
11
11
9
25
10
17
0
29
19
196
RLinear
14
15
9
12
12
31
191
DLinear
28
17
12
12
12
6
0
8
13
11
24
22
165
LCESN
29
1
14
19
14
15
148
Autoformer
24
18
11
10
10
3
12
1
7
103
ESN
0
5
8
2
9
60
16

---

Published as a conference paper at ICLR 2025
Table 5: The list of optimized hyperparameters with their genotype-to-pheentyee transformation, initial values in the phenotype space, initial a for CMA-ES in the genotype space, and their allowed range in genotype space. nin denotes the average number of nonzero inputs a neuron. of and oyr denote the coefficients for i-th input and feedback, respectively.
Hyperparameter Phenotype transform
Initial value
Initial o
Boundaries
Tres pres In
50r
2nin
0.01
[-0.1; 1.1]
2x x I'
0
0.05
-1.1;
J
1 x 10-5
0.01
-0.1:2.0
d'bb
I 50%
1 x 10-5
0.01
-0.1:2.0
μbb
I
2x [
0
0.05
1.11
A2
T e 50r
1 x 10-8
0.01
[..1:
C TRAINING DETAILS
The network is driven by the training data U E Txnm. (fed to the network row by row) with the target data Y E R''xnout, where 7' denotes the number of training time steps. Let us define matrix E IR'' whose i-th row is the vector reservoir activations a(i). Afterward, the linear least squares method with ridge regularization is used to find Wout as follows:
W = arg minw ||AW - 21 + A2 I|
where ||
denotes the Euclidean norm and 22 denotes the strength of the regularization.
D EXPERIMENTAL SETTINGS
To minimize the curse of dimensionality, we optimize only restricted set of parameters. The maximum memory length H is fixed at 100 because larger values require more GPU memory, and the model would also require longer washout sequence. Leakage is fixed at 1, because the optimizer had a tendency to avoid leakage in the NARMA task, and the model already has the history of states available through the memory mechanism described in 3.4 Sparsity is fixed at O, because a randomly pruned reservoir does not appear to have a significant impact even for fully connected networks (Matzner 2022). Furthermore, purpose is a regularization that is already covered by the ridge regression. The learning rate for NLMS weight updates (Mims is set arbitrarily 1 x 10-3
topology sparse by design. Onoise is set to zero because its
We adopt the CMA-ES hyperparameter tuning framework by Matzner (022), including genotype-to- phenotype transformations. Generally, the hyperparameters that are supposed to be positive (e.g.. rres) are optimized in the logarithmic space via x + e -50r and hyperparameters that need to take cautious steps around zero (e.g.. (rees) are optimized in square root space via + 2r|x Parameters denoting convex or affine combination coefficients are optimized directly. Table 5 summarizes the optimized hyperparameters, the corresponding transformations, and their upper and lower bounds.
In contrast to Matzner (2022), we do not create a new random network for every fitness evaluation. Instead, we reuse the same random seed, so the optimizer searches for the optimal scaling of single network. This technique reduces noise and makes the optimization more stable.
The initial neuron activation set to zero, and the first 500 steps of training data are used as washout period to eliminate the effect of state initialization. Each task is optimized five times, each time with a different random seed (in a reproducible manner). For NARMA10 110 validation, each optimization run is followed by evaluating the best network on 100 new unique random NARMA10 sequences, and the best result is reported (or visualized). For the real-world datasets, we select the hyperparameters using the train and validation set, and the network with the best validation accuracy over the five optimization runs is used for testing on the provided test set. We use a reservoir of 40x50 with a 7x7 kernel.
The models for all the real-world datasets comply with the same optimization rules. In fact, due to the same random seed, they are the same reservoir, just scaled with different hyperparameters.
17

---

Published as a conference paper at ICLR 2025
10"
state size
113x14 41
10
57x70
10"
10' 8
MSE
10
12
10
10 16
0
5000
10000 train steps
15000
20000
Figure 9: Comparison of various training set sizes of LCESN with 4000 and 16000 neurons and 7x7 kernel on NARMA10. Each point in the scatter plot represents single evaluation of one of the best hyperparameters found during the NARMAIO experiments. The only parameter that is changing is the number of train steps.
By far, the most time-consuming part of the whole training procedure is the hyperparameter optimiza- tion. The hyperparameter tuning requires thousands of network evaluations, and thus, it is desired to minimize its computational time even at the cost of a rougher approximation. Therefore, we use the pure LCESN variant without linear regression recomputation and NLMS weight updates. We optimize our network for a prediction horizon of 192 steps and validate only every 30th subsequence.
For sequence prediction tasks, the most important part of the sequence tends to be just before the sequence to be predicted. To focus on this last part of the sequence, the training sample at time is weighted by e//T where T the number of samples.
In sequence-to- toaeeee-t--eeeee sequuenc experiments (e.g., NARMA10), the feedback connections dramatically improve the results (Matzner) 2022) because the target sequence can be completely different from the input sequence. Without the feedback, the network would not be able utilize its own output. On the other hand, sequence prediction tasks (e.g.. our real-world datasets) already use the network output the input for the next step, using feedback connections would be redundant.
E ABLATION STUDY
E. EFFECT OF TRAINING SIZE
According to the results presented in Section 6 larger networks seem to have a larger learning capacity. An important topic whether the size of the training data sufficient for the largest of the evaluated networks and whether their size also brings downsides to smaller training sets. According to Figure 9 which compares networks of 4,000 and 6,000 neurons on various training set sizes, larger network dominates the smaller network even with smaller training sets. As the training set size increases, the smaller network reaches its limit, and the larger one keeps improving.
At least on the NARMA10 10 task, choosing a larger network does not appear to have negative conse- quences, except for higher computational demands.
E.2 EFFECT OF STATE SIZE
The aspect ratio of the state size does not appear to play significant role in the performance of the network. Figure I0 demonstrates the effect of changing the aspect ratio of the state. Aspect ratio denotes the ratio of height to width, spanning from square shape (aspect ratio 1) prolonged rectangle (aspect ratio 0). The state size 71x7 is significantly worse (p < 0.05) than all other evaluated state sizes. However, the other ratios performed similarly except for an outlier 26x19.
18

---

Published as a conference paper at ICLR 2025
10 8
MSE
22x23
24x21
26x19
29x17
33x15
38x13 3
45x11
56x9
71x7
state size
Figure 10: Comparison various state aspect ratios of LCESN with 500 neurons and 7x7 kernel on NARMA1O. Each violin represents the evaluation of 100 unique random sequences predicted by network selected as the best of five optimization runs.
MSE
10
10
convolution
Icesn
permutation
ring
topology
Figure 11: Comparison of various topologies of 4000 neurons on NARMA1O. LCESN uses a 7x7 kernel. Each violin represents the evaluation of 100 unique random sequences predicted by network selected as the best of five optimization runs.
According to the data, there a chance that performance may be degraded extreme cases where the kernel covers the entire height or width of the network. To mitigate the risk, state sizes closer to the 1:1 ratio.
E.3 ALTERNATIVE TOPOLOGIES
Figure compares multiple reservoir topologies. The ring topology was presented in Rodan and Tino 12011) minimum complexity ESN reservoir that does not affect performance. It simply connects the neurons to form a circle. The permutation topology (as evaluated, e.g., by Gallicchio and Micheli (2019) and Matzner (2022) splits the neurons into multiple rings, each independent of the others. The convolution topology uses the same kernel technique as LCESN, but the kernels for all neurons equal. In our experiments, LCESN significantly outperformed the three remaining topologies (p < 0.05).
E.4 RELATION BETWEEN MEMORY HORIZON KERNEL, SIZE
As discussed in Section 3.3 local connectivity does not inherently improve ESN accuracy when the network size is constant and shows no such effect on NARMA10 tasks (Figure55.. Its primary purpose to reduce computational costs and enable larger networks. contrast, forced memory is designed to improve the accuracy for tasks requiring memory, independent of local connectivity. To evaluate this empirically, Figure 12 presents heatmap of various kernel sizes and forced memory horizons the ETTml benchmark. The results reveal clear effect of longer memory horizons, while showing minimal impact from kernel size variations. The findings suggest no clear dependency between local topology and forced memory.
19

---

Published as a conference paper at ICLR 2025
3x3
0.60
5/5
kernel size
0.55
7x7
0.50
11x11
50
100
Figure 12: A heatmap of forced memory horizon (H) vs. multiple kernel sizes for 40x50 LCESN network on ETTml dataset. The color represents the average MSE validation error over five independent optimization runs.
60
method
duration [μs)]
sparse (C5R)
20
0
20x25
28x36
40x50
57x70
80x100 113x141
reservoir size
Figure 13: Average execution times of two different LCESN step implementations with a 7x7 kernel, based on 10 000 evaluations. Sparse matrix multiplication performed using the Arrayfire CUDA library (Yalamanchili et al. (2022), utilizing the compressed sparse row (CSR) format. Experiments were conducted on an NVIDIA GTX 1080Ti GPU and an Intel 17-4770 CPU.
F DETAILS ON GPU IMPLEMENTATION
The topology of LCESN was designed with GPU memory properties in mind. LCESNs leverage that sparse reservoirs do not appear to harm accuracy (Matzner) (022) while also providing op- portunities for more efficient computation. Generic sparse matrix algorithms, however, inherently suffer from significant overhead due to irregular memory access, which negatively impacts GPU performance (Bell and Garland 2009). To overcome these issues, we optimized our implementation to fully leverage the proposed local topology, minimizing memory overhead and ensuring coalesced GPU memory access (i.c., access in continuous blocks). The local connectivity is well-suited for GPU parallelization, allowing each neuron's state update to be handled independently by GPU threads, while still effectively utilizing shared GPU memory.
F.1 PERFORMANCE
Our GPU implementation, tested on NVIDIA GTX 1080 demonstrated 14x speedup compared to conventional fully connected ESN and 15x speedup compared CPU implementation (Intel 17-4770), using reservoir of 80x 100 neurons and 7x7 kernel.
The closest alternative to our custom implementation is sparse matrix multiplication. Figure 13 compares the execution times the LCESN step implemented with our method versus the sparse matrix method. For very small reservoirs, the performance difference is minimal. However, for larger reservoirs of size 80x100 our implementation demonstrates an improvement of over 50%.
F.2 ALGORITHM OVERVIEW
The reservoir weights stored in tensor of dimensions N M K x K, where N M represents the size of the reservoir, and K x K represents the kernel size. The element at index [i, kt corresponds the connection weight from the neuron position [i K//2 ////////jjj/////
20

---

Published as a conference paper at ICLR 2025
to the neuron at position [ of the reservoir. In other words, each kernel slice :, stores the connections directed inward to the newron at position [i. The reservoir activations are stored as a tensor of dimensions N x M. These tensors stored in memory in the conventional column-wist order.
The order of dimensions is crucial aspect of the implementation. Subsequent indices in the first dimension are contiguous memory, while subsequent indices in other dimensions represent more distant memory addresses. Specifically for the reservoir weights matrix, each kernel is intentionally scattered across the memory.
The algorithm roughly consists of the following steps:
1. Allocate a perimeter of dimensions (N + K) (M K). which is the reservoir grid padded by state activations from the opposite side of the reservoir. This ensures that every neuron can access its entire kernel neighborhood locally, simulating a periodic (torus-shaped) reservoir. Allocate this perimeter shared GPU memory.
2. Use the GPU thread index to access the reservoir grid in top-down, left-right manner. In other words, allocate the GPU warp subsequent neurons in the memory.
3. For each thread, traverse the weights of its kernel (top-down, left-right), multiply them by the corresponding presynaptic activation found in the perimeter, and accumulate the result in a local variable.
4. Write the accumulated result to the corresponding output in global memory.
By adhering to the specified dimension order and traversal sequence, GPU memory accesses coalesced, ensuring optimal performance. For a more detailed description, please refer to the source code file 1 ccn__sss stt_stee__coo _e____uda.. cu.
G DETAILED BENCHMARKS
One of the main benefits of ESNs is their fast training procedure and inference. However, LCESN, as a recurrent model, differs from feedforward models in its usage (see Section (355. To clarify the most suitable tasks for ESNs, we present a detailed analysis of the benchmarking results from Section 6.2 and compare LCESN with alternative approaches.
The majority of time is spent on hyperparameter optimization using the CMA-ES procedure described in Section 3.2. We conducted 2,000 full evaluation cycles, which, in the case of the ETThl dataset, took less than four hours. The model selected for testing was the one with the best validation error across five independent hyperparameter runs, resulting in a total optimization time of 20 hours. While this approach provides a robust lower bound on achievable performance, is also the weakest aspect of our method, offering significant opportunities for improvement. For example, early stopping or more efficient optimization algorithms could reduce this time. We adhered to the framework by Matzner (2022). leaving optimization improvements for future research.
is rather difficult to compare running time of our model to others, because of hardware differences and often complicated reproducibility. Using information from the TSMixer paper (Chen al.) 2023), it is reported that the hyperparameter search grid required 1,600 training and validation cycles for each dataset (400 for each of four time horizons), which is similar to model. Additionally, is mentioned that a single training cycle TSMixer took more than three hours on a high-end NVIDIA Tesla V100 GPU on large M5 dataset. It suggests that the hyperparameter tuning was very time-consuming and may have required a cluster of GPUs. Although Transformers may appear agnostic to hyperparameter perturbations. [Lu et al. (2024) (Appendix C, Figure 9) noted considerable sensitivity in the tested TSF datasets. Unfortunately. the work does not detail their hyperparameter optimization procedure or its computational costs, making direct comparison challenging. To focus on LCESN's 's practical performance, we assume known hyperparameters in the subsequent experiments.
G.1 TRAINING AND INFERENCE TIME
Liu et al. (2024) (iTransformer) provided a quality code, and we were able to reproduce their results on the same hardware (NVIDIA GTX 1080 Ti GPU supported by single core of Intel i7-4770 CPU).
21

---

Published as a conference paper at ICLR 2025
washout
training
testing
LCESN
iTransformer
19.25
28.6s
47.8s
Figure 14: Decomposition of washout, training, and testing times on ETTh1 datasets for LCESN and iTransformer for prediction horizon of 96 steps.
Table 6: Comparison of inference speed in low-latency (1-step) and high-through throughutt (96-steps) scenarios for LCESN and iTransformer. the 1-step scenario, only single prediction required as soon as possible. In the 96-steps scenario, LCESN has to be run subsequently 96 times in self-feeding loop, while Transffrmer predicts the entire horizon immediately. Both scenarios used the same LCESN model, while Transformer trained for each of the scenarios separately.
horizon
LCESN iTransformer
1-step 8361 steps/s
107 steps/s
96-steps
87 steps/s
97 steps/s
Since iTTrasffrmer was the second strongest competitor, will use it as baseline measurement. Figure 14 illustrates the decomposition of training and testing times for LCESN and iTransformer on the ETTh1 datasets with prediction horizon of 96 steps. For LCESN, the linear regression training procedure required just over two seconds, highlighting one of the strongest upsides of ESNs. The testing procedure involved running 96-step self-feeding loop for each testing data point, which is inherently more time-consuming than training since the training phase traverses the data only once.
G.2 LATENCY VS. THROUGHPUT
Figure dodes not show a complete picture. Feedforward models like iTranformer predict the entire prediction horizon at once and support parallel batch processing, while LCESN only predicts the next step. These differences offer distinet advantages. fast 1-step prediction excels low-latency scenarios, where only the immediate next prediction matters and the others wait (e.g., control systems, trading, online signal filtering). In contrast, predicting the entire horizon and supporting parallel batches provides higher throughput (e.g., processing offline datasets predicting many independent time series in parallel). Additionally, 1-step prediction model like LCESN can handle arbitrary horizons without retraining. Feedforward models like iTransformer, which predict fixed horizons, are traditionally retrained for each desired horizon separately. Table shooss a comparison of latency and throughput between LCESN and iTransformer.
22


Reviews:
[{"review_id": "uf9f3iSMgs", "reviewer": "Reviewer_p21G", "summary": "In this work the authors propose an Echo State Network for time series prediction whose most distinctive properties lie in a local (small-world) connectivity pattern of its neurons as well as an approach termed memory forcing where neuron activations of various previous timesteps are linearly combined with the current activations. In combination with rigorous hyperparameter optimization by means of an evolution strategy, the proposed models manage to outperform classical ESNs at a selection of timeseries prediction tasks as well as achieve on-par performance to transformer-based models on a wide variety of prediction benchmarks.", "questions": "Ln 129:  I am not familiar with describing a uniform distribution through  parameters /mu and /sigma (notation and methodology). They seem to loosely correspond to input scaling and bias scaling as used in reservoir literature? \n\nLn 143: Are hyperparameters for baseline ESNs optimized with CMA-ES as well?\n\nIn order to gauge the effectiveness of the proposed improvements to real-world tasks, additionally reporting the performance of an ESN baseline, for which the hyperparameters have been optimized using the same process (evolution strategy) and compute budget applied to find good LCESNs for all (or at the very least: several) tasks would be nice to have.. \n\nOn metrics: As the MSE of NARMA-10 is typically dependent on the scaling of the input signal, in literature it is typically normalized by the variance of the target signal and reported as the Normalized MSE (NMSE) [1]. While the very low MSE values reported in the manuscript suggest sufficient performance, they are hard to interpret without context. Considering the limits of floating point precision, I am unsure on whether comparing values between 1e-8 and 1e-13 yields applicable insight (Figure 5). I would recommend reporting the NMSE instead and eventually additionally plot signal traces to demonstrate how well the proposed approach approximates the target signal.\n\n\n[1] Jaeger, H. (2002). Adaptive nonlinear system identification with echo state networks. Advances in neural information processing systems, 15.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Overall, the work is executed well. All ideas and approaches are explained clearly, the manuscript is well-written and the visual quality of its figures is sufficient. \n\nThe strongest point of the paper is it's extensive experimental evaluation with a wide range of state-of-the-art methods as well as the proposed method showing a viable alternative to the transformer architecture. This is especially important as previous work [1] suggests that transformers are less than optimal for timeseries prediction due to the intrinsic permutation equivariance of the attention module. Therefore, in principle, I consider this work to be of interest to the ICLR audience.\n\n[1] https://github.com/valeman/Transformers_And_LLM_Are_What_You_Dont_Need", "weaknesses": "A comparison to transformer models with respect to runtime for a typical amount of hyperparameter-configurations is missing (see below for elaboration). Furthermore, in the reservoir computing literature a number of approaches closely related to the ones proposed in the manuscript have already been explored in the past (see below for references).\n\n## Experimental\nWhile the final least squares training of an ESN tends to be very fast, the authors conduct an extensive hyperparameter optimization. Due to the relatively large amount of hyperparameters optimized, as well as their strong influence on the dynamics of the echo state network, especially w.r.t. memory capacity and nonlinearity, I consider this optimization process as a form of training in itself, which should be accounted for when comparing model's training times. \nTo actually reliably assess whether the proposed ESNs constitute a faster/easier-to-train alternative to a small transformer model, where in general fewer hyperparameters need tuning, the manuscript should contain more detailed  timing information w.r.t. the time required to find an ESN suitable for a given task, and compare these timing with the time needed to find good hyperparameters to achieve comparable performance on a state-of-the-art transformer model on that task. \n\nIn my experience, transformers tend to perform rather well for most tasks with somewhat universal and task-agnostic default hyperparameters (i.e. Adam optimizer, learning rate 1e-3, dropout 0.2 etc.). Echo State Networks on the other hand, due to the fixed nature of their hidden layer, typically require much more rigorous hyperparameter tuning than current gradient-based-models (see e.g.  [1])\n\nA comparison between the execution time of the finally trained ESN and a similarly performing transformer model would be important as well to precisely quantize the dimension of the gained speedup. \n\n## Background and related work\nFor the proposed approaches and methodologies, closely related work exists in the reservoir computing literature:\n\n- In terms of drawbacks of ESNs as introduced by the authors, I strongly disagree with the third point (Ln 53): the output layers of reservoirs have been trained online using LMS, RLS and other gradient-based approaches since the early 2000s, most-prominently as Backpropagation-Decorrelation [2] but also in other forms [3,4,5].\n\n- Locally connected reservoirs and reservoir computing with small-world networks have been investigated in the 2010s especially in the reservoir-computing-as-neuromorphic-hardware-community, where fully connected networks imply high wiring effort, e.g. see [6,7]\n\n- I consider the memory-forcing-approach introduced by the authors a more explicit formulation of a mixed internode-delay reservoir computing approach which has previously been explored in the literature, most recently as Distance-Delay-Networks [8], where the parameters of a distribution determining the length of variable inter-node-delays in ESNs using CMA-ES. While this approach takes a more biological view than the memory-forcing approach in this manuscript, with the exception of an optimized \"memory-leak-factor\" (w_{\\text{mem}} in the manuscript), the actual implementations appear almost identical to me.\n  \n- In general, while the authors provide clear intuitions on the underlying issues, the discussion of background work on reservoir computing and dynamical systems falls somewhat short in the manuscript. For well-researched issues such as the proverbial \"edge of chaos\" or memory in reservoir computing systems, only very few references are given.\n\n\n## References\n[1] Viehweg, J., Worthmann, K., & Mäder, P. (2023). Parameterizing echo state networks for multi-step time series prediction. Neurocomputing, 522, 214-228.\n\n[2] Steil, J. J. (2004, July). Backpropagation-decorrelation: online recurrent learning with O (N) complexity. In 2004 IEEE international joint conference on neural networks (IEEE Cat. No. 04CH37541) (Vol. 2, pp. 843-848). IEEE.\n\n[3] Jones, B., Stekel, D., Rowe, J., & Fernando, C. (2007, April). Is there a liquid state machine in the bacterium escherichia coli?. In 2007 IEEE Symposium on Artificial Life (pp. 187-191).\n\n[4] Antonik, P., Duport, F., Hermans, M., Smerieri, A., Haelterman, M., & Massar, S. (2016). Online training of an opto-electronic reservoir computer applied to real-time channel equalization. IEEE Transactions on Neural Networks and Learning Systems, 28(11), 2686-2698.\n\n[5] Fiers, M. et al. Nanophotonic reservoir computing with photonic crystal cavities to generate periodic patterns. IEEE Trans. Neural\nNetw. Learn. Syst. 25, 344–355 (2014).\n\n[6] Kawai, Y., Tokuno, T., Park, J., & Asada, M. (2017, September). Echo in a small-world reservoir: Time-series prediction using an economical recurrent neural network. In 2017 Joint IEEE International Conference on Development and Learning and Epigenetic Robotics (ICDL-EpiRob) (pp. 126-131). IEEE.\n\n[7] Daniels, R. K., Mallinson, J. B., Heywood, Z. E., Bones, P. J., Arnold, M. D., & Brown, S. A. (2022). Reservoir computing with 3D nanowire networks. Neural Networks, 154, 122-130.\n\n[8] Iacob, S., & Dambre, J. (2024). Exploiting Signal Propagation Delays to Match Task Memory Requirements in Reservoir Computing. Biomimetics, 9(6), 355.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p0yGali9px", "reviewer": "Reviewer_XXJ8", "summary": "This paper builds on Echo State Networks by providing specific structural constraints through local connections. Further, specific memory context is encoded through skip/residual temporal connections. The updated architecture is presented as a novel framework titled Locally Connected ESN (LCESN) and it's performance is benchmarked against nine real-world datasets.", "questions": "1. Can the authors expand on why their method reduces the time and space complexity? \n2. The authors motivate forced memory for the network to be stable. Can the authors provide additional discussions/experiments relating this to the \"edge of chaos\"? i.e is there a temporal look-back connection at which the network transitions from being chaotic to stable, with highest performance? Additionally, can the authors provide clarification on why a max delay of 100 was set?\n3. In section 3.5 the authors present multiple ways to update W_out. Are these all novel methods? Is there a reason W_out is retrained every 100 steps? how does this change with larger/fewer steps?\n4. Fig 5,6,7: Confusion on results: Can the authors provide some clarification/insight on why locally connected ESNs without forced memory perform better that ESNs? From Fig 5 it appears larger networks are better, however kernel size doesn't significantly affect performance. Yet, table 1 and fig 7 show ESNs perform worse than LCESN. Can the authors provide theoretical/empirical experiments discussing at what kernel size LCESNs behave like ESNs?\n5. Is there a relation to the kernel size and forced memory with the data being presented? I.e how do LCESNs perform on datasets with longer term time dependencies compared to ESNs and other state of the art methods?\n6. Can the authors provide insight on how this method compares to methods like FORCE/full-FORCE where structure is provided in W_feedback instead of within the reservoir itself?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper introduces a unique architecture and framework to ESNs via the introduction of locally connections. Second, by providing a fixed context length the authors introduce memory forcing which is motivated to prevent the network from being unstable. Third, the authors present multiple weight adaptation strategies. Taken together the authors present a novel architecture of ESNs and a gradient-free training procedure for them. Lastly, they test their framework to be computationally and memory efficient and evaluate their results on real-world datasets.", "weaknesses": "1. While the authors motivate the problem, little to no information/references are provided on some critical components. For instance in the introduction section the authors discuss main advantages of ESNs but do not provide any references. Critically, the paper misses references to additional reservoir computing frameworks like FORCE (Sussillo, David 2009)/FULL-FORCE (DePasquale 2018).\n2. The paper is motivated by providing better time and space complexity on the ESN step, however this step is never formalized or discussed in the paper. \n3. They restrict their models to have a weight adaptation after every 100 steps, however it's unclear if more/less steps could lead to better/worse performance.\n4. The authors “proposed topology is designed with GPUs in mind”: however there is no link/discussion describing why the torus structure scales well with GPU architecture.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PJbIhYeX2Y", "reviewer": "Reviewer_ENar", "summary": "This work considers time series forecasting problem and introuces an ESN-based architecture to tackle this problem. The authors are motivated by ESNs' strengths but note that previous works show that their performance may be lacking compared to alterantive appraoches. The authors note three potential reasons for their lack of performance: constraint on reservoir size, ESN's memory-stability tradeoff and a lack of adaptive training of the readout as the new data arrives. \n\nThe authors address all of these points by intruducing local connectivity in the reservoir, memory forcing, and evaluate their approaches under a multitude adaptive strategies of the readout weights updates. Author's approach appears to outperform vanilla ESNs and are competitive with SOTA approaches.", "questions": "- Conceptually, is forced memory in your approach equivalent to stacking up the hidden state vectors from a pre-defined range in history under a vanilla ESN setting?\n- Could a relative ranking matrix be added across the methods? For example, an N x N matrix showing how many times model *i* outperformed model *j* across datasets, to summarize the results in Table 2.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Overall the presentation of the work is clear.\n- Three key components for enhancing ESNs' performance appear sound; their advantange is illustrated well. I think the ESN community will appreciate such results. Comparisons against vanilla ESN in terms of performance and wall-clock time are appreciated.\n- The methods are compared on a difficult benchmark.", "weaknesses": "1. The final performance of LCESN doesn’t fully reach SOTA levels, despite this being a main goal of the paper. While results are comparable to SOTA, the primary advantage of ESNs—fast training and inference times—is not fully emphasized, aside from a brief mention. Without a comparison of these speed benefits against SOTA methods (including hyperparameter search and inference times), the broader appeal of using ESNs, especially over vanilla ESNs, may be limited.\n2. The comparisons in Table 2 have some notable omissions:\n\t- There is no comparison against a vanilla ESN.\n\t- TSMixer, the best-performing model, isn’t evaluated on some datasets. Claims about model ranking may be misleading, as TSMixer could outperform the authors' LCESN model. While the original paper did not evaluate TSMixer on the two excluded datasets, either these datasets should be excluded, or TSMixer should be evaluated on them.\n\nI'm open to increasing the score by at least 1 point if these weaknesses are addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1bUNIZXCT7", "reviewer": "Reviewer_T1ip", "summary": "Interesting work and a solid paper.\n\nIt presents an ESN architecture with a cleverly designed local connectivity, whose primary goal is to overcome the computational limitations of traditional ESNs.\n\nThe interpretation of the results could be expanded. In particular, it is unclear why and how the forced memory positively affects the outcomes.", "questions": "- You should discuss why local clustering improves performance compared to classic ESNs. In particular, the performance of your algorithm seems to increase with kernel size (as highlighted by Figure 5). Doesn’t this suggest that a standard ESN, which is equivalent to having a kernel size equal to the size of the ESN, should perform better?\n\n- I have already emphasized the lack of justification for the forced memory mechanism. You briefly mentioned the leaky neuron; you should evaluate the performance of the leaky neuron compared to the forced memory mechanism.\n\n- You argue that the primary advantage of your algorithm is the faster matrix multiplication, which in turn allows for larger networks. However, ESN connectivity can be very sparse, meaning the methods used to compute matrix products will differ from what you propose in the paper. You should compare the performance of your algorithm with sparse matrix multiplication.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The algorithm’s motivations are clear, and they are fulfilled by the current algorithm.\n\nAs a bonus, it performs better on certain tasks compared to the other algorithms investigated.\n\nThe paper’s presentation is mostly clear, and the results are correctly interpreted.", "weaknesses": "The analysis of the model is really small. You proved that your ESN connectivity performs better than others ESN on a variety of benchmark, but advanced no reason as why it would be the case. See question 1.\n\nSimilarly,  It's still unclear to me what are the real advantage of the forced memory and there is really few motivations for its design, that would have been great to see further explanation of that.\n\nYou should the following minor changes anyway :\n* You should change the legend of figure 5 so that it matches the order of your results and be ordered by size. \n* LCESN-LMS is never clearly defined but is understood from context.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)