Distance-Based Delay Networks∗
Stefan Iacob1 , Joni Dambre1
1. IDlab-AIRO, Ghent University-Imec

Summary. Biological neurons have finite propagation speed and non-zero distances between eachother, which results in variable, distance-based delays. We implemented this phenomenon in echo
state networks, by modelling reservoir units as points in a physical space. We optimized the distribution of neuron locations using an evolutionary algorithm. The resulting networks outperform regularESNs on the NARMA-10 task.

Signal delay inherently exists in all physical and biological systems. Delay is often perceived as aphysical limitation that needs to be overcome. However, inter-neuron delay is an essential computationalfeature of the brain. Biological neurons have a physical position in space, and neuron axons have a nonzero length and a finite propagation speed. The alignment of delays can for example be used to synchronize
incoming signals and has an effect on plasticity [1]. In contrast, until recently implementations of artificial
neural networks (ANNs) consisted of abstract computational units with an inter-neuron communication
speed of exactly one simulation step, thus lacking distributed, location-based inter-neuron delay. In
recent work, we have introduced distance-based delays in recurrent neural networks [2]. We take echo
state networks (ESNs) as a starting point for our implementation. ESNs [3] are a type of rate-based
recurrent neural network with randomly sampled and fixed weights. Instead of training these “reservoir”
weights, a read-out layer is trained based on a teacher signal, usually using linear regression. Whileoriginally introduced as an alternative way to train RNNs, nowadays ESNs are mainly used as modelsystems for physical reservoirs. The ESN reservoir activity is described byx(n + 1) = (1 − a)x(n) + a · f (Wres x(n) + bres + Win v(n))(1)

where x(n), a, Wres , Win , b, f , and v(n) are respectively, the firing frequencies at timestep n, the decay
parameter, the recurrent reservoir weights, the input weights, the bias weights, the activation function,and the input at timestep n.
We implement distance-based delay networks (DDNs) by modelling reservoir nodes as points in a 2Dor 3D space. The discretized distances between nodes determine the amount of simulation steps of delaythat is applied. As such, an increase in firing rate from neuron A is only perceived at the input for neuron
B after d steps, with d being the length of the connection between neuron A and B. This is illustrated inFigure 1.

Figure 1: Diagram describing the flow of activity between DDN reservoir neurons A and B, with weightwAB and a connection length of d. xA (n) and xB (n) refer to the firing frequency of neuron A and B attime n.
The neuron activity update rule for DDNs is described by the following equation, which is an extensionof the conventional ESN equation.
x(n) = (1 − a)x(n − 1) + aσ(y(n − 1))y(n) =DmaxXresin
WD=dx(n − d) + WD=dv(n − d) + bres

(2)
(3)

d=0
∗ This project has received funding from the European Union’s Horizon 2020 research and innovation programme under
the Marie Sklodowska-Curie grant agreement No 860949.

x(n), a, σ(·), v(n), Dmax , and bres are respectively, the firing frequencies of all neurons at time n, thedecay parameter, the sigmoid activation function, the task input at time n, the delay applied by thelongest connection, and the bias weights. WD=d is a partial weight matrix, or a masked weight matrix,
where all elements corresponding to connections with a delay different from d are set to 0.
In general, any reservoir can be seen as a sample from a distribution defined by a set of hyperparameters. For conventional ESNs, these hyperparameters are the connectivity of the network, the scaling of thereservoir weights, input weights and bias weights, and the leakage parameter. Additionally, in DDNs thedelays in the reservoir need to be optimized to match the task at hand. Hence, neuron locations (on which
the delays are dependent) need to be optimized. In order to reduce the total amount of hyperparameters,
we do not optimize each neuron’s coordinate. Instead, we sample neuron locations from a multimodal
Gaussian distribution (Gaussian Mixture Model, or GMM), with a fixed number of components. We treat
the means and covariance matrices of this distribution as hyperparameters, to be optimized along with theconventional ESN hyperparameters. However, because we know from which component the neurons aredrawn, the reservoir is naturally clustered. Thus, we can define the conventional ESN hyperparameters on
a per-cluster basis. For example, instead of one global scaling parameter for all reservoir weights, we now
have a scaling parameter for each cluster. Instead of having a single connectivity paramater, we have aconnectivity matrix containing connectivity parameters for each cluster pair, and a diagonal that specifies
the within-cluster connectivity. Although our baseline ESNs do not make use of neuron location, we cannonetheless cluster ESN neurons such that we can define hyperparameters for each cluster (pair). This isto ensure a fair comparison between the baseline ESNs and DDNs. We used CMA-ES, an evolutionary
strategy, to optimize the hyperparameters of both network types. At each generation of the evolution
run, candidate hyperparameters are generated, from which several DDN/ESN reservoirs are sampled. theread-outs of these reservoirs are trained using a NARMA-10 training set. We use the reservoir validation
performance a fitness measure. An evolution run was performed for both the baseline ESN and DDNmodels, using single-cluster models and four-cluster models. The performance on a NARMA-10 test setof the networks drawn from the best hyperparameter sets can be seen in Table 1.
Most importantly, we showed that DDNs significantly and consistently outperformed conventionalESNs. Moreover, the use of multiple clusters further improved both baseline models and DDN models.

Table 1: Average normalized root mean squared error (NRMSE) on the test set of 40 networks sampledfrom the best hyperparameter sets found during the evolution. Bold font indicates the best performing
model type.
TypeK NRMSE (test)
Baseline 10.1588 ± 0.0124
DDN10.0639 ± 0.0018
Baseline 40.0848 ± 0.0056
DDN4 0.0391 ± 0.0025
Our findings have important implications for the field of physical reservoirs, suggesting it should bepossible to match inherently present delays to specific tasks.
Furthermore, in biological networks, delays influence synaptic plasticity. In subsequent work, we
explored the effects of delays on local plasticity in DDNs by introducing a delay-sensitive local learning
rule. This proved to be beneficial for learning capacity, while outperforming its delay-less counterpart.

References[1]

H. A. Swadlow and S. G. Waxman, “Axonal conduction delays,” Scholarpedia, vol. 7, no. 6, p. 1451,
2012, revision #125736. doi: 10.4249/scholarpedia.1451.

[2]

S. Iacob, M. Freiberger, and J. Dambre, “Distance-based delays in echo state networks,” in Intelligent
Data Engineering and Automated Learning – IDEAL 2022, H. Yin, D. Camacho, and P. Tino, Eds.,
Cham: Springer International Publishing, 2022, pp. 211–222, isbn: 978-3-031-21753-1.

[3]

H. Jaeger, “The “echo state” approach to analysing and training recurrent neural networks-with
an erratum note,” Bonn, Germany: German National Research Center for Information Technology
GMD Technical Report, vol. 148, no. 34, p. 13, 2001.