New insights on homeostatic activity-dependent structural
plasticity in rate based neural networks
Tanguy Cazalets1 , Joni Dambre1
1. Ugent University, Gent, Belgium

Summary. We introduce a novel rate-based homeostatic activity-dependent structural plasticity(HADSP) algorithm for echo state networks (ESNs). The algorithm employs solely homeostatic plasticity, yet enables the emergence of principles of Hebbian learning. Our analysis suggest that HADSP
is able to generate networks that effectively recombine redundant inputs to improve performancesand highlight the role of structural plasticity in influencing network function and organization.

Structural plasticity is a fundamental mechanism in the formation and maturation of biological neuralcircuits during development and has been the subject of significant scientific research [2] [1]. Although
fundamental in biology, structural plasticity is totally absent in the context of ESN where connections
between neurons are generated randomly [3] and remain fixed. Our algorithm aims to tackle this gap
between ESN and biology.
Echo state networks. The ESN framework is a system of n neurons with vector states x determinedby a combination of a connection matrix W, input matrix Win , input u[t], bias vector b, and activation
function σ. The evolution of our system is described by : x[t + 1] = σ(W × x[t] + Win × u[t]) + b, whereσ is the hyperbolic tangent function, b is a 1 × n vector sampled from a normal distribution with mean
0.1 and standard deviation 0.1, and the dimensions of W and Win are n × n and 1 × n, respectively. The
desired output y[t] is obtained from the learned matrices Wout and bout , which are obtained by ridge
regression to approximate a target output y target [t], as given by : y[t + 1] = Wout × x[t + 1] + bout . [3]
HADSP rule. Every ∆t steps, a growth indicator, ∆z, is calculated for each neuron according tothe following formula :1(1)∆z = − (⟨⟨x⟩⟩ − ρ)βwhere ⟨ x⟩⟩ is the average of the state vector x over the increment period. For the j-th neuron if ∆zj < −1one input connection is added (wij = wij + 0.05) or pruned (wij = wij − 0.05) if ∆zj > +1. Selfconnections are not allowed in our model and the number of connections k is constrained by k ≤ γ with γ
the maximum number of partners. We utilize only excitatory neurons to ensure that adding or removing
inputs will have a positive or negative impact on the rate.
∆z depends on two hyperparameters : the growth parameter β and the target rate ρ that are choosen
in order to obtain a given spectral radius while allowing the algorithm to converge. We indeed foundthat if the algorithm fails to converge, the network undergoes constant reorganization, leading to achanging dynamic over time and unstable representation of the inputs, thus making its training withRidge regression inefficient. Similarly the value 0.05 of weight increment is kept low enough to allow theHADSP to recombine the inputs without creating unstable regimes.

Figure 1: HADSP implements a form of Hebbian learning. Starting with a random connection matrix,
the neurons create connections between them through a homeostasis mechanism. When they receive
larger inputs, neurons 51 to 60 prune their weights to reach homeostasis. Once the larger input isremoved, the other neurons are already at homeostasis, hence neurons 51 to 60 create connections mostlybetween themselves. Thus neurons developed intricate connections based on their previous correlated
over-stimulation, demonstrating a form of Hebbian learning in our purely homeostatic algorithm.

Results. We postulate that HADSP algorithm can enhance the dynamics of a reservoir computing
network by combining redundant inputs into more useful ones. To validate this idea we use the popularu(t−τ )Mackey-Glass (MG) benchmark: dudt = β 1+u10 (t−τ ) − γu(t) where u(t) is the state of the system at time t,β and γ are positive constants, and τ = 17, 30 to obtain mild and strong chaotic behavior. Performanceare evaluated on T-steps ahead prediction using the Normalised Root Mean Square Error (NRMSE).

(a) Schematic overview of the ESN in the first ex- (b) Schematic overview of the ESN in the second experiment
periment

ModelMG17MG30T
130
100
11030

HADSP
0.021
0.57
0.73
0.00600.53
0.41random
0.00750.52
0.67
0.00430.50
0.40

ModelMG17MG30

T13010011030

HADSP+band
0.0022
0.096
0.44
0.035
0.15
0.24

random + band
0.0040
0.10
0.47
0.038
0.170.24(c) NRMSE averaged over 10 trials, for MG-17 and MG- (d) NRMSE averaged over 10 trials, for MG-17 and MG30 for first experiment
30 for second experiment

Figure 2: Comparison of ESN and results for our two experiments
We aim to prove that the HADSP algorithm can exploit input correlation to improve performance inESNs therefore we didn’t systematically optimise the hyperparameters of our ESNs.
First experiment : The unsupervised learning process of the HADSP algorithm with ρ = 0.8 and β = 0.1
was used to evolve a network using a MG time series as input with a correlation captured every ∆t = 20steps. The output weight matrix Wout was calculated in a supervised manner to reproduce the signal
T steps ahead. This resulting ESN is then compared against a network instantiated from a uniform
distribution between 0 and 1 with same size, connectivity and spectral radius. The results in 2(c) showed
no improvement in performance for the MG time series.
Second experiment : To address this, the input was transformed into 130 separate inputs using Butterworth band filters on a selected set of frequencies. A connection matrix was generated through theHADSP mechanism of size n = 260 with ρ = 0.8, β = 0.1, γ = 12 and ∆t = 20. The output weightmatrix Wout was calculated using the generated ESN and the 130 filtered inputs to reproduce the originalMG signal τ steps ahead. The HADSP algorithm was again compared to a second matrix randomly generated with the same size, connectivity and spectral radius. The results in 2(d) showed that the HADSP
algorithm outperformed the randomly instantiated network
Conclusions. Our study highlights the effectiveness of HADSP in synthesizing inputs that have
similar dynamics, which in turn leads to more useful dynamics in the reservoir. The interplay between
Hebbian learning and homeostasis, as showcased by HADSP, provides new insights into the generationof non-random ESN. Overall, our results emphasize the importance of considering input similarity in thedesign and optimization of reservoir computing systems.

References[1] M. Butz, F. Wörgötter, and A. van Ooyen. Activity-dependent structural plasticity. Brain Research
Reviews, 60(2):287–305, may 2009.
[2] J. Cang and D. A. Feldheim. Developmental mechanisms of topographic map formation and alignment.
Annual Review of Neuroscience, 36(1):51–77, jul 2013.
[3] M. Lukoševičius. A practical guide to applying echo state networks. In Lecture Notes in Computer
Science, pages 659–686. Springer Berlin Heidelberg, 2012.