Package: auto-esn
This is the repository connected to the paper"Grouped Multi-Layer Echo State Networks with Self-Normalizing Activations"(pdf availablehere). Below, we present additional information and experiments that were performed.UsageArchitecture overviewDeep ESNGrouped ESNGrouped Deep ESNGrid Search ConfigurationBest Architecture ConfigurationESNLSTMMemory CapacityInput scaling vs Activation RadiusUsageMake sure you have torch installed fromhttps://pytorch.org/get-started/locally/Runpipinstallauto-esnThen navigate tosimple example, read descriptions, copy code to your project and run!
For more complex examples exploreexamplesfolder.
More "formal" documentation is undergo. In the meantime, if you have any questions or requests don't hesitate to raise anissue.Architecture overviewFour different types of Echo State Networks were tested: shallow ESN, deep ESN(dESN), grouped ESN(gESN) and the generalisation of all of them which is grouped deep Echo State Network (gdESN).Deep ESNBriefly, Deep Echo State Network stacks several reservoirs one on top of another. The difference to classical deep neural network is that the output of all intermediate layers is concatenated giving the final result.Grouped ESNGrouped ESN consist of a group of shallow ESNs whose outputs are concatenated to create final output.Grouped Deep ESNGrouped Deep ESN puts both these approaches together by creating a group of Deep ESNs. The output ofgdESNis concatenated output of all its reservoirs.Grid Search ConfigurationTanh activation function highly  depends  on  input  scaling,  while  for  SNA(self-normalizing activation) its  effect  is  reduced  due  to  normalization  factor. This is why we used two different hyperparameter setups as an input to grid search. For each configuration, 5 concrete models were generated with different random seeds applied for weight initialization. ForgESNanddESNarchitectures Ng,Nl∈ {2,3,4,5,10}, where Ngis number of groups and Nlnumber of layers. ForgdESNarchitectures each of{(2,2),(2,3),(2,4),(2,5),(3,2),(3,3),(3,4),(4,2),(4,3),(5,2)}configurations of (groups, layers) was used.Each  tested  model,  shallow  or  decoupled  had  the  total  number  of  1000  neurons  (with  the  small deviations  resulting  from  subreservoir  integer  sizes).  Grid  search  optimization  was  performed  on all these hyperparameters and architectures. The best configuration was selected based on minimal NRMSE score obtained on validation set. In the main part of experiment, which includes 1-step ahead  prediction  of  time-series,  the  average and minimal NRMSE  on  the  test  set  was calculated for each architecture and the target hyperparameter set.HyperparametertanhSNAInput Scaling s{0.1,0.5,1.0,10}1.0Spectral Radius ρ{0.7,0.8,0.9,1.0}1.0Leaking Rate α{0.7,0.8,0.9,1.0}1.0Regularization β{0.5,1,2}{0.5,1,2}Actiation Radius r-{50k | k∈1,2,3...30}Washout100Total neurons1000Sparsity10%Weight distributionuniform, centered around 0Best Architecture configurationESNHyperparameter sets for best results in different tasks are listed below:MackeyGlass Series one step ahead prediction:ArchitectureLayersGroupsActivation RadiusRegularizationESN tanh11None0.5dESN tanh41None0.5gESN tanh12None0.5gdESN tanh24None0.5-------------------------------------------------------------------------ESN SNA111000.5dESN SNA311000.5gESN SNA1102000.5gdESN SNA33500.5Multiple Superimposed Oscillators one step ahead prediction:ArchitectureLayersGroupsActivation RadiusRegularizationESN tanh11None0.5dESN tanh21None0.5gESN tanh12None0.5gdESN tanh33None0.5-------------------------------------------------------------------------ESN SNA1114000.5dESN SNA4112001.0gESN SNA1313001.0gdESN SNA3214001.0Sunspot Series one step ahead prediction:ArchitectureLayersGroupsActivation RadiusRegularizationESN tanh11None2.0dESN tanh21None2.0gESN tanh120None1.0gdESN tanh33None2.0-------------------------------------------------------------------------ESN SNA118501.0dESN SNA3114001.0gESN SNA184002.0gdESN SNA3214501.0LSTMAdditionally, several experiments with moving average and LSTM networks were performed.For LSTM 4 architectures were testedArchitecture 1Architecture 2Architecture 3Architecture 4Each architecture was trained for 100 epochs with Adam optimiser.
Each model was trained 5 times for each learning rate of [0.001, 0.002, 0.0005,0.005].Best results were obtained with:Architecture 2 and learning rate 0.002 for MackeGlassArchitecture 3 and learning rate 0.002 for SunspotArchitecture 4 and learning rate 0.005 for Multiple Superimposed oscillatorsMemory CapacityThe objective of this experiment was to measure how ESN models can recall past input seen by reservoirNsteps ago, where N∈{1,5,10,15,...100}. In particular, we compare different  decoupled  SNA  architectures based  on  the  hyperparameter  configuration established in the one step ahead preadiction for MG dataset. For eachN, N steps behind NRMSE was averaged over 10 trials. Several phenomena observed in the results of the experiment are depicted in Figures below.In case ofdESNSNA architecture, memory capacity decreases faster for bigger number of layers. It is important to note that the total number of neurons remains the same(1000).Similar observation can be done forgESNSNA architecture, however here, the influence of adding more groups on the memory is lower.These conclusions find further confirmation forgdESN, where the smaller the product of layers and groups, the better the memory capacity. In case ofgdESNSNA, we observe that two-dimensional configuration of layers and groups (e.g. (2,4) vs. (4,2)) affects memory characteristics and can be used to find trade-off between memory and representational power.In general, two clean patterns emerge - one bigger reservoir has better Memory Capacity than two either stacked or grouped as long as total number of neurons stays the same.
Grouped reservoirs have better memory capacity than Deep ones for the same configuration, eg. 2 layers 500 neurons each vs two groups 500 neurons each.Input scaling vs Activation RadiusFor SNA architectures only the one hyperparameter from the set: {Input scalings, Acativation radiusr, Spectral radiusρ} can be adjusted. The others can be fixed with no negative influence on the model memory. This  conclusion  comes  from  normalization  effect  of  activation  radiusrand  is  confirmed  by  the empirical results presented below and obtained for SNA ESN, with divsr∈ {10,20,30,40}, where divsr=s/r. For each divsr, 20 different values ofsandrwere selected. For each of these configurations, the models with the same initial weights were trained in 10 trials. As presented in Figure below the models with the same divsrexhibit practically the same memory capacity.The figure below presents Memory  Capacity  NRMSE  scores for recalling input seen 30 steps before for ESN  with  fixed  ratio  of divsr,  where divsr∈ {10,20,30,40}.  For  each divsr,  20  different  values  ofsandrwere  chosen.  All experiments were conducted on SNA ESN with fixed weights (s and r were changing at the begging of the training but not the weights!). 10 diferent SNA ESNs were initialized with different weights and the results were averaged for each configuration.