# Memory-Non-Linearity Trade-Off in Distance-Based Delay Networks
## 4 Results
### Delayed XOR  
To allow for a comparison between ESNs and DDNs throughout a range of different delays \(k\), we would ideally optimize both systems for each of the 19 delays. This would result in 19 different hyperparameter sets. However, to decrease the optimization time, we instead chose to only run a single CMA-ES optimization for each model type, where a random \(k\) was sampled for each candidate evaluation (similarly to the approach in [14]). Because the task delay was random, and we performed multiple evaluations per candidate, the solutions with the highest fitness will be those that can perform well for most or all of the possible task delays, since over time, candidate solutions would be equally exposed to all XOR versions. As such, the optimization would ideally result in a single hyperparameter solution able to generate networks that generalize between the different task versions. This approach is sub-optimal from an application perspective, since the networks are not optimized to maximize their memory capacity at a specific lag. However, it is ideal to show the interplay between the non-linear processing power and memory, since generalizing over many different delays for such a non-linear task requires a larger short-term memory span compared to focusing on a single delay. For both ESNs and DDNs, we used a reservoir size of 200 neurons, equally distributed in four clusters, with a (global) non-zero connectivity percentage of 25%, a weight mean of \begin{table}
\begin{tabular}{l l l l} \hline \hline  & \multicolumn{2}{l}{**Equivalent Nr. of Generations**} & \multicolumn{2}{l}{**Equivalent Hyperparameters**} \\  & **Optimized ESN** & **Optimized DDN** & **Unoptimized ESN–Unoptimized DDN** \\ \hline NARMA-30 NRMSE & \(0.455\pm 0.070\) & \(0.087\pm 0.018\) & \(0.187\pm 0.170\) \\ NARMA-30 Task overlap & \(0.783\pm 0.066\) & \(0.940\pm 0.011\) & \(-0.192\pm 0.197\) \\ \hline \hline \end{tabular}
\end{table} 
Table 1: NARMA-30 results summary for optimized and unoptimized DDNs and ESNs: means \(\pm\) standard deviations. For optimized performance and overlap, means over multiple instantiations are reported. For unoptimized networks, difference scores are reported, since networks are identical, with only delays differing. Figure 7: ESN and DDN NARMA-30 test performance and overlap score throughout CMA-ES evolution. Each dot represents a single instantiation of a network. The same network activity was used to measure the performance and the regional IPC, which was then used to compute the NARMA-30 overlap score. The shade of the dot indicates the CMA-ES generation from which the network was sampled. Networks were sampled from the best candidate hyperparameter set of every 10 th generation, and five re-initializations were performed for each candidate. For readability, the generations were binned in five bins. Similarly to Figure 4, we see that task overlap and performance are correlated in both ESNs and DDNs. Moreover, we see that DDNs start off with a higher overlap and reach a higher overlap after hyperparameter optimization. zero, a weight scaling of one, a bias mean of zero, a bias scaling of 0.5, and a leak rate of 0.99. In the case of DDNs, which required neuron location parameters, the cluster centers were spread out evenly across the diagonal of the 2D space from which the neuron locations were sampled. The cluster variances in both the \(x\)- and \(y\)-axis corresponded to 0.84 delay steps, with an x-y correlation of zero. Similarly to the findings reported in [14; 17], we see that DDNs achieved a lower validation NRMSE. The best performing ESN and DDN candidates were each used to generate 20 networks. These were all trained and evaluated for all 19 different task delays, each time using a training set of 4000 samples and a test set of 10,000 samples. We report the BER for the test set in Figure 8.  
We compared the task delay-dependent performance between ESNs and DDNs for optimized and unoptimized networks.