# Memory-Non-Linearity Trade-Off in Distance-Based Delay Networks
## 5 Discussion  
In this paper, we showed that variable inter-neuron delays in recurrent neural networks improve the memory-non-linearity trade-off inherent to dynamical systems. We emphasize that the improvement of this trade-off means improving the total available  
Figure 8: Test performance for ESNs and DDNs measured in BER, both before and after CMA-ES optimization. The horizontal axis shows the task delay used to generate each of the 19 delayed XOR test sets. BERs are averaged over 20 network samples. With a test set of \(10^{4}\) samples, BERs higher than \(10^{-3}\) are reported at a 90% confidence level [39]. The vertical error bars represent the standard error. usable and task-relevant information processing capacity. To demonstrate this claim, we used several approaches. First, we showed that the random use of delays in unoptimized, randomly initialized DDNs results in a larger measured total IPC compared to ESNs. We further observed that unoptimized ESNs tend to have more non-linearity and unoptimized DDNs a longer range linear memory. However, in [1] it was previously shown that the total IPC is determined by the number of observed variables in the dynamical system (in this case, the number of neurons in the reservoir), and as such, an increase in linear memory capacity must necessarily come at the cost of non-linear computation. The measured IPC can be higher or lower depending on how we limit the capacity search space and on how many capacities are statistically insignificant. Nevertheless, we observed a better balance between memory and non-linearity in DDNs, even though the theoretical total IPC was constant, because they achieved a higher non-linearity at larger memory spans. To explain these findings, it is more relevant to only consider the portion of the IPC that is 'useful'. The notion of the useful IPC can only be defined in the context of a task. To quantify this, we used the task capacity, reflecting the computational requirements of a task. We computed an overlap score between the task capacity profile and the reservoir IPC profile as an inner product of the capacity vectors. We see that task performance and task overlap are highly correlated metrics, further supporting the hypothesis that the improved DDN task performance is due to strong non-linearity with larger memory spans, as required by the NARMA-30 task. However, the random introduction of delays does not reflect the improved tuning flexibility of DDNs. As such, we measured the IPC and task overlap for ESNs and DDNs optimized for the NARMA-30 task. We see that optimized DDNs achieved a better overlap score compared to ESNs. Similarly to the findings on linear memory capacity presented in [27], we see that DDN IPC profiles were not bound to monotonically decrease with delay parameters such as the lag and delay window. Instead, DDNs were flexible enough to have concentrated peaks in the IPC in specific areas, while conserving capacity in irrelevant computations. On the other hand, we consistently observed monotonically decreasing ESN capacity profiles, both for linear memory and the non-linear IPC. ESNs can only increase their capacities at any delay by increasing all previous capacities. This can be explained by the fact that, to increase their linear memory, ESNs must operate in a more linear regime. Any information passed through a non-linear activation function will decrease linear memory. Therefore, it is impossible for ESNs to achieve strong non-linearity at large lags. Although the total IPC is theoretically still the same as for equivalent DDNs, we can conclude that DDNs achieve a better balance between memory and non-linearity, not because they can increase both at the same time, but because they can have peaks in the non-linear capacity at large lags by offloading linear memory to the distributed inter-neuron delays. Moreover, we also showed that in the case of optimized ESNs and DDNs, the task overlap is correlated with task performance.