# Memory-Non-Linearity Trade-Off in Distance-Based Delay Networks
## 4 Results
### Narma  
We optimized DDN and ESN hyperparameters for the NARMA-30 system approximation task using the CMA-ES. We generated 10 ESN reservoirs and 10 DDN reservoirs from the best hyperparameter candidates, with both model types comprised of 100 neurons in four clusters. The exact hyperparameter settings of the best networks are shown in Appendix A for both ESNs and DDNs. IPCs up to a degree of two were computed for each of the 10 sampled networks and averaged. In Figure 5, we show the best candidate's IPC profiles up to a lag of 60 for the ESN and DDN models. These profiles are compared with the NARMA-30 task capacity (bottom graph). We observe that both ESNs and DDNs were able to maintain an equally high linear MC up to a lag of 30, and both had a high second-order IPC at a lag of 1. Next, we see that the total IPC measured for ESNs was higher compared to DDNs for both the linear and quadratic IPC. We observe that the task requirements were focused at a lag of one and thirty for linear memory, and at a lag of one for the second-order IPC. At first sight, this result seems contradictory to the fact that DDNs achieve a better task performance than ESNs, because the task overlap seems to be equivalent for both network types. However, only plotting the capacity against the lag ignores the IPC window size (i.e., the difference between the largest and the smallest delay in the basis function). Note that these networks were optimized for the NARMA-30 task. Therefore, the largest linear mem  
Figure 5: First- and second-degree IPC of networks optimized for the NARMA-30 task. Capacity profiles are the average of 10 reservoirs. The legend shows the total capacity for each degree, computed as the sum of capacities over all lags. The bottom graph shows the task capacity of both tasks, indicating task information processing requirements. ory requirements for the system were at lag one and lag thirty. By looking at Equation (10), we see that the NARMA-30 task also requires second-order computations: multiplying the input from 30 steps ago with the input from 1 step ago. This corresponds with a window size of 29. Indeed, when we plot the IPC for ESNs and DDNs and the NARMA-30 task capacities as a heatmap over the window size and lag (Figure 6), it becomes clear that there is no second-degree task overlap at all for the ESNs, as they do not achieve non-zero capacities for such large window sizes. In accordance with findings presented by [27], we see that DDNs are able to focus the IPC in specific regions in the capacity search space. In contrast, ESNs can only achieve memory over a larger lag span by operating in a more linear regime, due to IPCs that monotonically decrease with the lag and window size. Traditionally, this is inherent to the fading memory property of ESNs. DDNs offer more tuning flexibility in terms of the IPC, allowing for a high IPC at higher degrees, higher lags, and higher window sizes compared to ESNs. Since the only difference between ESNs and DDNs is the presence of delays, we can attribute the flexibility in IPC profiles to the tuning of delays, which allows for the concentration of the IPC in task-relevant capacity regions by sacrificing the IPC in irrelevant regions. Even though we do not see a large difference between the total IPCs, the DDN IPC profiles are much better aligned with the task capacities. To further demonstrate this effect, we repeated the experiment discussed in Section 4.1. Using the validation scores from the CMA-ES optimization of ESNs and DDNs for the NARMA-30 task, we picked the best performing candidate hyperparameter set from every 10 th generation. From each candidate, we sampled five networks. Each was re-trained and re-evaluated on a NARMA-30 test set. With the same network inputs and activity, the overlap score (Equation (9)) was computed.