# Memory-Non-Linearity Trade-Off in Distance-Based Delay Networks
## 4 Results
### Unoptimized Networks  
We randomly initialized (single-cluster) DDNs of 100 neurons, with normally distributed neuron locations centered around the input neuron, with a standard deviation corresponding to two delay steps in both the \(x\)- and \(y\)-axis and a correlation of zero between the x and y coordinates. The weights of these networks were uniformly distributed, centered around zero, with half of the weights set to zero (i.e., a connectivity parameter of 0.5). The bias weights were distributed uniformly between zero and one. We repeated the initialization with a varying spectral radius between 0.1 and 2 and leak parameter between 0.3 and 1 over five steps each. For each parameter combination, we repeated the initialization twice and measured the IPC for all capacities up to a degree of three, a lag of 60, a maximum window size of 30, and a maximum of two polynomials. To this end, we first generated the network activity with the same input set of 6000 samples distributed uniformly between minus one and one, and then measured the capacities using this network activity. The purpose of these measurements was to illustrate the difference in the average IPC profile between ESNs and DDNs; hence, the granularity of the parameter space was limited in the interest of computation time. In Figure 1, we show the IPC (i.e., capacities plotted against the lag for all measured degrees) averaged over all network initializations and all parameter settings. AlthoughESNs have a lower average total measured IPC compared to DDNs, we see that (at least for unoptimized networks) DDNs are more linear, with the IPC profiles spanning larger lags, whereas unoptimized ESN IPC profiles are concentrated in more non-linear regions, but reach smaller lags. As stated earlier, the total IPC should in theory always be equal to the number of observed variables; hence, the difference in the total measured IPC must either lie in unexplored regions (e.g., basis functions with more polynomials) or be spread out over many statistically insignificant (hence sub-threshold) capacities. As such, we must return to the concept of useful capacities to be able to explain the performance difference between DDNs and ESNs. To this end, we repeated the previous experiment, but instead of measuring the IPC over the full search space, we measured the region-specific IPC (i.e., we only measured those capacities that had a corresponding non-zero task capacity). Limiting the number of measured capacities reduced the computational load, so for this experiment, we increased the granularity for the spectral radius and leak rate space to 12 steps, and we repeated each network instantiation three times. Using the region-specific IPC, we then computed the task overlap as the inner product between the vector of task capacities and the reservoir IPC, as shown in Equation (9). For each network, we repeated the experiment with its ESN counterpart, obtained by setting all delays to zero. As such, we compared ESNs and DDNs which only differed in terms of delay, while keeping all other network parameters identical (i.e., the weights, input weights, biases, connectivity, and leak rate). In Figure 2, we present the resulting task overlap for each combination of the leak rate and spectral radius. We see that the overlap is higher for DDNs for most parameter combinations, and also reaches a higher maximum overlap. We generated a NARMA-30 training set of 4000 samples to train the readout layers. Then, using the same random inputs and corresponding network activity used to measure the IPC before, we generated NARMA-30 outputs and evaluated the network  
Figure 1: Average IPC profile for randomly initialized unoptimized ESNs and DDNs. Figure 2: Task–reservoir IPC overlap for various random network initializations. The right-hand side shows the overlap scores of DDNs initialized with random, normally distributed neuron locations. The left-hand side shows the overlap of equivalent ESNs. All scores are averaged over 3 random initializations with the same leak rate and spectral radius.