# Memory-Non-Linearity Trade-Off in Distance-Based Delay Networks
## 3 Methods
### Information Processing Capacity  
Linear memory capacity was later generalized by [1] to the IPC, to include non-linear processing. Instead of reproducing the delayed input, a non-linear function of the input is estimated by the readout layer. As long as the set of non-linear functions form an orthogonal basis for the used i.i.d. input domain, the individual capacities corresponding to each basis function should be independent, and thus measure separate computing properties of the network. The information processing capacity was originally only defined for discrete time systems such as ESNs, although this was later extended to time-continuous systems such as spiking neural networks [30]. In this work, we analyzed the IPC for systems driven with an input that was uniformly distributed over the interval \([-1,1]\). For this input domain, an orthogonal basis was formed by the products of all possible combinations of normalized Legendre polynomials. The IPCwas then given by measuring all possible capacities, with each individual capacity being defined by the lag parameter \(l\) (meaning that we reproduced a function with input from \(l\) steps ago) and the combination of polynomials \(\{d_{i}\}\). This capacity is defined as  
\[C_{l}^{\{d_{i}\}}=r(z_{l}^{\{d_{i}\}},z_{l}^{\{d_{i}\}}(n))^{2} \tag{6}\]  
where \(r\) is [PERSON]'s correlation, \(\{d_{i}\}\) is the set of polynomial degrees used for this capacity, \(z_{l}^{\{d_{i}\}}\) is the function to be reproduced, and \(\hat{z}_{l}^{\{d_{i}\}}(n)\) is the readout reproduction based on the reservoir activity at time \(n\). The product of polynomials is defined as  
\[z_{l}^{\{d_{i}\}}=\prod_{i}P_{d_{i}}(u(n-l-i)) \tag{7}\]  
where \(P_{d}(\cdot)\) is the normalized Legendre polynomial of degree \(d\) and \(u(n-l)\) is the input of \(l\) steps ago. As such, for a lag of \(l\), the function to be reproduced can be any combination of polynomials of inputs from at least \(l\) steps ago. The set \(\{d_{i}\}\) can contain any combination of polynomials with degrees larger than zero. With these conditions, each capacity is guaranteed to capture the independent computational properties of the system. Because we wanted to characterize the trade-off between memory and non-linearity, we ordered capacities based on their (maximum) degree (i.e., the largest \(d\) in each set, \(\{d_{i}\}\)) and the lag, \(l\). As such, linear memory was defined by the IPC of degree 1 and non-linear processing by larger degrees. As we computed the IPC for higher maximum degrees, we saw a combinatorial explosion of capacities, as the amount of combinations possible for set \(\{d_{i}\}\) increased. Similarly to the approach in [1], we therefore limited the amount of measured capacities, not only by the maximum degree, but also by a maximum window size, \(w\), such that \(i\leq w\), where \(i\) refers to the amount of delay steps above the lag (which is the smallest delay in the set of polynomials). This effectively limited the temporal distance between the latest and the earliest component of the capacity, thus reducing the IPC search space. Next, the total maximum number of polynomials, \(v\), was limited, such that the size of the set \(|\{d_{i}\}|\leq v\), where \(|\{d_{i}\}|\) refers to the size of the set of IPC polynomials. By limiting the set size of polynomials, we again reduced the number of capacities that needed to be computed. As such, the maximum window size and maximum number of polynomials shape the search space of the IPC, and should be informed by domain knowledge of the task to which the reservoir system is applied (further discussed in Section 3.5). The total theoretical IPC is the sum of all capacities for lags 0 to infinity, degrees 1 to infinity, and all possible combinations of polynomials. It was shown in [1] that the total IPC is always upper bounded by the amount of observed variables if these variables are linearly independent and the dynamical system has fading memory.