# Memory-Non-Linearity Trade-Off in Distance-Based Delay Networks
## 2 Background  
ESNs were initially introduced as a computationally efficient alternative to traditional RNNs trained using BPTT. Instead of BPTT, ESNs utilize a fixed recurrent'reservoir' comprised of (usually leaky) sigmoid or hyperbolic tangent neurons. Only a linear readout layer undergoes training. Conventional ESN reservoirs are input-driven dynamical systems described by the following equation. \[\mathbf{x}(n+1)=(1-a)\mathbf{x}(n)+a\cdot f(\mathbf{W}_{\text{res}}\mathbf{x}( n)+\mathbf{b}_{\text{res}}+\mathbf{W}_{\text{in}}\mathbf{v}(n)) \tag{1}\]  
In this context, the variables \(\mathbf{x}(n)\), \(a\), \(\mathbf{W}_{\text{res}}\), \(\mathbf{W}_{\text{in}}\), \(\mathbf{b}_{\text{res}}\), \(f\), and \(\mathbf{v}(n)\) denote the network state at timestep \(n\), the decay rate, the weight matrix for the (recurrent) reservoir, the weight matrix for inputs, the bias weights of the reservoir, the activation function, and the input at timestep \(n\), respectively. Balancing memory with non-linear computation is a fundamental problem in dynamical systems and RNNs that are used as reservoirs, and has received much attention in recent publications [1; 29; 30; 1]. Alongside the aforementioned DDNs, several ESN adaptations are claimed to improve this memory-non-linearity trade-off, which usually means increasing the memory of a system without degrading its non-linear processing power, or vice versa. Indeed, ref. [2] show that non-linear activation functions work best in tasks that require strong non-linear transformations with a short memory, and tasks requiring a long memory with weak non-linear transformations are best solved using linear ESNs. However, in the case of tasks that require both memory and strong non-linearity, mixture reservoirs, which use a combination of linear and non-linear neurons, outperform both linear and non-linear conventional ESNs [2]. Similarly to DDNs, mixture reservoirs include some degree of linear processing; however, DDNs do so without reducing the number of non-linear nodes, by offloading the linearity to inter-neuron delays. In the case of physical reservoirs, inter-node delay is computationally free, as it is inherent to the system. Another approach, called an edge of stability ESN (ES\({}^{2}\)N), is shown in [29]. Here, the proximity of the reservoir to chaotic behavior can be manipulated by hyperparameter tuning. Specifically, instead of having a single leak parameter for the entire network, individual leak parameters are optimized per neuron by scaling the leak rate with an orthogonal matrix. The resulting networks are shown to perform better at the same tasks used in [2], which were designed to require strong non-linear transformations with large short-term memory spans. Although both models were validated on tasks that require high degrees of non-linearity and large memory spans, it is difficult to formally quantify whether the trade-off is really improved. The balance between linear memory and the non-linear IPC is formalized in [1]. The IPC measures the capacity of a dynamical system to reconstruct a set of the non-linear basis functions of its past inputs. As such, capacities can be measured for different degrees of non-linearity and across different memory spans. However, it has been proven that the total IPC, which is the sum of all possible IPC components, is always equal to the number of observed variables, if the dynamical system has the fading memory property and linearly independent state variables. As such, it is impossible to increase the theoretical total IPC for reservoirs with the same number of neurons. The memory-non-linearity trade-off, as originally defined, cannot be improved, as increasing linear memory in a system will always decrease the remaining IPC. Nevertheless, improvements in tasks that require both are presented in [2], indicating that in practice, the amount of the 'usable' IPC (i.e., task-relevant IPC) can be improved.