# Memory-Non-Linearity Trade-Off in Distance-Based Delay Networks
## 1 Introduction  
Temporally complex machine learning tasks often require a combination of memory and non-linear processing. As such, using a computational substrate that facilitates a combination of non-linear computing and a large short-term memory span is beneficial for processing spatiotemporal information. However, in dynamical systems, there is a trade-off relation between linear memory and non-linear dynamics, since non-linear computations degrade memory [1; 2]. This poses a problem for tasks that require both. Reservoir computing (RC) is a brain-inspired machine learning paradigm based on harnessing the intrinsic temporal processing properties of dynamical systems. These dynamical systems, referred to as reservoirs, serve as temporal kernels [3] that map a low-dimensional input signal to a high-dimensional projection, which is then fed to a simple linear readout model. This increase in dimensionality usually increases the linear separability of the data. Traditionally, this involved (software-simulated) recurrent neural networks (RNNs) as reservoirs. More recently, the field of RC has been mainly focused on physical reservoir computing, where physical hardware systems such as photonic, electronic, or memristive hardware are used as reservoirs [4; 5; 6]. These physical substrates have the advantage of being fast, since they do not need to be simulated on traditional hardware, and energy-efficient, with many physical reservoirs being mostly passive [7]. This speed and efficiency offers the possibility of real-time processing with applications in speech recognition [8; 9], forecasting and signal generation [10; 11], and robot control [12]. However, besides physical reservoir computing, software-simulated reservoir computing is still used as a model system for physical RC, and as a test-bed for novel brain-inspired mechanisms (e.g., [13; 14; 15]). Some of the earliest software-simulated RC frameworks were echo state networks (ESNs), introduced by [16] as an alternative to optimizing recurrentweights using back-propagation through time (BPTT). ESNs consist of a fixed RNN reservoir with randomly initialized weights and a linear regression readout layer. The neurons are simple hyperbolic tangent or sigmoid units. ESNs have good performance in tasks with temporal dependencies, while requiring no RNN weight training. However, because ESN reservoirs are randomly initialized, they can be seen as a sample from a distribution. Although RNN weights are fixed, these distribution parameters need to be optimized as hyperparameters for specific tasks. In the RNNs used as ESN reservoirs, non-linear processing is provided by non-linear neuron activation functions. Furthermore, the maintains part of the input information through the use of recurrent connections and leak parameters, providing the system with short-term memory. Many temporal machine learning problems require a combination of linear memory and non-linear processing, with the span of this short-term memory varying per task, and even within the same task. ESNs are highly flexible in this regard. Due to a high-dimensional and temporally dependent reservoir state, they provide information from a wide temporal range. However, refs. [1; 2] show that, to increase ESN memory without increasing the reservoir size, reservoirs have to operate in a more linear regime. In dynamical systems, memory, especially for longer time spans, is in direct competition with the non-linear processing capacity. Intuitively, the amount of information about the input signal that can be retained in the reservoir is diminished by altering that information with non-linear transformations. The only way to increase both memory and non-linear processing is by increasing the reservoir size (i.e., increasing the number of neurons). This memory-non-linearity trade-off puts ESNs at a disadvantage for tasks that require recombining information non-linearly across a large memory span.