# Memory-Non-Linearity Trade-Off in Distance-Based Delay Networks
## 1 Introduction
A recently introduced class of ESNs with brain-inspired adaptations, called distance-based delay networks (DDNs) [17], achieve a higher performance for several highly non-linear temporal benchmark tasks. In [14], we showed that the variable inter-neuron delays introduced in DDNs can be optimized to match network timescales to task requirements. Various (modeled) physical reservoir computing approaches commonly employ delay connections [18; 19; 20; 21], but are usually limited to individually engineered feedback connections rather than task-optimized distributions of delays. In contrast, in DDNs, inter-neuron delays are proportional to the inter-neuron distance, inspired by the properties of biological axons. The axonal signal propagation speed depends on the myelination and axon diameter [22], but simplifying these two factors, an axon applies a delay proportional to its length. As such, DDNs use the inter-neuron distance as a determining factor for the delay size. The use of varying axonal propagation delays has been shown to be essential for several neural pathways: among others, for sensory processing, coincidence detection, and motor control [23; 24; 25]. Moreover, a large variety of axonal delay ranges can be observed in most mammals [26], suggesting an evolutionary advantage of maintaining a variety of delays. Not all inter-neuron delays are axonal, with some being caused by integration in dendrites, somata, and synapses, which also include non-linear computations. In contrast, axonal delays are linear, transmitting action potentials without any filtering. In [27], we showed an increase in linear memory capacity without increasing the reservoir size and without swapping non-linear neurons with linear neurons. It was hypothesized that the improved DDN performance was the result of an improved memory-non-linearity trade-off. It was also shown that DDNs indeed have a higher linear memory capacity concentrated over larger time spans. However, it was not systematically tested whether linear memory capacity is not increased at the cost of non-linear processing. In this paper, we confirm that DDNs indeed achieve a better balance between linear memory and non-linear processing over larger time spans. DDNs offer a more flexible tuning of memory and non-linear processing, leading to a better balance between the two. The novel contribution presented here is an empirical proof of an improved memory-non-linearity trade-off when using tuneable inter-neuron delays. We first show that randomly introducing distributed inter-neuron delays in unoptimized networks already has a measurable effect on the memory-non-linearity trade-off. Using the information processing capacity (IPC) framework introduced by [1], we quantified the computational properties of the reservoir, the computational requirements of tasks, and the overlap between the two. We show that task performance is correlated with the task overlap. We show that DDNs optimized for the NARMA-30 task (a commonly used RC benchmark task) indeed achieve an increase in non-linear processing with larger memory spans, without a decrease in task-relevant linear memory, which better aligns with the task requirements. Lastly, we optimized DDNs and ESNs for the delayed XOR task [28], a highly non-linear task with easily interpretable short-term memory span requirements. The delayed XOR task requires non-linearity at increasingly large lags, depending on the task parameter, which is challenging for conventional ESNs. We show that the performance of ESNs decayed faster than that of DDNs as we increased the delay in this task.