# Memory-Non-Linearity Trade-Off in Distance-Based Delay Networks
## 5 Discussion
Throughout hyperparameter optimization, DDNs achieved a lower task NRMSE and higher task overlap, again indicating better tuning flexibility. Lastly, we showed an improved performance of DDNs for the bitwise delayed XOR task. This provides another example showing a better flexibility of DDNs in producing strong non-linearities at large delays. As a result of the superior generalization power of DDN hyperparameter solutions to different delayed XOR timescales, we thus obtained a generic digital task solver. With XOR being one of the most challenging digital tasks, such a solution can be of great interest to the physical/photonic reservoir computing field. Although it has been shown before that DDNs improve linear memory capacity at larger lags, we show for the first time that DDNs do this without reducing useful, task-specific non-linearity. With these conclusions and previous findings, it has become clear that the use of distributed delays can be beneficial for RNNs. However, current limitations are the costly optimization procedures for finding task-relevant delay distributions. Future DDN re search could focus on a better 'genotype' encoding of the network structure and topology, meaning that instead of simply encoding all network aspects as distributions, specific microcircuit patterns could be captured by the optimized hyperparameters. Another important future research direction is testing the added value of inter-neuron delays in real-world spatiotemporal problems, such as robot control [12], pattern generations for time series (e.g., weather forecasting, electric load forecasting [11], and financial forecasting [10]), and the enhancement of physical reservoirs with an optimized linear inter-node delay. Lastly, the effect of delay tuning on network stability and the proximity to chaotic behavior should be studied in future work. Conceptualization, S.I. and J.D. ; software, S.I. ; validation, S.I. ; formal analysis, S.I. ; writing--original draft preparation, S.I. ; writing--review and editing, S.I. and J.D. ; visualization, S.I. ; supervision, J.D. ; project administration, J.D. ; funding acquisition, J.D. All authors have read and agreed to the published version of the manuscript. This project has received funding from the European Union's Horizon 2020 research and innovation program under the [PERSON] grant, agreement No. 860949. Not applicable. The raw data supporting the conclusions of this article will be made available by the authors on request. The authors declare no conflicts of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript; or in the decision to publish the results. The following abbreviations are used in this manuscript: \begin{tabular}{l l} RNN & Recurrent neural network \\ RC & Reservoir computing \\ ESN & Echo state network \\ LSM & Liquid state machine \\ BPTT & Back-propagation through time \\ DDN & Distance-based delay network \\ ADDN & Adaptive distance-based delay network \\ IPC & Information processing capacity \\ MC & Memory capacity \\ CMA-ES & Covariance matrix adaptation evolutionary strategy \\ GMM & Gaussian mixture model \\ NARMA & Non-linear autoregressive moving average \\ NRMSE & Normalized root mean squared error \\ XOR & Exclusive or \\ \end{tabular}