Nanowire Neural Networks for time-series processing

Authors: veronica  pistolesi
andrea  ceni
gianluca  milano
carlo  ricciardi
claudio  gallicchio
Venue: NeurIPS_cc_2024_Workshop_MLNCP
Published: 2024-09-09 10:56:16

Abstract: We introduce a novel computational framework inspired by the physics of nanowire memristive networks, which we embed into the context of Recurrent Neural Networks (RNNs) for time-series processing. Our proposed Nanowire Neural Network architecture leverages both the principles of Reservoir Computing (RC) and fully trainable RNNs, providing a versatile platform for sequence learning. 

We demonstrate the effectiveness of the proposed approach across diverse regression and classification tasks, showcasing performance that is competitive with traditional RC and fully trainable RNNs. Our results highlight the scalability and adaptability of nanowire-based architectures, offering a promising path toward efficient neuromorphic computing for complex sequence-based applications.

Main paper: 
 introduction: 
 Neuromorphic computing, inspired by the structure and function of the human brain, aims to replicate biological information processing systems for the development of efficient and brain-like computational frameworks [Marković et al., 2020] . A key approach within this field is leveraging physical substrates, which bypass traditional digital simulations, to perform computation based on their inherent physical properties [Nakajima, 2020] . Such systems offer an opportunity to implement more scalable and efficient computational models compared to conventional methods.Reservoir Computing (RC), and particularly the Echo State Network (ESN) approach, provides a theoretical framework that aligns well with these principles, particularly for recurrent neural networks (RNNs) implemented in physical systems [Nakajima and Fischer, 2021, Yan et al., 2024] . In RC, the recurrent layer, known as the reservoir, remains fixed during training, while only the readout layer is trained. This reduces training complexity and allows efficient mapping of input signals into high-dimensional spaces for tasks such as time-series processing. The fixed nature of the reservoir offers a natural avenue for integrating physical systems, enabling their use as computational reservoirs [Tanaka et al., 2019] . Milano et al. [Milano et al., 2022] takes this concept further by demonstrating how self-organizing memristive nanowire networks can serve as physical reservoirs for computation. These nanowire networks map spatio-temporal inputs into a feature space through their inherent nonlinear dynamics, offering fault tolerance and adaptability, akin to biological neuronal circuits. This approach, termed in-materia computing, underscores the potential for neuromorphic architectures to implement brain-inspired paradigms with reduced training costs while maintaining robust performance in tasks like time-series prediction.Building on this foundation, we propose to explore the scalability and potential of nanowire networks by introducing physics-inspired computational models, named Nanowire Echo State Network (NW-ESN), and Nanowire Recurrent Neural Network (NW-RNN). We analyze the architectural parameters of these models and assess their performance across several time-series tasks. To validate our approach, we perform experiments on both classification and regression tasks. Through these experiments, we demonstrate that nanowire-based models can achieve competitive results compared to traditional methods, providing insights into their scalability and applicability in neuromorphic computing. Our findings highlight the potential of these models in advancing next-generation intelligent systems and robust, adaptive computational frameworks. 
nanowire networks: 
 Nanowire networks consist of tiny, highly conductive wires, made from metallic materials, arranged in a mesh-like pattern, see Fig. 1 for a realistic picture. These networks mimic the physical structure of the human brain, with the nanowires acting like neurons and their intersections resembling synapses. This unique structure allows nanowire networks to exhibit properties similar to neural networks, including the ability to learn and remember. Interestingly, the single memristive dynamics Figure 1 : Image of a nanoscale self-assembled Ag nanowire network obtained through scanning electron microscopy (scale bar, 2 µm). This represents a memristive network, since each junction between nanowires is a memristive element and the current flow relies on the history of applied voltage and current. at the intersection of two intersecting nanowires can be modeled as the transfer of ions through the membrane of a neuronal cell, as is the case for the three ionic channels of the Hodgkin-Huxley model [Nelson and Rinzel, 1995] . Specifically, the state equation for the synaptic properties of the NW under electrical stimulation is expressed as the following potentiation-depression rate balance equation Miranda et al. [2020] :EQUATIONIn Eq. 1, g ij is the normalized conductance (memory state) that assume values in between 0 and 1 while κ P,ij (V ij ) and κ D,ij (V ij ) are the potentiation and depression rate coefficients that are assumed to be function of the applied voltage through exponential relations, as expected for diffusion of ions:κ P,ij (V ij ) = κ P 0 exp(+η P V ij ) (2) κ D,ij (V ij ) = κ D0 exp(-η D V ij ),(3) where κ P 0 , κ D0 > 0 are constants while η P , η D > 0 are transition rates. 
computational neural model: 
 In this work, we develop Nanowire Neural Networks for time-series processing where the physical memory state g becomes the nanowire network state. To achieve this, we rewrite Eq. 1, 2 and 3 as:EQUATIONK p = K p0 • e ηpx , K d = K d0 • e -η d x ,(5) where 0 ≤ h ≤ 1 is the reservoir state and K p and K d are the two non-linearities which process the input x, keeping the other physical variables (K p0 , K d0 , η p , η d > 0) as network hyperparameters. The state transition function (Eq.6) of a single Nanowire Neuron is obtained by discretizing Eq. 4 using the forward Euler's method, leading to the following formulation:EQUATION)Equation 6 is then used to construct the fully connected Nanowire Neural Network (Fig. 2 ).Figure 2 : Nanowire Network computational graph where: h(t) and h(t + 1), in green, are the current and the next hidden state vectors, respectively, x(t + 1), in red, is the external input vector, W h is the recurrent weight matrix (or recurrent kernel), W x , in magenta, is the input weight matrix (or kernel) and ∆ t is the discretization time (non-linearity operations and their trajectories are marked by a dashed line).The dynamics of the Nanowire Neural Network are described by the following set of equations:EQUATIONwhere h(t) indicates the state of the recurrent layer, x(t) is the external input signal, W h is the recurrent kernel, W x is the (input-to-recurrent) kernel, and b is a bias vector.Our proposed model admits the following fundamental set of (nanowire-related) hyperparameters: K p0 , K d0 , η p , η d , ∆ t , and γ. Notice that we introduce the γ factor to allow the model to show asymptotically stable dynamics (the Echo State Property [Yildiz et al., 2012] ) when γ < 1. Moreover, for a certain configuration of hyperparameters, we observe that manipulating the input x impacts the reservoir state. Notably, a specific input range allows system's fixed point of Eq. 4 to move in a suitable region for information representation and learning. For this reason, the result of the affine transformation in Eq. 7 is subject to a proper rescaling operation before entering in the non-linearities. This heuristic enhances the stability and performance of the computational model Eq.s 7-10, and is computed component-wise as follows:EQUATIONwhere a = 0.35, b = 1.15, and s is a slope parameter.The computational graph of the proposed Nanowire Neural Network is illustrated in Figure 2 .In the following, we give two specific instances of the approach introduced so far, by framing it in the context of RC and fully trainable RNNs.NW-ESN is the reservoir-based Nanowire Neural Network Model in which both the kernel (W x ) and the recurrent kernel (W h ) matrices are initialized under stability constraints and left untrained. As usual in RC applications, we rescale the kernel W x , the bias b, and recurrent kernel W h , with the input scaling (ω), the bias scaling (β), and the spectral radius (ρ), respectively, and we use them as hyperparameters of the model.NW-RNN is the fully trainable version of the Nanowire Neural Network Model where both the kernel (W x ) and the recurrent kernel (W h ) matrices are trained end-to-end with backpropagation through time (BPTT). 
experiments: 
 We have validated our models for both time series classification and regression problems. To this end, we have considered the following datasets: FordA, ECG5000 In all experiments, the performance achieved by our models NW-RNN and NW-ESN was compared with that of conventional RNNs and ESNs. Specifically, for the ESN model, we have considered the leaky-integrator neuron formulation [Jaeger et al., 2007] , which includes a leaking-rate hyperparameter α. For all experiments, we used networks with a number of N = 100 recurrent neurons.In this respect, also note that the output layer is applied to the final state computed by the recurrent component for each input sequence in all models. During the model selection, the nanowire-related hyparameters of the NW-ESN and NW-RNN models were set to the values reported in Tables 3c, 3a, 3d and 3b show the ranges of values explored for the remaining hyperparameters for NW-ESN, NW-RNN, ESN and RNN, respectively. After the model selection phase, the model evaluation of all models was based on the average performance over five trials.The results of our experiments, presented in Table 4 , cover both classification and regression tasks. The scores for the classification tasks (FordA, ECG5000, SyntheticControl, Earthquakes) are expressed as accuracy (↑), while the regression tasks (FloodModeling1 and FloodModeling2) are reported as root mean squared errors (↓). We can first observe that NW-RNN leads to a consistent improvement compared to the conventional RNN approach. This suggests that introducing nanowire-inspired dynamics into the fully trainable RNN model can yield benefits in complex time-series classification tasks, while still maintaining the flexibility of full backpropagation. However, what stands out most is the remarkable performance of the NW-ESN model. Notably, NW-ESN generally outperforms (with a single exception) the conventional ESN, sometimes by a significant margin. For instance, on the SyntheticControl dataset, NW-ESN shows a clear improvement in accuracy, highlighting the ability of the nanowire-based reservoir to suitably capture more complex temporal patterns. This improvement is especially noteworthy considering that NW-ESN retains the same computational complexity as ESN, as the reservoir dynamics are fixed and only the readout is trained.Finally, observe that NW-ESN achieves the best overall performance on the majority of the tasks, despite having fewer trainable parameters compared to fully trainable models like NW-RNN. This efficiency, combined with its strong performance, emphasizes the potential of nanowire-based reservoirs as a powerful yet lightweight alternative for time-series processing, offering both scalability and accuracy without the added complexity of training recurrent dynamics. 
conclusions: 
 In this paper, we introduced and explored the application of nanowire-based neural networks (NW-ESN and NW-RNN) for time-series processing, comparing their performance with traditional models such as ESNs and RNNs. Our experimental results demonstrate that nanowire-based models can offer competitive, and in many cases superior, performance.NW-ESN, in particular, achieved the best performance in the majority of the tasks, highlighting the effectiveness of integrating the inherent nonlinear dynamics of nanowire networks into the Reservoir Computing paradigm. Despite having the same computational complexity as ESNs, NW-ESN exhibited significant improvements in performance across tasks, showcasing the potential of nanowire reservoirs to capture complex temporal patterns effectively. Moreover, NW-RNN demonstrated its capacity to enhance fully trainable models like RNNs by incorporating nanowire dynamics, leading to slight gains in classification and regression tasks. These results suggest that nanowire-based models offer a promising path toward the development of more efficient, adaptive, and scalable neural architectures for time-series analysis.This work represents a step forward in the field of neuromorphic computing, underscoring the potential of nanowire-based architectures to bridge the gap between biological information processing and machine learning. Future research can build upon this foundation by further investigating the scalability of these models and extending their applications to more complex tasks and domains, ultimately advancing the capabilities of next-generation intelligent systems. 
 
 Appendix: Second Workshop on Machine Learning with New Compute Paradigms at NeurIPS 2024 (MLNCP 2024). 



Reviews: review: title: title: Nanowire dynamics increase performance of time-series processing
main_review: main_review: Wow very interesting paper! The authors explore nanowire dynamics by creating the NW-ESN reservoir model and the NW-RNN fully trainable nanowire inspired RNN model. The motivation here is to see if these nanowires can be used for novel Neuromorphic architectures. The authors not only showed that they can match performance with their nanowire networks, but they actually showed it actually improved upon the conventional echo state network in reservoir computing and for the common RNN architectures. The paper is well written, thorough in the math behind the nanowire dynamics, and the authors conducted experiments on 6 different datasets in two tasks (classification and regression).


I just have one minor question:
- would you be able to expand on why the nanowire dynamics actually improved performance? Especially why does NW-ESN perform better than the RNN version when the RNN is fully learnable? Could the RNN case not have learned the NW-ESN case? The experiments are all there, but the paper would benefit from a little more insight for why the nanowire based models are so successful. Why are the able to 'capture more complex temporal patterns'?
- typos: L.44 -- fig. 1 instead of fig. 2 shows the photo.

Overall, very interesting paper
title: title: Relevant topic and method, but missing standards of evaluation set by the ML community
main_review: main_review: # Summary
The paper investigates a model of interactions of nanowires from a machine learning perspective. Recurrent neurons are modeled via a potentiation-depression rate balance equation that was published in 2020. A discrete time recurrent neural network formulation is obtained from the continuous neuron dynamics. Experiments are conducted on 4 classification and 2 regression datasets, all of which are small scale in terms of samples and sequence lengths. The authors present results for fully trained RNNs and reservoir approches where the recurrent dynamics are kept constant after random initialization, and only the linear read-out is trained.
# Discussion
The topic of the paper, namely using the physics of networks of nanowires for time-series modeling, aligns well with the Workshop. 

Since the neuron model was already presented in Miranda et al. (2020), the contribution of this work is a study of the suitability of networks of these neurons for machine learning. Along these lines, the paper claims "competitive, and in many cases superior, performance". From a machine learning perspective, we would expect comparison and evaluation on standard benchmarks to back the claims made. Here, the reviewer sees major potential for improving the paper. 
1. No comparison with related works: To convince the reader that the implementation is competitive, we need fair comparison with relevant published baseline models on relevant datasets. How does the presented model compare against previously published models?
2. Lack of standard benchmarks: The machine learning community has worked out many standardized benchmarks where dozens of baselines from other works exist to compare against. Some simple ones serve as sanity checks such as copy and memory tasks. Then there is a hierarchy of difficulty with increasing value for convincing the reader (non-exhaustive list: sequential MNIST, small-scale audio classification like Speech Commands, small-scale language modeling like Penn Treebank, of course larger scale audio/language tasks). 
3. Hyperparameter search not extensive: RNNs can be quite sensitive to hyperparameters such as initialization and especially learning rates. The presented search does not convince the reviewer that one can conclude that the reservoir approach competes with fully trained RNNs (lines 133-137). E.g. all networks achieve basically random chance level (50%) on the FordA binary classification task, which might be evidence that the fully trained RNN was not able to learn properly.

Overall, the data presented is not sufficient to back the claims made. For example, we can be sure that fully trained networks are required to solve real problems like language modeling. To the best of my knowledge there is no competitive reservoir model that can compete with trained networks even on small scale language modeling (e.g. Penn Treebank), not even speaking of large-scale language modeling. The claim about "in many cases superior performance" (line 142) hence seems out of place.

An interesting observation of nanowire networks is that their state update is input dependent, a mechanism known as gating in the RNN community. Although the presented form is explicitly coded by the device physics instead of fully learned, the recent progress on gated RNNs such as Mamba and the likes, might serve as a motivation for further studying the presented model. For example, gated RNNs seem to admit stronger associative recall properties compared to linear-time invariant models. 

The paper is technically correct. However, it requires more convincing evaluation on the path to a full paper. I recommend accepting it for this workshop if the claims made are weakened and set in context of the limited evaluation. Please also add a few lines to explain why the datasets were chosen, what we can learn from experiments on these datasets and why we can learn it from the data. 
Further, I'd suggest to work on stronger evaluation for a full paper.
# Specific comments by line
- 29: Please clarify what is meant by "fault tolerance and adaptability" and provide evidence. e.g. spiking networks are often related with these attributes, but eventually are no more robust against e.g. adversarial examples than regular networks.
- 44: Figure 1 instead of 2?
- 63: Could you clarify why the state is 0<h<1 ? This is not immediately clear to me due to the exponential gating function, which might push the state to larger values?
- 79: Can the rescale operation be implemented in a physical nanowire network?
- 85-89: Can the authors discuss if these conditions can be met in manufacturing?
- 95-98: Datasets need a bit more description. It is neither clear nor convincing why these datasets are chosen and why well established datasets with established baseline models are ignored in this work.
- 133-137: As mentioned earlier, if an ESN performs as well as a trainable RNN there is either nothing to learn (i.e. the random features are fully descriptive of the task) or the RNN fails to learn. If there is nothing to learn, the task is too simple. I am absolutely sure that there is a scale at which random features will not suffice (and we probably don't have to move to GPT4 scale to see this). The presented evaluation methodology does not convince me that enough effort was spend to make the regular RNNs learn well enough to outperform the ESN. It rather seems that simple tasks and limited evaluation benefit the results.
- 142: Does the data back this claim?
- 143-145: does the data back this claim?
- 152: Overstating

# Limitations of the reviewer
The reviewer cannot evaluate the actually connections to manufacturing these systems, e.g. 
- what is the precision of manufacturing? 
- Is there drift in the parameters over longer times?
- Are the presented models robust against such physical processes?