The Expressive Leaky Memory Neuron: an Efficient and Expressive Phenomenological Neuron Model Can Solve Long-Horizon Tasks.
Authors: Aaron Spieler
Nasim Rahaman
Georg Martius
Bernhard Schölkopf
Anna Levina
Venue: ICLR 2024 poster (2024)
Keywords: Computational Neuroscience
Phenomenological Neuron Modeling
Cortical Neurons
Recurrent Neural Networks
Machine Learning
Biologically Inspired Modelling
Primary area: general machine learning (i.e., none of the above)
TL;DR: We propose the ELM neuron, and show that it can efficiently fit a biophysical cortical neurons I/O and solve challanging long termporal dependency tasks.

Abstract: Biological cortical neurons are remarkably sophisticated computational devices, temporally integrating their vast synaptic input over an intricate dendritic tree, subject to complex, nonlinearly interacting internal biological processes. 
A recent study proposed to characterize this complexity by fitting accurate surrogate models to replicate the input-output relationship of a detailed biophysical cortical pyramidal neuron model and discovered it needed temporal convolutional networks (TCN) with millions of parameters. 
Requiring these many parameters, however, could stem from a misalignment between the inductive biases of the TCN and cortical neuron's computations.
In light of this, and to explore the computational implications of leaky memory units and nonlinear dendritic processing, we introduce the Expressive Leaky Memory (ELM) neuron model, a biologically inspired phenomenological model of a cortical neuron.
Remarkably, by exploiting such slowly decaying memory-like hidden states and two-layered nonlinear integration of synaptic input, our ELM neuron can accurately match the aforementioned input-output relationship with under ten thousand trainable parameters.
To further assess the computational ramifications of our neuron design, we evaluate it on various tasks with demanding temporal structures, including the Long Range Arena (LRA) datasets, as well as a novel neuromorphic dataset based on the Spiking Heidelberg Digits dataset (SHD-Adding). Leveraging a larger number of memory units with sufficiently long timescales, and correspondingly sophisticated synaptic integration, the ELM neuron displays substantial long-range processing capabilities, reliably outperforming the classic Transformer or Chrono-LSTM architectures on LRA, and even solving the Pathfinder-X task with over 70\% accuracy (16k context length). These findings raise further questions about the computational sophistication of individual cortical neurons and their role in extracting complex long-range temporal dependencies.

Published as a conference paper at ICLR 2024
THE EXPRESSIVE LEAKY MEMORY NEURON: AN EFFICIENT AND EXPRESSIVE PHENOMENOLOGICAL NEURON MODEL CAN SOLVE LONG-HORIZON TASKS
Aaron Spieler! 12 Nasim Rahaman 22 Georg Martius Bernhard Schölkopf" and Anna Levina 1.
'University of Tübingen, Germany
Max Planck Institute for Intelligent Systems, Tübingen, Germany Mila, Quebec Al Institute, Canada
$ Max Planck Institute for Biological Cybernetics, Tübingen, Germany
ABSTRACT
Biological cortical neurons are remarkably sophisticated computational devices, temporally integrating their vast synaptic input over an intricate dendritic tree, subject to complex, nonlinearly interacting internal biological processes. A recent study proposed to characterize this complexity by fitting accurate surrogate models to replicate the input-output relationship of a detailed biophysical cortical pyramidal neuron model and discovered it needed temporal convolutional networks (TCN) with millions of parameters. Requiring these many parameters, however, could stem from misalignment between the inductive biases of the TCN and cortical neuron's 's computations. In light of this, and to explore the computational implications leaky memory units and nonlinear dendritic processing, introduce the Expressive Leaky Memory (ELM) neuron model, a biologically inspired phenomenological model of a cortical neuron. Remarkably, by exploiting such slowly decaying memory-like hidden states and two-layered nonlinear integration of synaptic input, our ELM neuron can accurately match the aforementioned input-output relationship with under ten thousand trainable parameters. To further assess the computational ramifications of our neuron design, we evaluate on various tasks with demanding temporal structures, including the Long Range Arena (LRA) datasets, well as a novel neuromorphic dataset based on the Spiking Heidelberg Digits dataset (SHD-Adding! Leveraging larger number of memory units with sufficiently long timescales, and correspondingly sophisticated synaptic integration, the ELM neuron displays substantial long- range processing capabilities, reliably outperforming the classic Transformer or Chrono-LSST architectures on LRA, and even solving the Pathfinder-X task with over 70% accuracy (16k context length). These findings raise further questions about the computational sophistication of individual cortical neurons and their role in extracting complex long-range temporal dependencies.
INTRODUCTION
The human brain has impressive computational capabilities, yet the precise mechanisms underpinning them remain largely undetermined. Two complementary directions are pursued in search of mechanisms for brain computations. On the one hand, many researchers investigate how these capabilities could arise from the collective activity of neurons connected into a complex network structure Maass (199)); Gerstner & Kistler (2002); Grüning & Bohte (2014), where individual neurons might be as basic as leaky integrators or ReI.U neurons. On the other hand, it has been proposed that the intrinsic computational power possessed by individual neurons Koch (1997); Koch
& Segev (0000); Silver (2010) contributes a significant part to the computations.
Even though most work focuses on the former hypothesis, increasing amount of evidence indicates that cortical neurons are remarkably sophisticated (Siveer (2010): Gidon et al. (2020); Larkum (2022).
comparable to expressive multilayered artificial neural networks Poirazi et al. (2003); Jadi et al.)
Beniague et al.] (202T); Jones & Kording (202T). I). and capable of discriminating between
dozens to hundreds of input patterns Gütig & Sompolinsky (2006); Hawkins & Ahmad (016)):

---

Published as a conference paper at ICLR 2024
Moldwin & Segev 2020). Numerous biological mechanisms, such as complex ion channel dynamics (e.g. NMDA nonlinearity Major et al. (2013); Lafourcade et al. (2022); Tang et al. (2023)), plasticity on various and especially longer timescales (c.g. slow spike frequency adaptation Kobayashı et al.)
(2009); y Bellec et al. 2) the intricate cell morphology (e.g. nonlinear integration by dendritic
tree Stuart & Spruston| (2015); Poirazi & Papoutsi (2020); Larkum (2022), ) and their interactions,
have been identified to contribute to their complexity.
Detailed biophysical models of cortical neurons aim to capture this inherent complexity through high-fidelity mechanistic simulations Hay et al. (20TT): Herz et al. (2006); [Almo & Korngreen (011)). However, they require a lot of computing resources to run and typically operate at a very fine level of granularity that does not facilitate the extraction of higher-level insights into the neuron's 's computational principles. A promising approach to derive such higher-level insights from simulations is through the training of surrogate phenomenological neuron models. Such models are designed to replicate the output of biophysical simulations but use simplified interpretable components. This approach was employed, for example, to model computation in the dendritic tree via simple two-layer ANN Poirazi et al. (2003); Tzilivaki et al.) (2019); Ujtalussy et al. (2018). Building on this line of research, a recent study by BBeniaguev al. (2021) developed temporal convolutional network to capture the spike-level input/output (I/O) relationship with millisecond precision, accounting for the complexity of integrating diverse synaptic input across the entirety of the dendritic tree of high-fidelity biophysical neuron model. It was found that a highly expressive temporal convolutional network with millions of parameters was essential to reproduce the aforementioned I/O relationship.
In this work, we propose that a model equipped with appropriate biologically inspired components that align with the high-level computational principles of cortical neuron should be capable of capturing the 1/O relationship using a substantially smaller model size. To achieve this, a model would likely need account for multiple mechanisms of neural expressivity and judiciously allocate computational resources and parameters in a rough analogy to biological neurons. Should such a construction be pos- sible, the required design choices may yield insights into principles of neural computation at the con- ceptual level. We proceed to design the Expressive Leaky Memory (ELM) neuron model (see Figure []. a biologically inspired phenomenological model of a cortical neuron. While biologically inspired, low-level biological processes are abstracted away for computational efficiency, and consequently, individual parameters of the ELM neuron are not designed for direct biophysical interpretability. Nevertheless, model ablations quired to emulate the cortical input/output relationship. The ELM neuron functions as a recurrent cell and can be conveniently used a drop-in replacement for LSTMs Hochreiter & Schmidhuber) (1997).
provide conceptual insights into the computational components re-
Our experiments show that a variant of the ELM neuron is expressive enough to accurately match the spike level I/O of a detailed biophysical model of a layer 5 pyramidal neuron at millisecond temporal resolution with a few thousand parameters, in stark contrast to the millions of parameters required by temporal convolutional networks. Conceptually, we find accurate surrogate models to require multiple memory-like hidden states with longer timescales and highly nonlinear synaptic integration. To explore the implications of neuron- neuron-internal timescales and sophisticated synaptic integration into multiple memory units, we first probe its temporal information integration capabilities on a challenging biologically inspired neuromorphic dataset requiring the addition of spike-encoded spoken digits. We find that the ELM neuron can outperform classic LSTMs leveraging a sufficient number of slowly decaying memory and highly nonlinear synaptic integration. We subsequently evaluate the ELM neuron on the well-established long sequence modeling LRA benchmarks from the machine learning literature, including the notoriously challenging Pathfinder-X task, where it achieves over 70% accuracy but many wansformer-based models do not learn at all.
Our contributions are the following.
1. We propose the phenomenological Expressive Leaky Memory (ELM) neuron model, a recurrent cell architecture inspired by biological cortical neurons.
2. The ELM neuron efficiently learns the input/output relationship of a sophisticated biophysi- cal model of a cortical neuron, indicating its inductive biases to be well aligned.
3. The ELM neuron facilitates the formulation and validation of hypotheses regarding the underlying high-level neuronal computations using suitable architectural ablations.
4. Lastly, we demonstrate the considerable long-sequence processing capabilities of the ELM neuron through the use of long memory and synapse timescales.
2

---

Published as a conference paper at ICLR 2024
2
THE EXPRESSIVE LEAKY MEMORY NEURON
exp(-at/ew)=k-
W
m
at/ee
MLP
S
S:
W.
X
(b) The ELM Neuron Architecture
Km = exp( -At/Tm)
Rg exp( -At/T.)
8t = K, ② St-1 + Wg O Zt
Am, tanh(MLP up (1 Kim @ m2-1l))
m Kim ④ m A Kmm) à Am,
Yt = Wy m2
(a) A Cortical Neuron
(c) The ELM Neuron Equations
Figure 1: The biologically motivated Expressive Leaky Memory (ELM) neuron model. The architecture can be divided into the following components; the input current synapse dynamics, the integration mechanism dynamics, the leaky memory dynamics, and the output dynamics. a) Sketch of a biological cortical pyramidal neuron segmented into the analogous architectural components using the corresponding colors. b) Schematics of the ELM neuron architecture, component- accordingly. c) The ELM neuron equations, where æ, E Rd, is the input at time 4, A/ E IR+ the fictitious elapsed time in milliseconds between consecutive inputs Øt-1 and Ft
colored
Rdm.
memory units, 8 E Rdd the synapse currents (traces), Tm E RR om and T. E RR+
their respective
timescales in milliseconds, ws E R+ are synapse weights, wp the weights of Multilayer Perceptron (MLP) with /mlp hidden layers of size dmlp- Wy R.d. the output weights, À E R scaling factor for the delta memory Am; E Rdd, and y E Idd the output.
In this section, we discuss the design of the Expressive Leaky Memory (ELM) neuron, and its variant Branch-FLMM. Its architecture is engineered to capture sophisticated cortical neuron computations efficiently. Abstracting mechanistic neuronal implementation details away, we resort to an overall recurrent cell architecture with biologically motivated computational components. This design approach emphasizes conceptual over mechanistic insight into cortical neuron computations.
The current synapse dynamics. Neurons receive inputs their synapses in the form of sparse binary events known as spikes Kandel et al. (2000). While the Excitatory/Iniibioory synapse identity determines the sign of the input (always given), the positive synapse weights act simple input gating W. (learned in Branch-ELMM believed aid coincidence detection and synaptic information integration in neurons König al. (199).. This implementation is known as the surrent-based synapse dynamic Dayan & Abbott (2005).
The synaptic trace denotes filtered version of the input,
The memory unit dynamics. The state biological neuron may be characterized by diverse measurable quantities, such as their membrane voltage various ion/molecule concentrations (e.g. Ca+, mRNA, etc.), and their rate of decay over time (slow decay <-> large timescale), endowing them with sort of leaky memory Kandel et al. (2000); Dayan & Abbott (2005). However, which of these quantities are computationally relevant, how and where they interact, and on what timescale, remains topic of active debate Aru et al. (2020); Herz al. (2006) Almog & Korngreen (2016); Koch (1997); Chavlis & Poirazi (2021); Cavanagh et al. (2020); ( et al. (2016). Therefore, match biological neuron's computations, the surrogate model architecture needs to be expressive
3

---

Published as a conference paper at ICLR 2024
enough to accommodate large range of possibilities. In the ELM neuron, we achieve this by making the number of memory units dm a leamable),, setting it apart most other computational neuroscience models.
and equipping each of them with a Tm (always
The integration mechanism dynamics. This dynamic refers to how the synaptic input S; is inte- grated into the memory units Am; in analogy to the dendritic tree of a cortical neuron. While earlier perspectives suggested an integration process akin to linear summation Jolivet et al. (0008). newer studies advocate for complex nonlinear integration Almog & Korngreen (2016); Gidon et al. | (2020);
Larkum (2022), specifically proposing multi-layered ANNs more suitable descriptions Poirazi et al. (2003); J adi et al. (2014); Marino (202T); I) Jones & Kording (202T); Latropoulos et al. (2022); Joness
& Kording (2022); Hodassman et al. (2022), also backed by recent evidence of neuronal plasticity
beyond synapses] Losonczy et al. (2008); [ollmaat et al. (2009); Abbaham et al. (2019). Motivated by
this ongoing discussion, we choose parameterize the input integration using Multilayer Perceptron (MLP) (wy always learnable, with 4mip = 1 and dmip- = 2dmm) which can be used to explore the full range of hypothesized integration complexities, while offering straightforward way to quantify and ablate the ELM neuron integration complexity. In the Branch-ELM variant (for motivation see details in Section 4 and Figure 4) we extend the integration mechanism dynamics; before the MLP is applied, the synaptic input 8, E Rd is reduced to smaller number (dtrew) of branch activations, each computed as a sum over dorch neighboring synaptic inputs (with diree* * durch dg). In this variant the w. need to be learnable, as they are responsible for weighting the sum and cannot be absorbed in the MLP later. Despite the biological inspiration, the MLP and synapses are only intended to capture the neuron analogous plasticity and dendritic nonlinearity, and cannot give a mechanistic explanation of these phenomena in neuron. Finally, incorporating previous memory units m into the integration process, the ELM accommodate state-dependent synaptic integration and related computations Hodgkin & Huxley ) Casparini & Magee (2006): [icknell & Häusser (2021). and enables the relationships among memory units
be fully learnable. The range of the values is controlled by X, and the mixing of the proposal values by the parameter km (for details on parameters see Appendix Section (A). Crucially, our approach sidesteps the need for expert-designed and pre-determined differential equations typical in phenomenological neuron modeling.
The output dynamics. Spiking neurons emit their output spike at the axon hillock roughly when their membrane voltage crosses threshold Kandel et al. (2000). The ELM neuron's 's output is similarly based on its internal state m (using a linear readout layer wy), which rectified can be interpreted as the spike probability. For task compatibility, the output dimensionality is adjusted based on the respective dataset (not affecting neuron expressivity).
3 RELATED WORK
Accurately replicating the full spike-level neuron input/output (I/O) relationship of detailed biophysical neuron models at millisecond resolution in a computationally efficient manner presents a formidable challenge. However, addressing this synamics-learning task could yield valuable insights into neural mechanisms of expressivity, learning, and memory Durstewitz et al. (2023). The relative
scarcity of prior work on this subject can be partially attributed to the computational complexity of cortical neurons only recently garnering increased attention Tzilivaki et al. (2019): Beniaguey
ctt al. 202T); Larkum (002)): Poirazi & Papoutsi (2020). Additionally, traditional phenomenological
neuron models have primarily aimed to replicate specific computational phenomena of neurons or
networks Koch (1997): Izhikevich (2004); Herz et al. (2006). rather than the entire I/O relationship.
Phenomenological neuron modeling research on temporally or spatially less detailed I/O relationship of biophysical neurons has been primarily centered around the use of multi-layered ANN structures in analogy to the neurons dendritic tree Poirazi et al. (2003): Tzilivaki et al. (2019); Ujfalussy et al. (2018). Similarly, we parametrize the synaptic integration with an MLP, while crucially extending this modeling perspective in several ways. Drawing upon the principles of classical phenomenological modeling via differential equations Izhikevich| 2004); Dayan & Abbott (2005). our approach em- braces the recurrent nature of neurons. We further consider the significance of hidden states m beyond membrane voltage, as seen in prior works with predetermined variables Brette & Gerstner (2005); Gerstner et al. 2014)) This addition enables us flexibly investigate internal memory timescales Tm.
4

---

Published as a conference paper at ICLR 2024
Rophyuraa Model
b)
Valagg Pradiction
Spka Prediction
0 199944
0 0.902
Inpus Soakes
RMISE
Surregate Modell
0.65
ELM
ELM
LSTM
Performance
TCN
Threshold
0.994
10
1C'
10"
10
10"
10"
10'
10
Traiatbe Parameneers
Trainale Parameters
Figure 2: The ELM neuron a computationally efficient model of cortical neuron. a) detailed biophysical model of a layer 5 cortical pyramidal cell used to generate the NeuronIO dataset consisting of input spikes and output spikes and voltage. b) and c) Voltage and spike prediction performance of the respective surrogate models, produced using joint ablation of dm with dmip 2d for ELM models. Previously around 10M parameters were required to make accurate spike predictions using a TCN Beniaguev al. (202I). D an LSTM baseline able to do with 266K, and our ELM and Branch-ELM, neuron model require 53K and 8K respectively (3rd from left each), simultaneously achieving much better voltage prediction performance than the TCN, For comparison in terms of TP/FP Rate performance or FL.OPS cost see Fig. 4 or [ST respectiie. Additional comparisons to other phenomenological neuron models, such as LIF and ALIF, are provided in Table
Deep learning architectures for long sequence modeling have a shift towards the explicit incorporation of timescales for improved temporal processing, as observed in recent advancements in
RNNs, transformers, and space models Gu et al. (202I); Mahto al. (202I); Smmth et al. (2023); ;
Ma et al. (003). Such an explicit approach can be traced back to Leaky-RNNY Mozer (1991) Jaeger (2002); Kusupati& et al. (2018); Tallec & Ollivier (2018), which
convex combination of old
memory and updates, as done in ELM using Km. Whereas the classic time-varying memory decay me- diated by implicit timescales Tallec & Ollivier (2018), is known from classic gated RNNs like LSTM Hochreiter & Schmidhuber (1997) and GRUCCa et al. (0014). In contrast to complex gating mecha- nisms, time-varying implicit timescales, or sophisticated large multi- stalii--taeddd architectures, the ELM fea- tures a much simpler recurrent cell architecture only using constant explicit (trainable) timescales Tm for gating, putting the major emphasis on the input integration dynamics using a single powerful MLP.
4 EXPERIMENTS
In the experimental section of this work, we address three primary research questions. First, the ELM neuron accurately fit high-fidelity biophysical simulation with a small number of parameters? We detail this investigation in Section 1.1 Second, how can the ELM neuron effectively
integrate non-trivial temporal information? We explore this issue in Section 4.2 Third, what
are the computational limits of the ELM design? Discussed in Section 4.3 For training details,
hyper-parameters. and tuning recommendations, please refer to the Appendix Section B and Table
4.1 FITTING A COMPLEX BIOPHYSICAL CORTICAL NEURON MODEL'S S I/O RELATIONSHIP
The NeuronIO dataset primarily consists of simulated input-output (I/O) data for a complex biophysical layer 5 cortical pyramidal neuron model Hay et al. (201I). Input data features biologically inspired spiking patterns (1278 pre-synaptic spike channels featuring -1,1 0 as input), while output data comprises the model's 's somatic membrane voltage and output spikes (see Figure 2 (n) The dataset and related code are publicly available [Bentaguev et al.) (202T), and the models were trained using Binary Cross Entropy (BCF) for spike prediction and Mean Squared Error (MSE) for somatic voltage prediction, with equal weighting.
Our ELM neuron achieves better prediction of voltage and spikes than previously used architectures for any given number of trainable parameters (and compute). In particular, crosses the "sufficiently good" spike prediction performance threshold (0.991 AUC) proposed in (Beniaguev et al. 2021) by using 50K trainable parameters, which is around 200 improvement compared to the previous attempt (TCN) that required around 10M trainable parameters, and 6x improvement over LSTM baseline which requires around 266K parameters (see Figure -d- Overall, this result indicates that recurrent computation is an appropriate inductive bias for modeling cortical neurons.
5

---

Published as a conference paper at ICLR 2024
a
0.882
0.980
0.988
0.986
Default Performance Threshold
Fnlp
0.984
Fnlp
0.982
510
20
40
60
80
0
20
40
60
90
120
150
1
10
15
20
25
35
45
55
65
75
dm
dml
A
d)
0.994
0.982
0.988
0.996
0.984
0.87
0.87
B-ELM
0.982
10 25
10 25 50 100
10 25 minimal Tm (ms)
35
65
100
200
maximal Tm (ms)
constant Ta (ms)
dorch
Figure 3: The ELM neuron gives relevant neuroscientific insights. Ablations NeuronIO of different hyperparameters of an ELM neuron with AUC x 0.992, and a Branch-ELM with the same default hyperparameters. The number of removed divergent runs marked with 1 a) We find between 10 and 20 memory-like hidden states to be required for accurate predictions, much more than typical phenomenological models Izhikevich (2004) Dayan & Abbott (2005). b) Highly nonlinear integration of synaptic input is required, in line with recent neuroscientific findings Stuart & Spruston (2015); Jones Kording (2022): Larkum 2022). e) Allowing greater updates the memory units is beneficial (see Appendix Aj. d-f) Ablations of memory timescale (initialization and bounding) range or (constant) value, with the default range being Ims-150ms. Timescales around 25 seems be the most useful (matching the typical membrane timescale in the cortex Dayan & Abbott (2005)): however, lack can partially compensated by longer timescales. g) and h) Ablating the number of branches deree and number of synapses branch dorch of the Branch-ELA neuron.
We the fitted model to investigate how many memory units and which timescales needed to match the neuron closely. We find that around 20 memory units are required (Figure 33) with timescales that are allowed to reach least 25 ms (Figure 3t). While diversity of timescales. including long ones, seems to be favorable for accurate modeling (Figure 31 and 31, ELM with constant memory timescales around 25 ms performs sufficiently well (matching the typical membrane timescales in computational modeling Dayan & Abbott (2005). Figure Removing the hidden layer or decreasing the integration mechanism complexity significantly reduces performance (Figure pp. Allowing for more rapid memory updates through larger À is crucial (Figure 3P), possibly to match the fast internal dynamics of neurons around spike times or to absorb information faster into memory (more details in Appendix (A). When fitting the simple leaky-integratt and fire (LIF) or adaptive LIF, we reach better prediction with only a few memory units (Figure (S7).
How much nonlinearity is in the dendritic tree? Within the ELM architecture, allow for nonlinear interaction between any two synaptic inputs via the MLP. This flexibility might be necessary in cases where little is a priori known about the input structure. However, for matching the I/O of cortical neurons, knowledge of neuronal morphology and biophysical assumptions about linear- nontinear computations the dendritic tree might be exploited reduce the dimensionality of the input to the MLP (parameter-costly component with d, = 1278 inputs). Consequently, we modify the ELM neuron to include virtual branches along which the synaptic input is first reduced by a simple summation before further processing (see Figure For NeuronIO specifically, we assign the synaptic inputs to the branches in a moving window fashion (exploiting that the dataset, neighboring inputs were also typically neighboring synaptic contacts the same dendritic branch of the biophysical model). The window size controlled by the branch size dorch. and the stride size is derived from the number of branches diree ensure equally spaced sampling across the d, 1278 inputs.

---

Published as a conference paper at ICLR 2024
b)
C)
Brench-EUM with SK Porometers
Spkka Prediction
MLP
Membrare Volage
0-
Turgel Vollage
Predicted Vellage
Prodicted 8p Probubility
0.2
Sp ko Procobiity
FFgg a rp
1.40
-20
Torget Spiies (1455)
0.50
dendras
-40-
CUP
-00
LSTM
TCH
1900
4200
Time
Tranrble Partmntters
Figure 4: Coarse-graines modeling of synaptic integration significantly improves model effi- ciency. a) The integration mechanism dynamics of the ELM now computes the activity of individual dendritic branches as a simple the MLPw,, wy, where dizee is the number of branches and darch the number of synapses per branch. b) Accurate predictions using Branch-ELn neuron with 8104 parameters (for zoomed-in version with model dynamics see Figure S9p. c). The new Branch-Fl ML neuron improves on the EL neuron by about 7x in terms of parameter efficacy (same ELM typer-parameters. Differences in model quality are highlighted when examining True-Positive rate at low False-Positive rate.
of their respective synaptic inputs first before passing them to
Surprisingly, even with this strong simplification, the Branch-ELM neuron model can retain its predictive performance while requiring 8K trainable parameters (roughly 7x reduction over the vanilla ELM) to cross the performance threshold substantially. We also find that a combination of duree = 45, dirch = 65 and dm 15 still achieved over 0.9915 AUC with only 5329 trainable parameters, corroborating the assumption of the near-linear computation within dendritic branches and inviting future investigation of minimal required synaptic nonlinearity. However, this simplification utilizes the knowledge of morphology for modeling the neuron (in our case, exploiting the neighborhood in the dataset), violating it leads to deterioration of performance (Figure (S6). therefore for most of the task we use the vanilla ELM neuron.
4.2 EVALUATING TEMPORAL PROCESSING CAPABILITIES ON A BIO-INSPIRED TASK
The Spiking Heidelberg Digits (SHD) dataset comprises spike-encoded spoken digits (0-9) in German and English Cramer et al. (2020). The digits were encoded using 700 input channels in biologically inspired artificial cochlea. Fach channel represents narrow frequency band with the firing rate coding for the signal power in this band, resulting in an encoding that resembles the spectrogram of the spoken digit (see Figure 5l).
Motivated by recent findings that most neuromorphic benchmark datasets only require minimal tem- poral processing abilities Yang al. (2021). we introduce the SHD-Adding dataset by concatenating two uniformly and independently sampled SHD digits and setting the target to their sum (regardless of language) (see Figure 5aa). Solving this dataset necessitates identifying each digit timescale and computing their sum by integrating this information over a longer timescale, which in turn requires retaining the first digit in memory. Whether single cortical neurons can solve this exact task is unclear; however, it has been shown that even single neurons possibly encode and perform basic arithmetics in the medial temporal lobe Cantlon & Brannon (2007); Kutter al. (2018) 2022).
shorter
The ELM neuron solves the summing task across various temporal resolutions (determined by the bin size). As we vary the bin size from Ims (2000 bins in total, the maximal temporal detail and longest required memory retention) to 100 (20 bins in total, the minimal temporal detail and shortest memory retention), the ELM neuron's 's performance remains robust, degrading proportionally (see Figure3b--re this drop in performance is not shortcoming of the model itself, but a consequence of loss of temporal information through binning. Further, the performance is also maintained when testing on two held-out speakers, showing that the ELM neuron remains comparatively robust nuttof-distributinn. Due to vanishing gradients, the LSTM performs worse on this task, especially when the bin size is below 50. As the bin size increases, the LSTM's performance improves but does not surpass ELM because larger bin sizes likewise lead to the loss of crucial temporal details. This outcome underlines the importance of model's 's ability to integrate complex synaptic information effectively (see Figure 5lb) and the utility of longer neuron-internal timescales for learning long-range dependencies, potentially necessary for cortical neuron's operation (see Figure 5
the bin sizes
7

---

Published as a conference paper at ICLR 2024
a)
Spiking Heidelberg Digits
Adding
e)
Complexity by
Defautt Chence
600
Inpur chaneee
Accuracy
400
200
english Hight (8)
german 'eins' (1]
400
Bin
2
10 dww
20
2dm
50 100
b)
Train Set
Valid Sat
d)
(OCD) Test Sat
Timescales
0.8
Appuracy
Accuracy
066
0.5
B-ELM
LSTM
0.4
Chance
0.2
7800
5000
11 000
112200
(.150)
[1. 00000
1.79
0.0
10
30 100
20 50 100
2
20
100
Bin Size
Bin Size
Bin Siza
[mmmm Em.mad (ms)
Figure 5: The ELM neuron performs well on long and sparse data using longer timescales. a) Sample from the biologically motivated SHD-Adding dataset (based on Cramer et al. (2020), each dot is an input spike, and a vertical dashed line is guide for the eye indicating the separation of the two digits (not communicated to the network). b-d) The ELM neuron (186K params.) consistently outperforms classic LSTM (956K params.), especially for smaller bin sizes (meaning longer training samples), and SSSTM-performange cannot be fully recovered even for larger bin sizes. The Branch-ELM (67K params.) can retain performance for fine ehaiige binning at much reduced model size. Our LIF neuron based Spiking Neural Network (SNN) (51K params.) does not manage achieve good performance for any bin size, and training becomes unstable for long sequences. e) and f) Ablations using a bin size of 2ms with test set performance reported. e) Solving requires ELM neuron to have higher complexity than required for NeuronIO, and much larger models become unstable. Potentially network of smaller ELM neuron might be preferable. f) Longer Tm are erucial for extracting long-range dependencies. Possibly shorter ones might suffice in a ELM network, as longer timescales can emerge through dynamics Khajehabdollahi et al. (2023).
4.3 EVALUATING ON COMPLEX AND VERY LONG TEMPORAL DEPENDENCY TASKS
To test the extent and limits of the ELM neuron's 's ability extract complex long-range dependencies, we use the classic Long Range Arena (LRA) benchmark datasets Tay et al. (202T). It consists of classification tasks; three image-derived datasets Image, Pathfinder, and Pathfinder-X (images being converted to a grayscale pixel sequence), and three text-based datasets ListOps, Text, and Retrieval. Pixel and token sequences were encoded categorically, however, only considering 8 or 16 different grayscale levels for images. In particular, the Pathfinder-X X task is notoriously difficult, as the task to determine whether two dots are connected by a path in 128 x 128 image (~16k length).
Our results are summarized in Table I where we compare the ELM neuron against several strong baselines. The model most comparable to ours an LSTT with derived explicit gating bias initialization for effectively longer internal timescales Tallec & Ollivier (2018) (Chrono-LSTMM) When comparing the two, we find that both models consistently perform well, except on the Pathfinder-XTi task which only the ELM can reliably solve, albeit using longer тa than usual. The larger self-attention based models trail further behind, with both Transformer Vaswani et al. (2017) and Longformer (Beltayy al. (2020) completely failing to solve the Pathfinder-X X task Tay et al. (2021). Only the purpose-built architectures such as S4 Gu et al. (202T) and Mega Ma et al. (2023) (current SOTA) perform better, but they require many layers of processing and many
parameters than an ELM neuron, which uses 150 memory units and typically - 100k parameters.
Overall, the results suggest that the simple ELM neuron architecture is capable of reliably solving challenging tasks with very long temporal dependencies. Crucially, this required using memory timescales initialized according to the task length and highly nonlinear synaptic integration into 150
"Only once during hyper-paramene tuning did single Chrono-I.STM run achieve harely above chance
8

---

Published as a conference paper at ICLR 2024
Table The ELM neuron can solve challenging long-range sequence modeling tasks. The table shows the mean accuracy on Long Range Arena (LRA) Benchmark [Tay et al. (2021). The ELM neuron routinely scores higher than the Chrono-LSTM or the much larger Transformer or Longformer, and only the large multi-layered architectures tuned specifically for these tasks, such as S4 or Mega, outperform Surprisingly,
also the only non purpose-built model that can reliably solve the notoriously challenging 16K sample length Pathfinder-X task. Model sizes of the bottom baseline models are extracted from Gu et al. 2021]MMa et al. (2023TTa ct al. (2021). Training details and model hyper-parametor parameters detailed Appendix Section and Tables S2 and S3
Image Pathfinder Pathfinder-X X ListOps
Text Retrieval
ELM Neuron (ours) Chr.LLSTM Tallec & Ollivier 2018
49.62
71.15
77.29
46.77
80.3
84.93
46.09
70.79
FAIL
44.55
75.4
82.87
parameters
-100k
-100k
~100k
-100k
-200k
-150k
Transformer Vaswani et al. 22017
42.44
71.4
FAIL
36.37
64.27
57.46
Longformer Beltagy et al. 2020
42.22
69.71
FAIL
35.63
62.85
56.89
S4 Gu et al. (2021)
87.26
86.05
88.1
58.35
76.02
87.09
Mega Ma et (2023)
90.44
96.01
97.98
63.14
90.43
91.25
parameters
-600k
~600k
~600k
~600k
~600k
~600k
memory units (See Appendix B While the LRA benchmark revealed the single ELM neurons limits, we hypothesize that assembling ELM neurons into layered networks might give it enough processing capabilities to catch up with the deep models, but we leave this investigation to future work.
5 DISCUSSION
In this study, we introduced a biologically inspired recurrent cell, the Expressive Leaky Memory (ELM) neuron, and demonstrated its capability fit the full spike-level input/output mapping of high-fidelity biophysical neuron model (NeuronIO). Unlike previous works that achieved this fit with millions of parameters, variant of our model only requires a few thousand, thanks to the careful design of the architecture exploiting appropriate inductive biases. Furthermore, unlike existing neuron models, the ELM can effectively model neuron without making rigid assumptions about the number of memory states and their timescales, or the degree of nonlinearity in its synaptic integration.
We further scrutinized the implications and limitations of this design on various long-range dependency datasets, such as biologically-motival neuromorphic dataset (SHD-Adding) and some notoriously challenging ones from the machine learning literature (LRA). Leveraging slowly decaying memory units and highly nonlinear dendritic integration into multiple memory units, the ELM neuron was found to be quite competitive, in particular, compared to classic RNN architectures like the LSTM, notable feat considering its much simpler architecture and biological inspiration.
should be noted that despite its biological motivation, our model cannot give mechanistic explanations of neural computations biophysical models do, and that the task of fitting another neuron's 's I/O is not itself biologically relevant task for a neuron. Many biological implementation details are abstracted away favor of computational efficiency and conceptual insight, and the re- quired/recovered ELM neuron the model's subsequent use case. Furthermore, ELM is trained using BPTT, which is not considered biologically plausible in itself, and ELM learning likely relies on neuronal plasticity beyond synapses, the extent of which in biological neurons is still a subject of debate. Additionally. our neuron model dendrites are rudimentary (e.g., lacking apical vs basal distinction) and rely on oversampling synaptic inputs for performance so far. Finally, given the use of biologically implausible BPTT as a training technique and comparatively larger ELM neuron sizes on the later datasets, should be careful to directly draw conclusions about the learning capabilities of individual biological cortical neurons.
depend on what constitutes a sufficiently good fit and
Despite these caveats, the ELM's ability to efficiently fit cortical neuron I/O and its promising performance on machine learning tasks suggests that we are beginming to incorporate the inductive biases that drive the development of more intelligent systems. Future research focused on connecting smaller ELM neurons into larger networks could provide even more insights into the necessary and dispensable elements for building smarter machines.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was supported by Sofja Kovalevskaja Award from the Alexander von Humboldt Foun- dation. We acknowledge the support from the BMBF through the Tübingen AI Center (FKZ: 011S18039A and 011S (81388.3)B. AL, GM, and BS are members of the Machine Learning Cluster of Excellence, EXC number 2064/1 Project number 39072764. AS would like to thank the Max Planek Society for their generous financial support throughout the project. We would like to thank Antonio Orvieto for help with table SS6] We thank the Max Planck Computing and Data Facility (MPCDF) staff, as the majority of computations were performed on the HPC system Raven.
10

---

Published as a conference paper at ICLR 2024
REFERENCES
Wickliffe C CAbraham, Owen D Jones, and David L Glanzman. Is plasticity of synapses the mechanism of long-term memory storage? NPJ science of learning, 4(1):9, 2019.
Mara Almog and Alon Komgreen. Is realistic neuronal modeling realistic? Journal of neurophysiol- ogy, 11665):2180-222 2209, 2016.
Jaan Aru, Mototaka Suzuki, and Matthew E Larkum. Cellular mechanisms of conscious processing.
Trends in Cognitive Sciences, 24(10):814 825, 2020.
Guillaume Bellec, Darjan Salaj, Anand Subramoney, Robert Legenstein, and Wolfgang Maass. Long short-term memory and learning-to-lee to-learn in networks of spiking neurons. Advances in neural information processing systems, 31, 2018.
Iz Beltagy, Matthew Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05/150 2020.
David Beniaguev, Idan Segev, and Michael London. Single cortical neurons as deep artificial neural networks. Neuron, 109((7))2727-2733 2021.
Brendan A Bicknell and Michael Häusser. A synaptic learning rule for exploiting nonlinear dendritic computation. Neuron, 109(24):400144017, 2021.
Romain Brette and Wulfram Gerstner. Adaptive exponential integrate-and-fire model as an effective description of neuronal activity. Journal of neurophysiology. 94(5)333773342,, 2005.
Jessica F Cantlon and Elizabeth M Brannon. Basic Math in Monkeys and College Students. PLoS
Biology, 5(12),338, December 2007. ISSN 1545-7885. doi: 10. 10.1371/journ 1:177///ooall..bio00000022222 URL
https://dx.. www.spppo.....o org 10. 1371/ journall pbio o 0050328
Sean E Cavanagh, Laurence T Hunt, and Steven W Kennerley. A diversity of intrinsic timescales
underlie neural computations. Frontiers in Neural Circuits, 14:61 14:615626, 2020.
Spyridon Chavlis and Panayiota Poirazi. Drawing inspiration from biological dendrites to empower artificial neural networks. Current opinion in neurobiology, 70:1-10 2021.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprini arXiv: 1078, 2014.
Benjamin Cramer, Yannik Stradmann, Johannes Schemmel, and Friedemann Zenke. The heidelberg spiking data sets for the systematic evaluation of spiking neural networks. IEEE Transactions Neural Networks and Learning Systems, 2020.
Peter Dayan and Laurence F Abbott. Theoretical neuroscience: computational and mathematical modeling of neural systems. MIT press, 2005.
Daniel Durstewitz, Georgia Koppe, and Max Ingo Thurm. Reconstructing computational system dynamics from neural data with recurrent neural networks. Nature Reviews Neuroscience, 24(11): 693-710, 2023.
Sonia Gasparini and Jeffrey C Magee. State-dependent dendritic computation in hippocampal cal pyramidal neurons. Journal of Neuroscience, 26(7):2088 -1000 2006.
Wulfram Gerstner and Werner Kistler. Spiking neuron models: Single neurons, populations, plasticity. Cambridge university press, 2002.
Wulfram Gerstner, Werner M Kistler, Richard Naud, and Liam Paninski. Neuronal dynamics: From single neurons to networks and models of cognition. Cambridge University Press, 2014.
Albert Gidon, Timothy Adam Zolnik, Pawel Fidzinski, Felix Bolduan, Athanasia Papoutsi, Panayiota Poirazi, Martin Holtkamp. Imre Vida, and Matthew Evan Larkum. Dendritic action potentials and computation in human layer 2/3 cortical neurons. Science, (676647)) 2020.

---

Published as a conference paper at ICLR 2024
Julijana Gjorgjieva, Guillaume Drion, and Eve Marder. Computational implications of biophysical diversity and multiple timescales in neurons and synapses for circuit performance. Current opinion in neurobiology. 17:44-52, 2016.
André Grüning and Sander M Bohte. Spiking neural networks: Principles and challenges. In ESANN. Bruges, 2014.
Albert Gu, Karan Goel, and Christopher Re. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations, 2021.
Robert Gütig and Haim Sompolinsky. The tempotron: a neuron that learns spike timing-based decisions. Nature neuroscience, 9(3):420 428, 2006.
Jeff Hawkins and Subutai Ahmad. Why neurons have thousands of synapses, theory of sequence memory in neocortex. Frontiers in neural circuits, pp. 23, 2016.
Etay Hay, Sean Hill, Felix Schürmann, Henry Markram, and Idan Segev. Models of neocortical layer
5b pyramidal cells capturing a wide range of dendritic and perisomatic active properties. PLoS computational biology, 7(7)xx1022107, 2011.
Andreas VM Herz, Tim Gollisch, Christian K Machens, and Dieter Jaeger. Modeling single-neuron dynamics and computations: a balance of detail and abstraction. science, 314(5796):80-85, 2006.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Shiri Hodassman, Roni Vardi, Yael Tugendhaft, Amir Goldental, and Ido Kanter. Efficient dendritic learning an alternative to synaptic plasticity hypothesis. Scientific Reports, 12(1):6571, 2022.
Alan L Hodgkin and Andrew Huxley. quantitative description of membrane current and its application to conduction and excitation in nerve. The Journal of physiology, 117(4):500, 1952.
Anthony Holtmaat, Tobias Bonhoeffer, David K Chow, Jyoti Chuckowree, Vincenzo De Paola, Sonja Hofer, Mark Hübener, Tara Keck, Graham Knott, Wei-Chung A Lee, et al. Long- term, high-resolution imaging in the mouse neocortex through chronic cranial window. Nature protocols, 4(8): 111--11-14411 1144, 2009.
Georgios latropoulos, Johanni Brea, and Wulfram Gerstner. Kernel memory networks: A unifying framework for memory modeling. Advances in Neural Information Processing Systems, 35: 353328,333,, 2022.
Eugene M Izhikevich. Which model to use for cortical spiking neurons? IEEE transactions neural networks, 1563)10063 2004.
Monika Jadi, Bardia F Behabadi, Alon Poleg-Polsky, Jackie Schiller, and Bartlett W Mel. An augmented two-layer model captures nonlinear analog spatial integration effects in pyramidal neuron dendrites. Proceedings of the IEEE, 102(5):782-798, 2014.
Herbert Jaeger. Tutorial on training recurrent neural networks, covering bppt, rtrl, ekf and the" echo state network" approach. 2002.
Renaud Jolivet, Felix Schürmann, Thomas K Berger, Richard Naud, Wulfram Gerstner, and Arnd Roth. The quantitative single-neuron modeling competition. Biological cybernetics, 977444177- 2008.
Henna Simone Jones and Konrad Paul Kording. Might a single neuron solve interesting machine learning problems through successive computations on its dendritic tree? Neural Computation, 33 (6):1554-1571, 2021.
llenna Simone Jones and Konrad Paul Kording. Do biological constraints impair dendritic computa- tion? Neuroscience, 489:262-274, 2022.
Eric R Kandel, James H Schwartz, Thomas M Jessell, Steven Siegelbaum, A James Hudspeth, Sarah Mack, et al. Principles of neural science, volume 4. McGraw-hill New York, 2000.
12

---

Published as a conference paper at ICLR 2024
Sina Khajehabdollahi, Roxana Zeraati, Emmanouil Giannakakis, Tim Jakob Schäfer, Georg Martius, and Anna Levina. Emergent mechanisms for long timescales depend on training curriculum and affect performance in memory tasks. arXiv preprint wrXiv:23099119221 2023.
Ryota Kobayashi, Yasuhiro Tsubo, and Shigeru Shinomoto. Made-to-order spiking neuron model equipped with multi-timescale adaptive threshold. Frontiers in computational neuroscience, pp. 9, 2009.
Christof Koch. Computation and the single neuron. Nature, 385(6613):207-2111 1997.
Christof Koch and Idan Segev. The role of single neurons in information processing. Nature neuroscience, 3(1 1117117771 2000.
Peter König, Andreas K Engel, and Wolf Singer. Integrator or coincidence detector? the role of the cortical neuron revisited. Trends neurosciences, 19(4):130-137, 1996.
Aditya Kusupati, Manish Singh, Kush Bhatia, Ashish Kumar, Prateek Jain, and Manik Varma. Fastgrun: A fast, accurate, stable and tiny kilobyte sized gated recurrent neural network. Advances in neural information processing systems, 31, 2018.
Esther F. Kutter, Jan Bostroem, Christian E. Elger, Florian Mormann, and Andreas Nieder. Single Neurons in the Human Brain Encode Numbers. Neuron, 100(3):753-7614 .e4, November 2018. ISSN 08966273. doi: 10.10166j.neuron.. 2018 0.0.036....... URL com/ /
wettchubbelseeeee elsevier
Esther F. Kutter, Jan Boström, Christian E. Elger, Andreas Nieder, and Florian Mormann. Neuronal codes for arithmetic rule processing in the human brain. Current Biology, 1275):127-- March 2022. ISSN 09609822. doi: 10.1016/j.cuut 202..01.22222....... URL /ttps::/Iinkknghhs elsevierr 22leepawwr..oo....r.eeeee////662
Mathieu Lafourcade, Marie-Sophie H van der Goes, Dimitra Vardalaki, Norma Brown, Jakob Voigts, Dae Hee Yun, Minyoung Kim, Taeyun Ku, and Mark Harnett. Differential dendritic integration of long-range inputs in association cortex via subcellular changes in synaptic ampa-to-nmed receptor ratio. Neuron, 2022.
Matthew Larkum. Are dendrites conceptually useful? Neuroscience, 2022.
Attila Losonezy, Judit K Makara, and Jeffrey C Magee. Compartmentalized dendritic plasticity and input feature storage in neurons. Nature, 452 (52(7186):436-44,, 2008
Xuezhe Ma. Chunting Zhou, Xiang Kong. Junxian He, Liangke Gui, Graham Neubig, Jonathan May, and Luke Zettlemoyer. Mega: Moving average equipped gated attention. In The Eleventh International Conference Learning Representations, 2023. URL [ttl:stttt:://eeeeeeee
Wolfgang Maass. Networks of spiking neurons: the third generation of neural network models. Neural networks, 1659-161,,,, 1997.
Shivangi Mahto, Vy Ai Vo, Javier S. Turek, and Alexander Huth. Multi-timescale representation learning [lstm] language models. In International Conference Learning Representations, 2021. URL https: httpp///wwweeee.eeeeetrorraaaaaa
Guy Major, Matthew E Larkum, and Jackie Schiller. Active properties of neocortical pyramidal neuron dendrites. Annual review of neuroscience, 36:1-24, 2013.
Joseph Marino. Predictive coding, variational autoencoders, and biological connections. Neural Computation, 34(1):1-44, 2021.
Toviah Moldwin and Idan Segev. Perceptron learning and classification in a modeled cortical pyramidal cell. Frontiers in computational neuroscience, 14:33, 2020.
Michael Mozer. Induction of multiscale temporal structure. Advances in neural information processing systems, 4, 1991.
13

---

Published as a conference paper at ICLR 2024
Panayiota Poirazi and Athanasia Papoutsi. Illuminating dendritic function with computational models. Nature Reviews Neuroscience, 20(6):303-32,, 2020.
Panayiota Poirazi, Terrence Brannon, and Bartlett W Mel. Pyramidal neuron as two-layer neural network. Neuron, 37(6):98-9999, 2003.
Angus Silver. Neuronal arithmetic. Nature Reviews Neuroscience, 11(7):774489, 2010.
Jimmy T.H. Smith, Andrew Warrington, and Scott Linderman. Simplified state space layers for sequence modeling. In The Eleventh International Conference Learning Representations, 2023. URL
net. fo
Greg J Stuart and Nelson Spruston. Dendritic integration: 60 years of progress. Nature neuroscience, 18(12)1773-1171, 2015.
Corentin Tallec and Yann Ollivier. Can recurrent neural networks warp time? International Conference on Learning Representations, 2018.
Yuanhong Tang, Xingyu Zhang, Lingling An, Zhaofei Yu, and Jian K Liu. Diverse role of nmda receptors for dendritic integration of neural dynamics. PLOS Computational Biology, 19(4): e1011019, 2023.
Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena A benchmark for efficient transformers. In International Conference Learning Representations, 2021. URL https //openreview net foee
Alexandra Tzilivaki, George Kastellakis, and Panayiota Poirazi. Challenging the point neuron dogma: Fs basket cells as 2-stage nonlinear integrators. Nature communications, 10(1):3664, 2019.
Balázs B Ujfalussy, Judit K Makara, Máté Lengyel, and Tiago Branco. Global and multiplexed dendritic computations under in vivo-like conditions. Neuron, 100(3):579-592, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Qu Yang, Jibin Wu, and Haizhou Li. Rethinking benchmarks for neuromorphic learning algorithms. In 202/ International Joint Conference Neural Networks (IJCN), pp. 1-8. IEEE, 2021.
14

---

multiplicative factor À improved training performance. Interestingly, the À (1 Km,) term can then be seen effectively using À times faster input timescale than Tm This notion can be explicitly Published as a conference paper at ICLR 2024
Appendix
A IMPLEMENTATION DETAILS
All computations were performed using Python 3.9, and the following libraries were instrumental in our implementation: jax 0.3.14 (coupled with jaxlib 0.3.10 GPU back-end) for auto-grad
and auto-vectorization; equinox 0.8.0, jax-based neural network library; optax 0.1.3] jax-based
optimizer library; and pytorch 1.12.1 for data-loading. The accompanying git repository for the
project can be found under:
Table S1: The ELM neuron parameters and recommendations
Paraeerrr
NeuronIO
Recommendation
80
Wa
0.5
Wa (bounds)
>0
>=0 time sparse data (tune)
Ts
all 5ms
mo
dm
20
up to 250 (primary TUNE) evenly spaced on log scale
Tm (init technique)
equally spaced Ims, 100ms
Tm
(init range)
I ms, data length same as init range
Tm (bounds)
Oms, 500ms
MLP (nonlineary)
ReLU
MLP up (bias)
True
MLP Mpp
(init technique)
Kaiming Uniform
Lllp
dmip
2 dm 10
/
Wy (bias)
True Kaiming Uniform
Wy (init technique)
diree
45
x inp. 2-5x sampled (tune)
dbech
100
x space sparse data (tune)
For all experiments Wys Up and Tm were learnable, with w. crucially also learnable for Meanch-EMM,
Recommended default and tuning parameters: We primarily recommend ablating dm. case of small dm. exploring larger relative dmip might yield improved performance. For ELM with many small Tm or larger we have observed spome trainig instability; seemingly resolved through modified memory update (see below). The timescales Tm should generally be derived from the dataset length and the suspected timescales the temporal dependencies within the data; reasonably initialized, Icarnability doesn't 't seem to be necessary. Increasing тa may help enhance learning speed in case of temporally very sparse data. When using the Branch-EL.M M it important to sufficiently over-sample the input (more synapses than inputs) the selection. Additional recommendations are summarized in Table[S]
suspect significant expressivity stemming from w. doing
Timescale parametrization: The memory timescales Tm are directly learnable model parameters. They are constrained an apriori-specified bound, which is enforced through sigmoid rectification (the lower bound being > 0). By defining &mm exp( (.p(-) the resulting values are ensured to be within J0, for all valid Tm. irrespective of At. In preliminary experiments we observed increased training stability as opposed to directly learning Ki..
Improved implementation: We found that enabling greater changes in Am, by introducing
implemented by substituting HY = exp( - At/Tx) where Tx = Tm/A. The appoximation holds for Tm À 1, and only diverges for small Tm, where it allows for less stark m changes than the
15

---

Published as a conference paper at ICLR 2024
original implementation. In preliminary experiments found this modification to result in increased training stability for ELM neuron with many small Tm or larger (see Figure S4p.
Stable memory dynamics: A necessary prerequisite for solving long-range credit assignment problems in recurrent architectures is to address the vanishing and exploding gradient problem. In the ELM neuron we achieve this by enforcing a stable dynamic of the memory units, which are primarily responsible for carrying information forward through time. The combination of controlled (slow) decay using Km (addressing the vanishing gradient problem) and controlled (bounded) growth using the complementary 1 - Km (addressing the exploding gradients problem), couples input to forget timescales using À in a principled way such that m will be bounded if Am is bounded (e.g. ensured through tanh rectification), even if latter was generated using highly nonlinear MLP (see Figure (9 Note that in principle, this construction could also be applied to layer wise processing in depth, instead of in recurrent processing in time; however, we leave this experiment to future investigations.
The SNN implementation: The LIF neuron based SNN consisted of an output layer (N, = "number of classes") and recurrent layer (N, 500-N,)) with 20% of neurons being inhibitory. The spiking threshold was Ushr = I, and neurons were partially reset after firing using Ut+Ai = 8t 0.9 -phr- The membrane timescale was initialized to 25ms, and directly learnable like in the ELM neuron. Each of the 100 synaptic weights were initialized to W' = 0.3/ (33/sqr((100) and were rectified using Re All neurons were randomly connected on a synapse-by-synapse basis with 90% probability to the previous-layer, and 10% probability to the own-layer. The output neurons output was low-pass filtered using constant 20ms, before being used by the cross-entropy function.
B DATASETS AND TRAINING DETAILS
General training setup: For each task and dataset, the training dataset was deterministically split to create a consistent validation dataset, which was used for model selection during training and hyperparameter tuning. All models were trained using Backpropagation Through Time (BPTT), and used a cosine-decay learning-rate schedule across the entire training duration of the training run. All experiments were run on single A100-40GB or A100-80GB and ran less than 24h, Pathfinder-X being the notable exception.
For training and evaluation the dataset was pre-processed in accordance with Beniaguev et al. (202I), by capping somatic membrane voltage at -55mV and subtracting bias of -67.7m Additionally, the somatic membrane voltage was scaled by 1/10 for training. Training samples were 500ms long with Ims bin size and At. The ELM neuron used the default parameters from Table Si Models were trained using the Adam optimizer with an initial learning rate of -4 and a batch size of 8 for 30 epochs with 11 11,400 batches per epoch using Binary Cross Entropy (BCE) for spike prediction and Mean Squared Error (MSE) for somatic voltage prediction, with equal weighting. Loss was calculated after a 150ms burn-in period. The mean and standard deviation over three runs is reported, with Root Means Squared Error (RMSE) and Area Under the Receiver Operator Curve (AÚC) for voltage and spike prediction, respectively. The model aaper-parramter and training settings were chosen based on validation RMSE in preliminary ablations.
NeuronIO Dataset:
The digits were preprocessed by cutting them to uniform length of one second and binning the spikes using various bin sizes, the default being 2ms. The models were trained using the Adamax optimizer with initial learning rate of 53- and a batch size of for 70 epochs, with 814 or 2000 batches per epoch for SHD and SHD-Adding respectively, with At set to the bin size, and dropout probability set 0.5. The ELM and Franch-ELM, used A = 5, d... = 100 and Tm initialized evenly spaced between Ims and 150ms with bounds of Oms to 1000ms, whereas the L.TT used a hidden size of 250 and additional recurrent dropout of 0.3, while the SNN used learning rate of -e no dropout but a 11 regularization on the spikes of 0.01. The Branch-ELM over-sampled the input with diree = 100, dorch = 15 and used random synapse to branch assignment. Models were trained using the Cross-Entropy (CE) loss the last float output of the respective model, and the performance was reported as prediction Accuracy, with mean and standard deviation caleulated over five runs (chance performance being 1/19). The model hyper-paramene parameters and training settings were chosen based on validation. Accuracy in preliminary ablations.
Spiking Heidelberg Digits (Adding) Datasets:
16

---

Published as a conference paper at ICLR 2024
Long Range Arena Benchmark: The images- based datasets preprocessed by binning the individual grey-scale values (256 total) into 16, or for Pathfinder-X. X. different levels. For the text based datasets simple one-hot token encoding was used. For the Retrieval task with two-tower setup, the latent-dimension was 75 for both models. All models were trained using the Adam optimizer, with the ELM neuron using initial learning rate of 2e 4, and the the LSTT models working best with le 3. All were trained using Cross-Entropy (CE) loss on the last output of the model, and the performance is reported prediction Accuracy. The mean over three runs is reported for all experiments.
Table S2:
The ELM neuron configuration
Dataset
Input Dim
Batch Size
Epochs
Ta
Timescales
Image
384
300
5
logspace: 1 -103
Pathfinder
384
300
logspace: 1 -103
Pathfinder-X
8
768
300
150
logspace: 1 1-2 -2* 104
ListOps
25
384
150
5
logspace: 1-2 103
Text
169
384
150
logspace: -44 103
Retrieval
105
384
150
5
logspace: 1- -44 103
The ELM memory timescale bounds matched to the initialization range. The ELM used synapse tau 150ms on the Pathfinder-X dataset, which we observed to increase the learning speed significantly, however, smaller synapse also work (e.g. see Figure (S8). Otherwise, hyper-parameters were harmonized as much as possible, to demonstrate the robustness of the hyper- parameter choice.
Table S3: The Chrono-LSTM configuration
Dataset
Input Dim
Batch Size
Epochs
Timescales
Image
16
384
300
uniform: 1 2 10
Pathfinder
16
384
300
uniform: 1 1 -2 * 10a
Pathfinder-X
8
768
300
uniform: -2* 10
ListOps
25
384
150
uniform: 1-4 1-4 103
Text
169
384
150
uniform: 1-8 103
Retrieval
105
384
150
uniform:
10
The Chrono-LSTM param-eeretttr tuning primarily concerned the learning rates and hidden sizes, however learning rate of 1e (among the tested) and hidden size of 150 (max tested) consistently performed best, the exception being Pathfinder-X. where during tuning a single run using smaller hidden size performed slightly above change.
C ADDITIONAL RESULTS
For comparison, we fit the classic computational neuroscience models to the same NeuronIO data. Namely, we use the Leaky integrate-and-Ff and-fire (LIF) neuron model (with learnable membrane timescale, weights, and bias unit for linear integration), adaptive LIF (ALIF) that has additionally a single timescale of spike-frequency adaptation. To have fair comparison, model with only a single memory unit (and timescale) with linear synaptic integration (thus no MLP, additional parameters due to bias units, readout, and other implementation details). The LIF's internal membrane voltage was directly fit to the target voltage, and its output spike directly to the target spikes, using otherwise training methodology described in Section B
also fitted an ELM neuron
17

---

Published as a conference paper at ICLR 2024
Table S4: Evaluating Tiny Models on NeuronIO
Model
Soma RMSE
Spike AUC
# parameters
(classic) LIF
2.29
0,9117
1280
(classic) ALIF
2.29
0,9115
1281
(simplest) ELM
1.60
0.9255
1286
(d. 15) Branch-ELM
0.66
0.9915
5329
While all three models perform much worse than the reference performance threshold of 0.991AU'C, the ELM neuron performs slightly better, particularly for somatic prediction. This could result from the ELM neuron, similar to the underlying biophysical model, not enforcing an explicit hard memory reset when spiking. The noticeable lack performance difference between LIF and ALIF might be due to the fitted models consistently staying below the spiking threshold. Finally, the LIF and ALIF model's shortcoming accurately capturing the I/O relationship highlights the need for more flexible phenomenological neuron model.
a)
Voltage Prediction
b)
Spike Prediction
0.994
B-ELM
ELM
0.992
LSTM
RMSE
0.70
TGN
0.990
0.65
0.988
0.60
Threshold
0.984
10
10"
10
10"
10"
10" 19
FLOPs
FLOPs
Figure S1: The ELM neuron is a computationally efficient model of cortical neuron. Similar however, displaying FLOPs required do inference on a single sample. a) and b) Voltage and spike prediction performance of the respective surrogate models. While previous works required around 10M parameters make accurate spike predictions using a TCN Beniaguev et al. (2021), an LSTM baseline is able to do it with 266K, our ELM neuron model requires merely 53K, and our Branch-ELM neuron only humble 8K, simultaneously achieving much better voltage prediction performance than the TCN. A throughput optimized ELM neuron implementation can potentially reduce the required FLOPs even further.
figure to :: and
a)
Some Prediction
b)
Spika Predittion
C)
Spkke Prediction
0.40
0.85
0.8675
0.30
0.66
0,9850
0.50
0.9625
0.58
0.9600
10K
15K
6K
10K
Tisinable Paramoters
Trainable Parameters
Trainable Paromotors
Figure Branch-ELLM neuron training more stable with smaller X. Evaluating a Branch-FILM with same parametarseeters as in Figure except with À 5. The variability of test set performance reduced for most configurations, particularly for ones with a larger number trainable parameters (and memory units). Additionally, it allows for an improved max performance for the model type.
18

---

Published as a conference paper at ICLR 2024
a)
9)
0.994
0.992
0.980
AUC
0.988
0.986
Defautt
lopp
0.984
Performance Threshold
Imp
0.982
510
80
80
0
20
40
60
90
120
25
35
65
75
dm
dlp
dlowe
d)
h)
0.984
0.992
AUC
0.988
0.986
0.98
0.98
0.984
-0.7
0.97
B-ELM
0.982
50
maximal Tm (ms)
constant Tm (ms)
minimal T. (ms)
Obre
Figure S3: The ELM neuron gives relevant neuroscientifie insights. Ablations on NeuronIO of different hyperparameters an Branch- ELM neuron with AUC z 0.992, with default hyperparame- ters. The number of removed divergent runs marked with 1 a) We find above memory- like hidden states to be required for accurate predictions, much more than typical phenomenological models use Izhikevich| (2004); Dayan & Abbott (2005). b) Highly nonlinear integration of synaptic input is required, in line with recent neuroscientific findings Stuart & Spruston (2015); Jones & Kording (2022); Larkum (2022). c) Allowing greater updates the memory units is beneficial, however, too large ones increase training instability. d-f) Ablations of memory timescale (initialization and bounding) range or (constant) value, with the default range being Ims- 150ms. Timescales around 25ms-50ms seem to be the most useful (matching the typical membrane timescale in the cortex Dayan Abboti (2005)); however, lack can partially compensated by longer timescales, even better than by the vanilla ELM. g) and h) Ablating the number of branches dirce and number of synapses per branch dprch-
a)
d)
0.980
AUC
0.988
0.986
Default
0.984
Threshold
Performance
0.97
0.87
0.982
25
maximal Tm (ms)
constant T.n (ms)
A
Qm. dmp
2dm
Figure S4: Ablations with the improved ELM neuron implementation. Ablations on NeuronIO that previously displayed training instabilities are now stable throughout and more consistent when rerun with the updated implementation (see section A for details on the implementation) a) Rerun of experiment in S3d.. b) Rerun of experiment in 35. e) Rerun of experiment in SS3.. d) Rerun of experiment in 25. Furthermore, we reran experiment in 3p, and training was stable for lincar integration. Lastly,
experiment in 5e, however, did not observe significant improvements.
19

---

Published as a conference paper at ICLR 2024
a)
Train Set
b)
Valid Set
C)
(OOD) Test Set
1.0
0.8
Accuracy
0.6
ELM
0.4
LSTM
0.2
SNN
0.0
10 20
50
100
2
10 20
20 Bin Size
50 100
Bin Size
Bin Size
The ELM neuron performs well on typical neuromorphic datasets. The following results on the original Spiking Heidelberg Digits dataset Branch-ELM neuron reliably outperforms classic LSTM, especially for smaller bin sizes (meaning longer training samples), and LSTM PerM-pprformaccee cannot fully recovered even for larger bin sizes. Our LIF neuron based Spiking Neural Network (SNN), however, does manage to achieve decent performance for bin sizes around 20, in contrast to the SHD-Adding dataset (see Figure 5).
Figure S5:
dataset/Cramer Cramer al. (2020). a-c) The ELM and
Spike Prediction
0.992
0.990
0.988
AUC
0.986
Performance Threshold
Default
0.984
ReLU branch activation
Sigmoid branch activation
0.982
Synapse weight non leamable
Random synapse assignment
0.980
20
30
dm
Figure S6: Ablating the ELLM branch architecture on NeuronIO. Average test AUC displayed, using otherwise same parameeas and training setup in experiments in Figure 2 Exploiting the ordering in the synaptic input, and having learnable synapses is crucial for the Branch-ELM neuron model. Applying specific nonlinearity branch output slightly degrades performance.
20

---

Published as a conference paper at ICLR 2024
Figure S7: Fitting simplified neuron models using ELM. Accurately fitting the classic Leaky Integrate and Fire (LIF) model possible with only linear inte- gration (mmlp 0) and single memory unit (dm 1), therefore matching the ground truth architecture. When fitting both LIF and Adaptive-LIF with an ELM neu- ron with two memory units, the LIF fit yields better results; this is expected, as the ground truth ALIF ar- chitecture has an additional hidden state; the adaptive threshold. We suspect that neither LIF nor ALIF display chaotic dynamics, an overall higher AUC may be achieved than for BioPhys; the AUC plateau may display residual uncertainty inherent to the dataset.
Spike Prediction
LIF ALIF
0.988
BioPhys
0.995
(best)
0.994
0.992
0.988
dm
Figure S8: Ablating the T, parameter on Pathtinder-X, Av- erage and max test Accuracy displayed, using otherwise same hyper-parameters and training setup in experiments in Table While reliably achieving high performance requires larger Tax smaller timescales can achieve even higher performance, al- though less reliably and take longer to pick up the learning signal. The intermediate drop in performance for 75ms and 100ms could be an artifact due nontrivial interactions between and the cycle length (128) of the flattened image (128 x 128) data.
Accuracy
0.7
S2
0.6
Default
Max Accuracy
Ts
Table S5:
Ablation of ELM neuron d,m Pathfinder
dm
10
25
50
100
150
200
300
# params
822
3552
12k
44k
96k
168k
373k
ACC
57.83
62.95
66.31
69.31
71.54
72.98
71.85
In Table $5 we show the dependence of test accuracy the ELM neuron model size, using otherwise same training setup as before. Performance levels out around 72% accuracy at d,, = 150 (default), and decreases to 57% accuracy dm, 10. An S5 model (see Table S6 with likewise a single layer (and 186K parameters) outperformed with an ELM neuron with dm = 25 memory units (and 3.5K parameters).
Table S6: Ablation of $5 model layers Pathfinder
# layers
2
4
params
186k
37lk
742k
1.
ACC
58.19
78.69
91.63
95.33
In Table S6 provide an ablation of the S5 model Smith et al. (2023). close to state of the art model on the Long Range Arena, as reference of how such models perform with varying number parameters and layers. The reported training ablated per model size. The mean accuracy over three runs is reported. Note, the steep drop-off in performance between layers and one.
were used, with the learning rate individually
21

---

Published as a conference paper at ICLR 2024
Table S7: Ablation ELM neuron dropout probability on Pathfinder
dropout
0.1
0.2
0.3
0.4
0.5
Train ACC
0.8525
0.9092
0.9244
0.9149
0.8987
0.8763
Test ACC
0.7275
0.7631
0.811
0.823
0.815
0.8063
In Table S7 provide ablation of the ELM neuron training with varying dropout probability and longer training (1000 Epochs), but otherwise same hyper-parameters as before (dm. 150). Notice, how using a dropout 0.3 outperforms even layered S5 less than third its size. Given the significant gap between train and test accuracy, we expect that further improvements to the training setup by using weight decay, layer normalization, etc. (as routinely used the training of SOTA models on LRA) might improve convergence speed and generalization.
D ADDITIONAL VISUALIZATIONS
Input Excitatory Input
Synapee Index
500
b)
Membrane Vollage (mV)
440
Predicted Voltage
Target Vollage
Probabiity
Predicted Spike Probebilty
Targat Spikes (N=3)
0.25
0.00
Synepse Values
N-20
Memory Values
-5
250 800
1000
1250
1500 1750
2000
(ms)
Figure S9: Visualization of ELM neuron dynamics. Extended visualization of Figure 2p for an ELM neuron achieving around 0.992 AUC. a) The synaptic input as the neuron model receives it, with excitatory input (+1) marked in red, and inhibitory input (-1) marked in blue. b) The ELM neurons predictions and the ground-truth targets for regular sample from the data. Interestingly, the whole two seconds were inferred one go (similar to Figure 4p), which shows its generalization capabilities beyond the training horizon of 500ms. e) A random subset of 20 synapses receiving negative input will be deflected downwards. d) All 20 memory values are visualized. Some fluctuate rapidly than others, typically proportional their memory timescales.
visualized. Synapses
22

---

Published as a conference paper at ICLR 2024
E
RESULTS IN TABLE FORMAT
Table S8: The Branch-ELM on NeuronIO
Trainable Parameters
Voltage Prediction RMSE
Spike Prediction AUC
FLOPs
Memory Units
10
5.9K
16.04M
0.695 0.004
0.989 = 0.0001
15
6.9K
17.06M
0.652 # 0.0
0.9913 0.0007
20
8.1K
18.27M
0.638 0.003
0.9924 0.0002
25
9.5K
19.69M
0.639 = 0.013
0.9922 0.0006
30
11.1K
21.3M
0.628 0.002
0.9927 0.0001
40
14.9K
25.13M
0.633 = 0.014
0.9925 = 0.0005
50
19.5K
29.76M
0.629 0.012
0.9925 0.0009
Table 89: The ELM on NeuronIO
Trainable Parameters
Voltage Prediction RMSE
Spike Prediction AUC
FLOPs
Memory Units
10
26.06K
28.49M
0.709 + 0.001
0.9878 0.0004
15
39.39K
41.84M
0.662 + 0.003
0.99 + 0.0013
20
52.92K
55.38M
0.643 0.004
0.9918 0.0002
25
66.65K
69.13M
0.648 # 0.008
0.9918 + 0.0002
35
94.71K
97.22M
0.631 # 0.001
0.9922 0.0001
50
138.3K
140.85M
0.629 = 0.002
0.9925 + 0.0002
75
214.95K
217.58M
0.622 + 0.004
0.9927 0.0001
100
296.6K
299.3M
0.623 = 0.006
0.9927 + 0.0001
Table S10: The LSTM on NeuronIO
Trainable Parameters
Voltage Prediction RMSE
Spike Prediction AUC
FLOPs
Hidden Size
15
77.67K
77.77M
0.717 0.021
0.9864 + 0.0008
25
130.45K
130.61M
0.673 0.003
0.9898 + 0.0002
50
265.9K
266.23M
0.641 + 0.004
0.9913 + 0.0002
100
551.8K
552.45M
0.624 0.002
0.9923 0.0002
150
857.7K
858.68M
0.623 + 0.005
0.9925 + 0.0001
250
1529.5K
1531.13M
0.626 0.005
0.9925 0.0004
23

---

Published as a conference paper at ICLR 2024
Table $11: SHD Results
Train Accuracy
Valid Accuracy
Test Accuracy
Model
Bin Size
0.99 + 0.01
0.96 # 0.01
0.86 + 0.02
2
0.99 + 0.01
0.96 0.02
0.86 0.01
4
0.99 0.01
0.96
0.0
0.9
0.06
B-ELM
8
1.0
0.96
0.01
10
1.0
0.96
0.06
20
0.99 4.99
0.96
0.01
0.99 0.0
0.9
0.09
100
0.97
0.9
0.72
1.0 0.0
0.98 # 0.0
0.9
2
1.0 0.0
0.98 0.0
0.91 0.01
1.0
0.0
0.9
6
1.0
0.98
0.0
ELM
1.0
0.9
0.09
10
1.0
0.9
0.0
1.0
0.9
0.00
50
0.99
0.95
0.04
100
1.0
0.93
0.70
1
0,26 + 0.41
1.24:00.
0.21
0.29 + 0.34
0.27 0.33
0.3
4
0.94 + 0.07 0.99
0.91
0.78 + 0.09
0.96
0.83
LSTM
8
1.0
0.9
0.00
1.0
0.9
0.07
20
1.0
0.9
0.02
1.0
0.94
0.01
100
0.9
0.91
0.74
0.05
0.0
0.05 0.0
2
0.07 + 0.02
0.07
0.07
0.16+0.00
0.16
0.16
6
0.36
0.05
0.05
SNN
0.04
0.05
0.45
10
0.07
0,49 0.07
0.5
20
0.61 0.04
0.59
0.58 0.04
50
0.49
0.47 # 0.02
0.02
100
0.41 + 0.03
0.39 # 0.02
0.39 # 0.03
24

---

Published as a conference paper at ICLR 2024
Table S12: SHD-Adding Results
Train Accuracy
Valid Accuracy
Test Accuracy
Model
Bin Size
0.99 + 0.0
0,95 0.01
0.83 0.02
2
0.99 = 0.0
0.95 # 0.0
0.81 0.02
0
0.91
0.79
6
0.99 #
0.9
0.03
B-ELM
8
0
0.15
0.1
10
0.21
0.21
0.16
20
0.22
0.68
0.59
50
0.93
0.86
0.72
100
0.83 + 0.03
0.06
0.59
1.0
0
0.82 0.01
1.0
0.96
0
4
0.98 + 0.03
0.03
0.81
6
0.95 + 0.05
0.05
0.06
ELM
8
0.94 0.09
0.09
0.76
10
0.98
0.04
0.03
20
0.0
0.94
0.79
50
0.0
0.92
0.75
100
0.0
0.86
0.02
0.1 0.0
0.1 0.0
0.1
2
0.1
0.1
0.1
4
0.1
0.1
0.0
6
0.0
0.1
0.1
LSTM
8
0.02
0.11
0.1
10
0.21
0.19
0.15
20
0.33
0.35
0.6
0.99 0.0
0.87
0.02
100
0.98 + 0.0
0.75 + 0.01
0.55
10
0.09 + 0.01
0.09 + 0.01
0.08 + 0.01
20
0.08 + 0.01
0.08 # 0.01
0.08 0.01
SNN
50
0.1 # 0.01
0.1
0.1
100
0.09 0.0
0.09 0.0
0.09 0.0
25


Reviews:
[{"review_id": "uwiP7IjHsu", "reviewer": "Reviewer_YDMm", "summary": "The authors propose a phenomenological neuron model called the Expressive Leaky Memory (ELM) neuron that uses various biologically inspired features. Specifically, it has separate synapse and memory dynamics and an integration mechanism defined by a learnt MLP. The model also allows learning of the various time constants. The authors demonstrate that this model is able to fit the input-output relationship in a dataset generated from a detailed biophysical model. Moreover, the authors show that this model can perform long-range dependency modelling better than vanilla transformers and a LSTM-based recurrent model.", "questions": "## Questions\n\n- Would this neuron be able to model synapse dynamics such as short-term plasticity (Tsodyks et al. 1998)?\n\n## Suggestions\n\n- Spike frequency adaptation for spiking neurons was proposed in (Bellec et al. 2018) rather than (Bellec et al. 2020).\n\n### Minor:\n\n- In the abstract, \"exploiting a few such slowly decaying...\" sentence reads very odd.\n- sentence above beginning of Sec. 4: \"puting the major emphesis\" has typos. That paragraph is very hard to read.\n\n(Bellec et al. 2018) Bellec, G., Salaj, D., Subramoney, A., Legenstein, R., and Maass, W. (2018). Long short-term memory and Learning-to-learn in networks of spiking neurons. In Advances in Neural Information Processing Systems 31, pp. 787–797.\n\n(Tsodyks et al. 1998) Tsodyks M, Pawelzik K, Markram H. Neural networks with dynamic synapses. Neural Comput 10: 821– 835, 1998.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 4, "strengths": "- The paper is well motivated, and the model uses abstract simplifications to represent known biological details. This provides a relatively parsimonious but abstract model to model biological neurons, which is a very interesting approach and novel to my knowledge.\n- The fact that this model, even though abstract and motivated by biology, still performs well in long range arena is very interesting.\n- The description of related work is very comprehensive.\n- The authors include a good discussion of the potential shortcomings of the model.\n- Overall, the quality and significance of this work is high.", "weaknesses": "- There are major clarity issues in the paper. Many aspects of the model and notation are unexplained (e.g. $\\lambda$, $1-\\kappa_m$ in Fig. 1(c)). The explanation of Branch-ELM comes much later, even though it's referred to multiple times before that, which makes it very hard to read. The role of $w_s$ is also not clear at all (the given explanation on Pg. 3 doesn't help).\n- The behaviour of Branch-ELM is unclear -- if the input is shuffled, does it affect performance? Since it depends on the local window to group inputs?\n- It's a bit odd that ELM doesn't perform well for short sequences (large bin size) as seen in Fig. 5 whereas LSTM does. The performance of ELM for short sequences could be explored more, since it sounds like that might be a major shortcoming of the model.\n- It is not clear how the number of parameters for the various cases were chosen.\n- I think exploring the multi-layer case would have made the paper much stronger. It's also not clear if this was avoided because of the computational constraints, since for mid-size LSTMs at least, multi-layer networks still fall very much in the computationally tractable regime.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2c423ciTlo", "reviewer": "Reviewer_27bc", "summary": "This paper proposes artificial neural network models that incorporates important inductive bias (i.e. leaky memory dynamics, nonlinear synaptic integration) inspired from biological cortical neurons. The model is aimed to achieve two types of goals, one is to match the spike-level dynamics of pyramidal neuron, and the second is evaluated on bio-inspired tasks to evaluate temporal integration. For evaluation, they compare the model with others SOTA baselines (SNN, LSTM, Transformer) are evaluated on multiple biological inspired datasets and long sequence modeling task. It shows the benefits of efficiency in parameterization, and comparable or better performance compared to SOTA models. Meanwhile, the hyper-parameter tuning studies show overlap with previous literatures.", "questions": "1. What other critical components might be helpful to improve ELM model accuracy?\n2. After applying sparse regularization or quantization to S4 and Mega to match the number of parameters to ELM, how much accuracy drop they will have?\n3. Is the model able to be trained with other biological plausible learning rules instead of BPTT? How they might end up with different parameters?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-motivated, and take the reductionist view to minimize the parameters from a more detailed modeling for cortical neurons, and aim to address the computational efficiency needs of standard models.\n2. Solid evaluations on multiple biological inspired datasets, and compared with multiple SOTA models, and follow by multiple hyperparameter tuning studies. \n3. The biological realism side shows interesting overlaps with previous neuroscience literatures.\n4. The results show the model is capable of achieving better accuracy with efficiency and fewer parameters than traditional deep model LSTM. The finding about simplification does not sacrifice the predictive performance is valuable.\n5. The paper is well-written, and organized in a good structure.", "weaknesses": "1. This paper is aimed to balance the trade-off between fidelity, efficiency and biological realism. It did a fair job while still failed to capture some important aspects. For example, using MLP sacrifices the interpretability and biological realism to compensate accuracy. On the other hand, the model still sacrifices the accuracy and has a big performance gap when compared to SOTA models (S4 and Mega). \n2. Scalability of the method: as scaling law plays a big role for improving transformer's predictivity, one concern is that transformer might perform better with increasing number of parameters. However, it might not be the same for ELM. As firstly shown in Fig 3, the accuracy quickly saturates with simply increasing $d_m$ and $d_{mlp}$, and not able to get further improved to minimize the performance gap between ELM and Mega.\n3. Only one ML task is evaluated, more evaluations and benchmarks needed to demonstrate contributions in addressing long-range sequence modeling.\n4. As shown in Table S1, large number of hyper-parameters still needed in advance or be tuned based on prior knowledge from neuroscience literatures.\n5. The efficiency side might be over-claimed, as Table 1 shows ELM still requires 100k-200k parameters?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qySp5lyAls", "reviewer": "Reviewer_nAEZ", "summary": "The authors introduced the Expressive Leaky Memory (ELM) neuron model, a bio-inspired model of a cortical neuron. It incorporates slowly decaying memory-like hidden states and a two-layered nonlinear integration of synaptic input. They showed that this model is able not only to capture the input-output mappings of cortical neurons efficiently but also to solve long-range dependency problems.", "questions": "- A paragraph defining Branch-ELM would be necessary. Can you elaborate more on the intuition behind this variant? How does it work? When is it more suitable compared to the vanilla ELM? Why is it important to over-sample the input in this case? \n\n- The term \"fixed trainable\" time constant is confusing. Is it a single tau value learned for each neuron that does not change after training?\n\n- In Appendix B, the general training setup is detailed (batch size of 8, etc.). However, later on, different hyperparameters are used for the datasets. It is not clear where this general training setup was used.\n\n- Does Figure S6 a) show any specific patterns?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The ELM model achieves a notable decrease in trainable parameters compared to temporal convolutional networks for simulating cortical neurons.\n- The paper is well-written and easy to follow, particularly in the second section, where the design of the ELM neurons is explained.\n- The selected experiments are suitable and effectively demonstrated the model's capabilities.", "weaknesses": "- The main text doesn't define the Branch-ELM variant. See questions.\n- Minor training details need some clarifications. See questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xu25XpFGIU", "reviewer": "Reviewer_8X2o", "summary": "In this work, the authors propose a model of neurons called Expressive Leaky Memory (ELM) (and Branch ELM later introduced in the paper). The goal is to build a model with fewer parameters than existing models that can still learn to replicate the input/output relationship of some pyramidal neurons. The authors also extend the evaluation to other tasks that are less neuroscience-inspired but aim at evaluating the ability of the model to capture long-term dependencies in the dataset. \n\nThe model is reminiscent of an LSTM with significant modifications leading to the possibility of better “synaptic” integration and longer timescales. \n\nThe authors do not aim to build a biologically plausible model of pyramidal neurons but to replicate some of its capabilities using less learnable parameters than existing literature, mostly other LSTMs and a TCN.", "questions": "Based on the weaknesses highlighted above, I would suggest the authors address how they would provide results that more closely align with the claims. And provide some improvement on the various fronts that I have highlighted.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper's strengths are the following. \n\nThe paper is well-written and the task at hand is very interesting. As pointed out by the authors, not many models exist that can effectively represent the IO relationship of pyramidal neurons, and being able to do that could lead to improvement for various machine learning problems. \n\nThe presentation of numerical simulations and the exploration of machine learning tasks beyond neuroscience are commendable.", "weaknesses": "The weakness of the paper. \n\nAlthough the paper argues that “merely” a few thousand learnable parameters is a good way to replicate what a single pyramidal neuron performs, it seems to still be a lot to me. In my understanding, the paper doesn’t address the task that the pyramidal neurons would perform, and thus, it is not clear such a model would be able to learn the same input/output relationship if only inputs were given. \n\nVarious papers, including recent papers on predictive coding and canonical correlation analysis, have suggested possible tasks performed by some pyramidal neurons. It would be interesting to see if such a model of neurons could learn the same IO based on this learning paradigm.\n\nAs pointed out, the synaptic plasticity and how the model's training is performed are not addressed. \n\nRegarding the numerical experiments, the model is only compared to other “modern” ML models and not those that are more biologically plausible. I would like to see where simple pyramidal neuron models perform on spiking methods such as the one presented in this work. Compared to LSTMs and TCNs, it is not merely as relevant. \n\nThe term inductive bias is often used in papers when trying to characterize the “reasonable” choice of architecture as being inspired by biological facts. I can appreciate that the term is currently “hype,” but it is possibly misleading when considered at a machine learning conference where inductive bias means something else. The term hand-engineered would be more appropriate here. \n\nAlthough the paper is well written, the choice of wording in many places is unscientific, e.g., “struggle to learn at all,” “merely a few thousand,” “merely meant to capture,” “degrading only gracefully,” and more. I would appreciate it if the authors paid more attention to possible bias in the writing of the paper. \n\nThe introduction of the Branch-ELM in Figure 4 appears too late in the paper. We are introduced to the concept at the same time as the results of the experiments when it would have been better to have it in Section 2 when the ELM is introduced. \n\n\nIn conclusion, I believe that the paper has some value, but I am not certain that it is well-suited for the venue. I also believe the paper doesn’t deliver on the claims made in the abstract or at the end of the introduction. I believe that what is achieved in the paper is overstated. Also, I would like to get the authors to write the paper with less biased words, as was mentioned in the weakness section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)