LMUFormer: Low Complexity Yet Powerful Spiking Model With Legendre Memory Units
Authors: Zeyu Liu
Gourav Datta
Anni Li
Peter Anthony Beerel
Venue: ICLR 2024 poster (2024)
Keywords: Legendre Memory Unit
Spiking Neural Network
Recurrent Neural Network
Primary area: general machine learning (i.e., none of the above)
TL;DR: In this paper, we propose a novel model named LMUFormer, which enhances the LMU with patch embedding, channel mixer and SNN, and is low-computational, and powerful in the STC and NLP fields.

Abstract: Transformer models have demonstrated high accuracy in numerous applications but have high complexity and lack sequential processing capability making them ill-suited for many streaming applications at the edge where devices are heavily resource-constrained. Thus motivated, many researchers have proposed reformulating the transformer models as RNN modules which modify the self-attention computation with explicit states. However, these approaches often incur significant performance degradation.
The ultimate goal is to develop a model that has the following properties: parallel training, streaming and low-cost inference, and state-of-the-art (SOTA) performance. In this paper, we propose a new direction to achieve this goal. We show how architectural modifications to a fully-sequential recurrent model can help push its performance toward Transformer models while retaining its sequential processing capability. Specifically, inspired by the recent success of Legendre Memory Units (LMU) in sequence learning tasks, we propose LMUFormer, which augments the LMU with convolutional patch embedding and convolutional channel mixer. 
Moreover, we present a spiking version of this architecture, which introduces the benefit of states within the patch embedding and channel mixer modules while simultaneously reducing the computing complexity. 
We evaluated our architectures on multiple sequence datasets. Of particular note is our performance on the Speech Commands V2 dataset (35 classes). In comparison to SOTA transformer-based models within the ANN domain, our LMUFormer demonstrates comparable performance while necessitating a remarkable $70\times$ reduction in parameters and a substantial $140\times$ decrement in FLOPs. Furthermore, when benchmarked against extant low-complexity SNN variants, our model establishes a new SOTA with an accuracy of 96.12\%. 
Additionally, owing to our model's proficiency in real-time data processing, we are able to achieve a 32.03\% reduction in sequence length, all while incurring an inconsequential decline in performance.

Published as a conference paper at ICLR 2024
LMUFORMER: Low COMPLEXITY YET POWERFUL SPIKING MODEL WITH LEGENDRE MEMORY UNITS
Zeyu Liu", Gourav Datta** Anni Li, Peter A. Beerel University of Southern Californi,, Los Angeles, CA, USA (1 uuzey, gdatta, annili, pali,pabeeeee 1 edu 'Equally contributing authors CCurrently employed at Amazon Inc.
}@scc
ABSTRACT
Transformer models have demonstrated high accuracy in numerous applications but have high complexity and lack sequential processing capability making them ill-suited for many streaming applications at the edge where devices are heavily resource-constrained. Thus motivated, many researchers have proposed reformu- lating the transformer models as RNN modules which modify the self-attention computation with explicit states. However, these approaches often incur signifi- cant performance degradation. The ultimate goal is to develop a model that has the following properties: parallel training, streaming and low-cost inference, and state-of-the art (SOTA) performance. In this paper, we propose new direction to achieve this goal. We show how architectural modifications to a fully-sequential recurrent model can help push its performance toward Transformer models while retaining its sequential processing capability. Specifically, inspired by the recent success of Legendre Memory Units (LMU) in sequence learning tasks, we propose LMUFormer, which augments the LMU with convolutional patch embedding and convolutional channel mixer. Moreover, chitecture, which introduces the benefit of states within the patch embedding and channel mixer modules while simultaneously reducing the computing complexity. We evaluated our architectures on multiple sequence datasets. Of particular note is our performance the Speech Commands V2 dataset (35 classes). In compari- son to SOTA transformer-baa based models within the ANN domain, our LMUFormer demonstrates comparable performance while necessitating remarkable 53x re- duction in parameters and substantial 65x decrement in FLOPs. Furthermore, when benchmarked against extant low-complexity SNN variants, our model es- tablishes a new SOTA with accuracy of 96. 92%..%. Additionally, owing to our model's proficiency in real-time data processing, we are able to achieve a 32.03% reduction in sequence length, all while incurring inconsequential decline in performance. Our code is publicly available here!
present spiking version of this ar-
INTRODUCTION
In recent years, Transformers (Vaswani et al. 2017) have become the most prevalent deep learning models for many applications. The global attention mechanism enables Transformers capture long-distance dependencies in input data. Moreover, their state-less structure makes their training easily parallelizable, helping them perform well with large-scale datasets, leading to SOTA accu- racy in a wide range of applications, including Natural Language Processing (NLP) (Brown et al.) 2020 Touvron et al. 2023). Computer Vision (CV) (Dosovitskiy et al.) 2020; Liu al.) 2021). I), and spoken term classification (STC) (uuati et al.) 2020) Gong et al. 2021). However, their self- attention mechanism imposes compute and memory complexity that is quadratic in the sequence length N. Moreover, the nature of the self-attention computation prevents the underlying hardware from performing much of the computation until after the entire input sequence is available.
In contrast, Recurrent Neural Network (RNN) models designed to process data sequentially and their complexity is only (NV making them an attractive complexity alternative to transform- ers. However, RNN models traditionally have higher training time as the training procedure must

---

Published as a conference paper at ICLR 2024
accommodate the long sequence of dependencies within the model, making parallelization more dif- ficult. Moreover, unlike transformers, RNN models like LSTMs (Hochreiter & Schmidhuber, 1997) and GRUs (Cho et al. 2014) can only leverage past and current information and traditionally suffer from forgetting due to having a limited memory horizon. Recently, however, Legendre Memory Voelker al. (219)) have been proposed that have the ability remember information over a theoretically infinite time horizon. Benefiting from this property, LMU-based models outperform many RNN models while still underperforming compared Transformer alternatives. We hypoth- esize that the remaining performance gap between LMUs and Transformers is not only because Transformers benefit from future information, but also because they possess more complex network structures, such as the self-attention computation, and thus have higher representational capacity.
In an attempt to explore this hypothesis, we propose a novel sequential network architecture, dubbed LMUFormer, that augments the LMU module with convolutional patch embedding and convolu- tional channel mixers. Importantly, our convolutional patch embedding only interacts with neigh- boring input samples and our convolutional channel mixers and the final classifier only operate on the current state of the LMU, Therefore, our LMUFormer model can process data sequentially. Moreover, present spiking version of this architecture which extends the benefits of state that is explicit in the LMU implicitly to the patch embedding and channel mixing structures to improve ac- curacy while simultaneously further enabling complexity reduction. In summary, this paper makes the following contributions:
We propose a novel architecture LMUFormer which outperforms most existing RNN mod- els with similar complexity on wide range of sequence learning tasks. On evaluat- ing the Speech Commands dataset, our LMUFormer model, when benchmarked against transformer-basee models with competitive performance levels, manifests a significant re- duction - accounting for 53x fewer parameters and 65x diminished FLOPs.
We further devised a spiking variant of the LMUFormer that not only achieves state-of- the-art (SOTA) performance within the realm of SNN models on the Speech Commands dataset but also presents comparable results on the Long Range Arena benchmark.
Attributable to the real-time data processing capabilities inherent to our models, we evalu- ate the performance of the model when different proportions of sequences are received. We demonstrate that our model can achieve 99% of the original performance while reducing 32.03% sequence length.
PRELIMINARIES
Legendre Memory Unit & Parallel Training: The Legendre Memory Unit (LMU) is a ory cell proposed by Voelker et al. (019) designed to efficiently capture and represent temporal dependencies in sequential data rooted in the mathematical properties of Legendre polynomials. Mathematically, the LMU is based on two state-space matrices (A, B) that approximate a linear transfer function in continuous time as follows
mtt) = Am(() + Bu(t).
(1)
where u(t) represents the input signal and m(t represents the memory state vector. This continuous- time system is mapped onto discrete time with time step f as follows
m|t] = Ámii - 1] Bult],
(2)
where Á and B are the discretized version of Á and B.
To better support parallel training, we adopt the same module design as Chilkuri & Eliasmith (2021) get u(]]] from z((], which is the input signal of the LMU cell in time step, as follows
ul f Act,, W [t( + b.))
(3)
and obtain the output of the LMU module as described as follows
olt] Act, ( m| Wa + bo),
where W., W and W. are the learnable parameter matrices. Note that Act,, and Act, repre- sent the activation functions. Therefore, the module only has recurrent connection and can be
2

---

Published as a conference paper at ICLR 2024
regarded as a linear time- invariant (LTT) system which can be solved in non-iterative way which is the key for parallel training. We also adopt fast Fourier transform (FFT) as Chilkuri & Eliasmith (2221) to further reduce the training complexity to ( log2(N) de), where N is the length of the sequence and de is the feature dimension of the input
Spiking Neural Network: As the third-generation neural network (Maass 1997, SNNS have gained lot of attention for their potential for higher energy efficiency than traditional ANNs. Mim- icking the behavior of biological neurons which communicate using brief electrical pulses, SNNs use binary "spikes" to process and transmit information. However, to convert sequential multi-bit input data such as audio text into spikes, a coding scheme is required. Rate coding (Lee al. 2016) and direct coding (Wu et al. (219)) are two of the most commonly used methods. Rate coding translates the input sequence into a spike train across T time steps, with the spike count correlating to input magnitude and spikes following a Poisson distribution (Lee et al. 2020p. In direct coding, in contrast, the multi-bit inputs are fed to the first convolution layer in the models and spikes are used only in subsequent portions of the network. If the dataset does not contain temporal information, the outputs of the 1s convolution layer need to be repeated T time steps, and converted to binary or multi-bit spikes through spiking neurons.
In this paper, we use direct coding as well as the Leaky antegrate-and-Fin (LIF) Maass 1997
neuron model. The behavior of the LIF neuron is described as follows:
f1, if of -f of of -
ui = -ui- +
(0,
(5)
otherwise
uf represents the membrane potential tensor of the 7th layer at the 8th time step, À is a leak factor that varies between 0 and 1, wy is the weight connecting layers (1-1) and 7, of is the spike output
of the (1-1)th layer at the hhh time step, 1fth is the threshold that is kept constant for layer
3 RELATED WORK
SNN for sequential learning: In the domain of computer vision. various SNN models have been proposed, serving purposes ranging from image recognition (Fang et al.] [021] Meng et al., 2022)
to object detection (Kim et al., 2020 Barchid et al.) 2023), and these models have demonstrably achieved competitive performance relative their ANN counterparts. Notwithstanding, the explo-
ration of SNNs in sequential tasks, such as text classification and spoken term classification, remains
notably limited, with scant literature (Lotfi Rezaabad & Vishwanath 2020 Datta et al. 2023 Lv
et al. 2022) addressing these applications.
Since the inception of the work Katharopoulos et al. (020)) that pro- posed linear transformers, many researchers focused on modifying the self-attention mechanism to make transformer- model have lower costs during inference. Linformer (Wang et al. (0200) in- corporates fixed-size linear projections, facilitating streamlined approximation of attention across longer sequences. Similarly. Nystromformer (Xiong et al.] (eeerreessages the Nyström method to realize an approximation of the standard self-attention mechanism, achieving linear complexity. In a distinct approach, Peng et al. (2023) proposed a novel model architecture, Receptance Weighted Key Value (RWKV). Notably, the time-mixing block within RWKV can arguably be interpreted as computing the product of the matrices K and V. subsequently multiplied by R, suggesting its founda- tional roots in the Transformer paradigm. Building upon this foundation, [Zhu al. (2023) design the SpikeGPT based on RWKV which has demonstrated competitive performance across both Natural Language Generation (NI.) and Natural Language Understanding (NLU) tasks.
Recurrent Transformers:
The MLP-Mixer (Tolstikhin et al. 2 has emerged as novel paradigm in the field of computer vision, diverging from the conventional approaches of Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs). The MLP-Mixer leverages multilayer perceptrons (MLPs) for both spatial and channel-wise data processing. By sequentially employing independent MLPs, it avoids convolution and attention mechanisms, resulting in streamlined architecture. Inspired by this, we followed a similar framework to design our LMUFormer but further improved it to handle data with temporal information.
MLP-Mixer:
3

---

Published as a conference paper at ICLR 2024
Conv Patch
LMU Block
Conv Channel Mixer
FC
Embedding
Figure 1: The overall network architectures of LMUFormer.
Convid BNId
Layer
Convid BN1d
Layer
Convld BN1d
Layer
1
6
Convid
Layer
6
BN1d SN +Convld
Layer
5 6
10
(a)
(b)
Figure 2: (a) The structure of the convolutional patch embedding: (b) The delay analysis of the convolutional patch embedding.
PROPOSED SPIKING LMUFORMER
We propose spiking LMUFormer as shown in Fig. which can be trained in parallel and process data sequentially during inference. Our model is primarily based on LMU and augmented with several techniques from Transformers. Specifically, embedding. LMU block, convolutional channel mixers, and a final fully-connected layer as the classification head. This section first introduces the structure of the patch embedding and channel mixers and then elaborates on how to convert the model to an efficient spiking version.
model consists of convolutional patch
4.1 CONVOLUTIONAL PATCH EMBEDDING
Embedding is the first component of Transformer models and is responsible for converting words/tokens into continuous vectors. For NLP tasks, we we simply use the built-in PyTorch embedding module to model the embedding functionality. For STC tasks, design two types of Convolutional patch embedding. The first is inspired by Gong et al. (2021) and Xiao et al. (2021) which used series of 2D convolutional layers split the spectrogram into sequence of N 16x 16 patches. Specifically, we create a new dimension for the embedding features and regard the spectrogram as an image perform convolution. This structure mixes temporal and frequency information ex- tract superior feature representations. However, it also yields significant algorithmic delays when processing sequential input data, hindering real-time processing.
To mitigate this concern, we propose to use several ID convolutional layers with a kernel size of for patch embedding, as we illustrate in Fig, aa). When the kernel size is the patch embedding layer is ideal for processing sequential data with no algorithmic latency but also suffers from poor performance. We empirically observed that convolution kernel of size 3 can dramatically improve the performance since it can capture information about nearby samples and, as analyzed in Fig. 2bb), the added latency is negligible. As an example, the blue lines in Fig. ( represent that to obtain the first output in layer 5, i.e., the output of the patch embedding, input sample arrives. After that, for each input sample fed into the model, the patch embedding can calculate output, shown by the red lines. Thus, in real-tin time after a delay only samples, which is negligible compared to the total number of samples (typically at least few hundreds) present in any sequence learning task.
need to wait until the 9th
network process the input sequence
4

---

Published as a conference paper at ICLR 2024
4.2 CONVOLUTIONAL CHANNEL MIXER
Inspired by Tolstikhin et al. 2021). we expect the LMU module to be primarily responsible for mixing information across tokens due to the presence of the state variables. To effectively capture the information across the different feature channels, we augment the LMU module with a channel mixer module. that consists of BN and non- non-linear activation layers followed by 1x1 convolutional layers as described in Eq. 6 The first activation layer employs Gaussian Error Linear Units (GELU) as introduced by Hendryyks & Gimpel (016). while the subsequent activation layer utilizes the Rectified Linear Units (ReLU) (Krizhevsky et al.) 2012). Since the convolutional layer, BN layer, and non-linear activation layer do not interact with temporal information, we omit the time notation T in Eq. 6 for simplicity. We also add residual connection (He et al. 2016) between the input X; and the output X, of the convolutional channel mixer and refer to this enhanced structure as the Conv Channel Mixer Block.
X = Conuu (GELU X. = Con
(6)
4.3 SPIKING LMUFORMER NETWORK STRUCTURE
To further improve the energy efficiency, we propose a spiking version of the LMUFormer, named spiking LMUFormer.
For NLP tasks, we use the batch normalization and the spiking LIF (SN) layer in the first LMU block convert the floating point values spikes, that results in sparsity and accumulate-only operations. For STC tasks, we use the first convolution layer together with its followed BN and SN layer in the patch embedding as the direct encoder. Inspired by Yao al., 2023), we also adjust the positions of residual shortcuts to make sure the output is the addition between two floating point numbers rather than the spikes.
As depicted in Fig. 3] the spiking LMU block encompasses several layers, structured sequentially as: a BN layer, followed by a SN layer, succeeded by the core spiking LMU cell, which is then coupled with Conv layer, and concluded with another BN layer. Notably, the integration strategy we employed capitalizes on the inherent temporal dynamics of both the LMU and the SNN. Considering the concurrent updates of both the memory and hidden states in the LMU at every discrete time step, in conjunction with the analogous updates of the membrane potentials and spikes within the SNN, we have devised a merged process optimize the overall operational efficiency of their integration.
In particular, we feed the input spikes [t([] into a convolutional layer and a BN layer to get the input signal of the memory cell.
Ult = (N(ooccl((xsstt)
(7)
Mcanwhile, they are also regarded as the input of spike neurons. As adopt the Leaky Integrate- and-Fire (LIF) neuron (Maass) 1997) model, the update of the membrane potential Un[t] at time step t is described below (Wu et al. 2018)
Un|t = Uvlt-1]++
1| (UTT) (Uv[t- - 1] Upeset))
where Uv(l - 1] means the membrane potential after emitting spikes time step (- 1, is the time constant, and Ureaet represents the reset potential. The firing of spikes is then described as follows
Us| = F(Un[t] U(A)
where Us[4] means the spikes of the input signal of the memory cell and F denotes the firing func- tion, which outputs I when the input is greater than O, and otherwise. Finally, to reset the mem- brane potential Uy at time step t we use the following equation
Uv [t] = U (t) (1 Us(t)) + Uteset Us[t].
(10)
After obtaining Ug [t] and the memory vector M tt 1] in the last time step, we can formulate the update of M [4] as shown below
MI4 A. MMst 1] B Bss()
(11)
5

---

Published as a conference paper at ICLR 2024
[5[c] Conv1d
BN1d
BN1d
SN
U[c]]
SN
LMU
Us[t]
Ms[t
Convid
A Ms (t 1 B Us[t]
M[[]
BN1d
SN
Ms[t]]
Figure 3: The network architectures of spiking LMU block and the details within the spiking LMU cell.
where A and B are the discretized space metric from Voelker et al. (2019). To simplify the notation,
we use the function Spiking Neuron (N)-) as the abbreviation of Eqs. to 100 Therefore, as shown below
Ms[|] = SN(M))) O S
(12)
we first feed the [t] into the spiking neurons to get the spikes of the memory state at time step t. Then concatenate Mg it] and (t(t) along the feature dimension and feed them into the ID Conv layer, BN layer, and SN to get the output spikes of the LMU block at time step f.
EXPERIMENTS
5.1 PERMUTED SEQUENTIAL MNIST
To evaluate the performance of our models on vision task, we use the permuted sequential MNIST (psM- NIST) dataset (Le et al. 2015). Un-
Model
SNN
Ace (%)
LSTM Voelker et al. 2019
N
89.86
LMU
Voelker et al. 2019
97.15
like the original MNIST dataset (Le- Cun et al. 1998) showing each 28 x 28 grayscale image as whole, psM- NIST presents the pixels of each im- age as sequence, typically in per- muted order. It results in this task being much more challenging than the original task since the models need to remember and integrate formation over time, i.c.. have the ability to handle longerange dependencies. PsMNIST contains 10 classes, involving handwritten digits from 0 to and we split 50k images as training dataset, 10k images validation dataset, and the rest 10k images test images.
HiPPO-Leay LegS (Gu et 2020
98.3
PLMU (Chilkuri & Elrasmith
2021
98.49
F.Muorrer
N
98.55
LMUFormer
Y
97.92
Table 1: Performance comparison of our LMUFormer mod- els and previous works on PsMNIST dataset.
As shown in Table S our models not only outperform all the RNN models but also surpass the existing LMU-based models. Since the PsMNIST dataset is relatively small, directly applying LMUFormer model could easily lead to overfitting, so we used a simplified version. We only use Cony layer as well as BN layer the patch embedding and remove the channel mixer.

---

Published as a conference paper at ICLR 2024
Table Performance comparison of our LMUFormer models and previous works on Speech Com- mands dataset. (Spikformer" and Spike-driven ViT represent models in which we replace our LMU module with Spiking Self Attention Zhou et al. (2022) and Spike-Driven Self-Attention Yao et al. (023) modules, respectively.)
Sequential Inference
Paraliel Training
Model
SNN
Accuracy (%)
RNN (Bittar & Garner 2022
Yes
No
No
92.09
Attention RNN (De Andrade et al.
2018
No
No
No
93.9
liBRU (Bittar & Garner
2022
Yes
No
No
95.06
Res15 (Vygon Mikhay lovskiy
2021
Yes
Yes
No
97.00
KWT2 (Berg et al (021)
No
Yes
No
97.74
AST (Gong et al. 2021
No
Yes
No
98.11
LIF (Bittar & Garner (0022)
Yes
Yes
Yes
83.03
SFA (Sala) et al. 2021)
Yes
No
Yes
91.21
Spikformer (Zhou et al.
2022)
No
Yes
Yes
93.38
RadLIF (Bittar & Garner
2022
Yes
No
Yes
94.51
Spike-driven VII" Yao al.
2023
No
Yes
Yes
94.85
LMUFormer
Yes
Yes
No
96.53
LMUFormer (with states)
Yes
Yes
No
96.92
Spiking LMUFormer
Yes
Yes
Yes
96.12
5.2 SPEECH RECOGNITION
We use the Speech Commands V2 dataset (Warden 2018) contains 105,829 audio files of 35 differ- ent words. Each recording lasts up to second and the sampling rate is 16KHz. We use the official dataset splits, where the training, validation, and test dataset contains 84,843, 9,981, and 11,005 samples, respectively. We chose the hardest classification task with 35 classes to test our models.
As shown in Table there still about 1.5% drop in test accuracy between our model with the SOTA transformer- model. our model has fewer parameters (1.622 Million 86.933 Million, 86.933 0083/1.622 11.622 = 53.66), fewer FLOPPs (1.89 x 10 vs. 1.24 x 1010 124/1.89 Rs 65.61) and can handle data sequentially. Additionally, evaluations indicate that the LMUFormer (with states) achieves noteworthy performance of 96.92%. This empirical outcome substantiates our initial hypothesis that the integration of states within patch embedding and the channel mixer block can enhance model performance, In pursuit of unparalleled energy efficiency, we convert this model to the 1-bit SNN model, i.e., the spiking LMUFormer, which surpasses all the contemporary SNN models with mere 3.09 x 10? theoretical synaptic operations (SOPs) (Merolla et al. 2014).
LMUFormer model outperforms the existing RNN models. Although
In addition, we also added the result of two experiments for two transformer- based SNN models, Spikeformer (Zhou et al. 2022 and Spike-driven VIT (Yao al. 2023). that achieve the SOTA per- formance in the vision tasks. Because they do provide the official version of their models for the SC dataset, we tested new models in which use their proposed structure of self-attention as a substitute for the LMU module in our model. It is important to note that the Spikformer (Zhou et al.) 2022) model has a dramatic drop compared with the non-spiking transformer-bee based AST (Gong et al.) 202I). This suggests that if restrict the patch embedding to only mix the information from neighboring samples, the power of the original global attention in transformer models will be degraded. On the contrary, the Spike-driven ViT Yaoo et al. 2023) can preserve a higher accuracy which indicates its linear attention more robust to the degrada- tion of the features extracted by the patch embedding. Moreover, our spiking LMUFormer achieves
Model
Params. (M)
OPs (G)
AST (Gong et al. 2021
86.93
12.4
Former
1.62
0.189
Spiking LMUFormer
1.69
0.0309
Table 3:
Comparison of the number of model parameters and operational counts between our LMUFormer models and AST model. (OPs refers to FLOPs in ANNs, and SOPs in SNNs)

---

Published as a conference paper at ICLR 2024
Table 4: Performance comparison of our LMUFormer models and previous works on Long Range
Arena (LRA) Dataset.
ListOps
Retrieval
Pathfinder (1K)
Model
Text(4K)
Image( (K)
Avg
(2K)
(4K)
S4
58.35
76.02
87.09
87.26
86.05
80.48
Linear Trans.
16.13
63.90
53.09
42.34
75.30
50.55
Linformer
35.70
53.94
52.27
38.56
76.34
51.36
Transformer
36.37
64.27
57.46
42,44
71.40
54.39
BigBird
36.05
64.02
59.29
40.83
74.87
55.01
Nystromformer
37.15
65,52
79.56
41.58
70,94
58,95
LMUFormer 34.43
68.27
78.65
54.16
69.9
61.08
Spiking LMUFormer 37.30
65.80
79.76
55.65
72.68
62.24
Table 5: Ablation study on different patch embeddings and channel mixers in the LMUFormer
model in Speech Commands V2 dataset. ("Naive LMU" means only using the LMU cell as the token mixer while "LMU Block" means adding a Conv layer and BN layer after the LMU cell. "Conv PE" means our convolutional patch embedding and "Conv CM" means convolutional channel mixer.)
Model type
Patch Embedding
Token Mixer
Channel Mixer
Acc. (%)
A (%)
Non-spiking
Identity
Naive LMU
Identity
76.17
0.0
Non-spiking
One-layer Conv
Naive LMU
Identity
89.28
13.11
Non-spiking
One-layer Conv
LMU Block
Identity
89.8
0.52
Non-spiking
Conv PE
LMU Block
Identity
95.57
5.77
Non-spiking
Conv PE
LMU Block
Conv CM
96.53
0.96
Spiking
One-layer Conv
Naive LMU
Identity
83.78
0.0
Spiking
One-layer Conv
LMU Block
Identity
87.88
4.1
Spiking
Conv PE
LMU Block
Identity
95.20
7.32
Spiking
Conv PE
LMU Block
Conv CM
96.12
0.92
further 1.27% increase in accuracy, demonstrating that our model is superior to the traditional transformer-basee spiking models when handling information with temporal information.
5.3 LONG RANGE ARENA BENCHMARK
To showcase the capability of our LMUFormer with longer tokens, we utilize the Long Range Arena (LRA) benchmark introduced by Tay al. (2020). The LRA benchmark assesses machine learn- ing models on long-context understanding through subtasks including text classification, document retrieval, image classification, pathfinder, and listops, highlighting their performance domains. Adhering to the evaluation protocol from Tay et al. (2020), which establishes specific train/test splits, we report the classification accuracy for each task and present an aggregated perfor- mance measure across all tasks.
various
We conducted comparative study involving five bransformer-based models: the vanilla trans- former (Vaswani et al. 2017), Linear Trans. (Katharopoulos et al.) (020), Linformer (Wang et al., 2020), BigBird (aheer al. 2020) and Nystromformer (Xiong et al. (021). And the results of the first four models are from Tay et al. (2020). As detailed in Table 4 there is still significant gap the performance between our models with the recent $4-based models (Gu et al.) 202T), but our LMUFormer excels against these munsformer-basees models in almost all tasks except Pathfinder. Furthermore, our spiking variant LMUFrrmrr surpasses them the ListOps and Retrieval tasks.
Notably, our spiking LMUFormer is the inaugural SNN model to not only demonstrate comparable performance on the LRA dataset but also to outshine the majority of its transformer based coun- terparts. Intriguingly, the spiking LMUFormer outperforms the regular LMUFormer by average margin 0.53%, suggesting the potential of SNN models to harness their inherent temporal dynam- ics for superior performance with long sequences.
8

---

Published as a conference paper at ICLR 2024
5.4 ABLATION STUDY
We first conducted extensive ablation studies to show the impact of the different patch embeddings and channel mixers in our proposed LMUFormer model on the final performance. All the models have one final linear layer that acts the classification head and are trained and tested the Speech Command V2 dataset with 35 classes. As shown in Table [ a naive LMU module with pLMU, can only achieve 76.17% accuracy while simply adding convolution layer can boost the result to 89.28%. Although the performance of the non-spiking model with our LMU block only slightly better than the model with naive LMU, our LMU block significantly improves the performance of the spiking model by 4.1% (an increase from 83.78% to 87. 7. Moreover, the results show our convolutional patch embedding can further improve the accuracy of the models compared to models with a simple convolution layer. Finally, using convolutional channel mixer, our models achieve a final accuracy 96.53% for the non-spiking model and 96. 12% for the spiking model.
classification head, same
5.5 RECURRENT PROCESSING
To more clearly showcase our model's profi- ciency with sequential data, we evaluated the trained spiking LMUFormer on the Speech Command V2 test dataset, gradually increas- ing the sequence length from to its full length of 128 samples. As shown in Fig. the in- crease in the model's accuracy with the num- ber of samples first accelerates continuously and then levels off. This indicates that af- ter certain number of samples have been ob- tained, the model has been able to predict the results almost correctly. Specifically, our model achieves 99% of its original performance while getting a 32.03% (1 87/128) reduction in the sequence length, yielding results of 95.17% compared to 96. 12% which is even higher than the spikformer model (93.38%) which utilizes all the information from the whole sequence.
87.0
(128,0.9612)
(63.0.8692)
80
Acc. (%6
(36.04977)
40
20
20
40
60
80
100
120
Sequence
Figure 4: Plot of test accuracy as the number of samples in the sequence increases.
6 CONCLUSIONS
The transformer model excels across various domains, particularly with large-scale datasets, owing to its exceptional performance and straightforward parallel training. However, its O(N2) complexity during inference, coupled with the necessity to acquire the complete sequence before calculating the self-attention, limits its utility in resource-constrainee environments demanding low latency. Hence, we introduce the LMUFormer and Spiking LMUFormer models, uniquely designed to switch be- tween parallel and recurrent forwarding when training and testing. Through extensive experiments, we demonstrate that our non-spiking model achieves close to the performance of state-of-the. art (SOTA) models while markedly reducing the model size by approximately 53x and computational complexity by roughly 65x. Furthermore, our Spiking LMUFormer achieves SOTA performance, registering a notable accuracy of 96.12%% among prevailing SNN models evaluated on the Speech Commands V2 dataset, without compromising on efficiency. It is our aspiration that this contribution serves catalyst for further exploration and advancements in model design for SNNs, particularly in the domain of sequence learning tasks.
7 ACKNOWLEDGEMENTTS
This work is supported by a gift funding from Intel Labs. We would also like to thank Sumit Bam Shrestha and Timothy Shea from Intel for insightful discussions on LMUFormer.

---

Published as a conference paper at ICLR 2024
REFERENCES
Sami Barchid, José Mennesson, Jason Eshraghian, Chaabane Djéraba, and Mohammed Bennamoun. Spiking neural networks for frame-based and event-based single object localization. Neurocom- puting, pp. 126805, 2023.
Axel Berg, Mark O' D'Connor and Miguel Tairum Cruz. Keyword transformer: A self-attention model for keyword spotting. arXiv preprint arXiv:2104.00769 2021.
Alexandre Bittar and Philip N Garer. A surrogate gradient spiking baseline for speech command recognition. Frontiers in Neuroscience, 16:865897, 2022. 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Narsimha Reddy Chilkuri and Chris Eliasmith. Parallelizing legendre memory unit training. In International Conference Machine Learning, pp. 1898-1907. PMLR, 2021.
Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv: 1409.12092555 1259, 2014.
Gourav Datta, Haoqin Deng, Robert Aviles, Zeyu Liu, and Peter A. Beerel. Bridging the gap be- tween spiking neural networks & Istms for latency & energy efficiency. In 2023 IEEE/ACM International Symposium on Low Power Electronics and Design (ISLPED), pp. 1-6, 2023. doi: 10.
Douglas Coimbra De Andrade, Sabato Leo, Martin Loesener Da Silva Viana, and Christoph Bernkopf. A neural attention model for speech command recognition. arXiv preprint 1808::880.08880 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenhorn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint MXXV:20100.11929 2020.
Wei Fang, Zhaofei Yu, Yanqi Chen, Tiejun Huang, Timothée Masquelier, and Yonghong Tian. Deep residual learning in spiking neural networks. Advances in Neural Information Processing Systems, 34:21056-21069, 2021.
Yuan Gong, Yu-An Chung, and James Glass. Ast: Audio spectrogram transformer. arXiv preprint arXiv:2104.01777 2021.
Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. Hippo: Recurrent memory with optimal polynomial projections. Advances in neural information processing systems, 33: 1474-1487, 2020.
Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces, arXiv preprint arX0v:2111.00396, 2021.
Anmol Gulati, James Qin, Chung-Cheng Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al. Conformer: for speech recognition. arXiv preprint 00X10:2005.08100, 2020.
transformer
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Dan Hendryeks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint
3XXi:1606.004155 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
10

---

Published as a conference paper at ICLR 2024
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers mnn:: Fast autoregressive transformers with linear attention. In International conference on ma- chine learning, pp. 1156-5165, PMLR, 2020.
Seijoon Kim. Seongsik Park, Byunggook Na, and Sungroh Yoon. Spiking-yolo: spiking neural network for energy-efficient object detection. In Proceedings of the AAAI conference artificial intelligence, volume 34, pp. 11270-11277, 2020. 2020)
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprini arXiv: 1412.6980 2014.
Alex Krizhevsky, Nya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Quoc V. Le, Navdeep Jaitly, and Geoffrey E. Hinton. A simple way to initialize recurrent networks of rectified linear units. ArXiv, abs/1504.00941,, 2015. URL https://api.
semant wtt.aaiicccccaa..oo/oooorooooo.e
Yann LeCun, Corinna Cortes, and Christopher J.C. Burges. The mnist database of handwritten digits, 1998. URL http:://yamp moman.leccmpp lecun. com cowp://////////mm/////////
Chankyu Lee, Syed Shakib Sarwar, Priyadarshini Panda, Gopalakrishnan Srinivasan, and Kaushik Roy. Enabling spike-based backpropagation for training deep neural network architectures. Fron- tiers in neuroscience, pp. 119, 2020.
Jun Haeng Lee, Tobi Delbruck, and Michael Pfeiffer. Training deep spiking neural networks using backpropagation. Frontiers in neuroscience, 10:508, 2016.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision, pp. 10012-10022, 2021.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
arXiv:1711.05101 2017.
Ali Lotfi Rezaabad and Sriram Vishwanath. Long short-term memory spiking networks and their applications. In International Conference on Neuromorphic Systems 2020, pp. 1-9, 2020.
Changze Lv, Jianhan Xu, and Xiaoqing Zheng. Spiking convolutional neural networks for text classification. In The Eleventh International Conference Learning Representations, 2022.
Wolfgang Maass. Networks of spiking neurons: the third generation of neural network models.
Neural networks, 10(9):1659-1671, 1997.
Qingyan Meng, Mingqing Xiao, Shen Yan, Yisen Wang, Zhouchen Lin, and Zhi- Quan Luo. Training high-performance low-latency spiking neural networks by differentiation spike representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
12444-12453, 2022.
Paul Merolla, John Arthur, Rodrigo Alvarez-Icaza, Andrew S Cassidy, Jun Sawada, Filipp Akopyan, Bryan L Jackson, Nabil Imam, Chen Guo, Yutaka Nakamura, et al. A million spiking- neuron integrated circuit with a scalable communication network and interface. Science, 345 (6197):668 0001666666666.066 2014.
Bo Peng. Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, Kranthi Kiran GV, et al. Rwkv: Reinventing mns for the transformer era. arXiv preprint arXiv:2305.13042 2023.
Darjan Salaj. Anand Subramoney, Ceca Kraisnikovie, Guillaume Bellec, Robert Legenstein, and Wolfgang Maass. Spike frequency adaptation supports network computations on temporally dis- persed information. Elife, 10:e65459, 2021.
11

---

Published as a conference paper at ICLR 2024
Leslie N Smith and Nicholay Topin. Super-convergence:: Very fast training of neural networks using large learning rates. In Artificial intelligence and machine learning for multi multi-domain operations applications, volume 11006, pp. 366- SPIE, 2019.
Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient transformers. arXiv preprint arXiv:2011.04000 2020.
Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Un- terthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlp-mixer: An all-mlp architecture for vision. Advances in neural information processing systems, 34:24261- 24272, 2021.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Aaron Voelker, Ivana Kajić, and Chris Eliasmith. Legendre memory units: Continuous-time repre- sentation in recurrent neural networks. Advances in neural information processing systems, 32, 2019.
Roman Vygon and Nikolay Mikhaylovskiy. Learning efficient representations for keyword spotting with triplet loss. In Speech and Computer: 23rd International Conference, SPECOM 2021, St. Petersburg, Russia, September 27-30, 2021, Proceedings 23, pp. 773-785. Springer, 2021.
Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04476 04768, 2020.
Pete Warden. Speech commands: dataset for limited-vocabulary speech recognition. arXiv preprint
2018.
Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, and Luping Shi. Spatio-temporal backpropagation for
training high-performance spiking neural networks. Frontiers in neuroscience, 12:331, 2018.
Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, Yuan Xie, and Luping Shi. Direct training for spiking neural networks: Faster, larger, better. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 1311-1318, 2019.
Tete Xiao, Mannat Singh, Eric Mintun, Trevor Darrell, Piotr Dollár, and Ross Girshick. Early convolutions help transformers see better. Advances in neural information processing systems, 34:30392-30400, 2021.
Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. Nyströmformer: A nyström-based algorithm for approximating self-attention. 2021.
Man Yao, Jiakui Hu, Zhaokun Zhou, Li Yuan, Yonghong Tian, Bo Xu, and Guoqi Li. Spike-driven transformer. arXiv preprint 230.00207,,, 2023.
Manzil Zaheer, Guru Prashanth Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Al- berti, Santiago Ontanon, Philip Minh Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Mahmoud El Houssieny Ahmed. Big bird: Transformers for longer quences. 2020. URL https::/procceedngg... neurips cc/paperr2220/Eiiee pdf
Zhaokun Zhou, Yuesheng Zhu, Chao He, Yaowei Wang, Shuicheng Yan, Yonghong Tian, and
Li Yuan. Spikformer: When spiking neural network meets transformer. arXiv preprint arXiv:2209.15425 2022.
Rui-Jie Zhu, Qihang Zhao, and Jason K. Eshraghian. Spikegpt: Generative pre-trained language model with spiking neural networks. arXiv preprint 2rXiv:2302.13930 2023.
12

---

Published as a conference paper at ICLR 2024
The number of batch size and maximum learning rate for different subtasks in the LRA dataset. The number in parentheses indicates the learning rate for spiking models it is different from the learning rate of the non-spiking models.
Table 6:
ListOps (2K)
Retrieval (4K)
Pathfinder
Hyperparameters
Text(4K)
Image( IK)
(1K)
Batch size
32
32
32
256
256
Max. Learning rate
0.01
0.0001
0.0001(0.001)
0.01
0.001
Table 7: Performance comparison of our LMUFormer models and previous works on the LRA
dataset.
ListOps
Retrieval
Pathfinder (1K)
Model
Text(4K)
Image(1K)
Avg
(2K)
(4K)
LMUFormer
34.43
68.27
78.65
54.16
69.9
61.08
Spiking LMUFormer
37.30
65.80
79.76
55.65
72.68
62.24
pLMU
22.83
63.72
74.1
53.50
68.32
56.49
Spiking PLMU
17.94
60.84
65.19
51.89
49.88
49.15
A APPENDIX
A.1 LRA BENCHMARK
The benchmark comprises:
ListOps: A synthetic task where models operate on nested lists, gauging their understand- ing of hierarchical structures.
Byte-level classification of IMDb reviews, pushing the boundaries sequence lengths.
Text:
Retrieval:
Document retrieval tasks centered around byte-level tokens.
Image classification, but uniquely handled as pixel sequences, sidestepping tradi- tional 2D processing.
Image:
Pathfinder:
An masge task assessing if an unobstructed path connects two high- lighted points, challenging models on long-range spatial dependencies.
A.2 TRAINING HYPERPARAMETERS
For PsMNIST dataset, we trained the models 50 epochs with Adam (Kingma & Ba 2014) optimizer and the initial learning rate is 0.0001 and the batch size is 100. As for the Speech Commands V2 dataset, the batch size is set to 128 during 100 (200) epochs with an initial learning rate of 0.00025 (0.0025) and weight decay of 5e-7. After the very first 5 (10) epochs, we multiply the learning rate by 0.85 every (10) epochs. The numbers brackets are the settings for spiking models. For the LRA dataset, we generally used the setting as Gong al. (202I). For all subtasks,
adopted Adam W (Loshchilov & Hutter (017) optimizer and the Icycle learning rate policy (Smith & Topin, 2019. The weight decay for the 'Image" task is 0.01 and for other tasks. More hyperparameters are shown in Table. [
A.3 MODEL BIAS & LIMITATIONS
There may be a few potential limitations scenarios where LMUFormer may not perform very well. For example, LMUFormer may lack the pre-training capabilities Transformer, that can enable high performance when fine-tuned on range of downstream tasks. Rather, it needs to be trained on the streaming use-case directly by the user. That said, it is not very clear whether pre-training on large text corpus that empower Transformers, streaming use-cases. Moreover, to our best knowledge, there is no large-scale streaming dataset, such the BookCorpus, that can pre-train models. Given large-scale pre-trained
improve the performance of our models
13

---

Published as a conference paper at ICLR 2024
Table Performance, training time and memory comparison LRA 'Text' subtask.
Peak memory
Model
Text(4K)
Params. (M)
Training (s)
Validation (s)
(MB)
Linear Trans.
65.90
0.174
644
5.91
2.37
Linformer
53.94
0.698
1180
8.69
2.46
Transformer
64.27
0.174
1780
20.60
2.49
Nystromformer
65.52
0.174
836
11.24
4.09
LMUFormer
68.27
0.174
738
6.29
2.46
Spiking LMUFormer
65,80
0.175
928
12.02
5.66
pLMU
63.72
0.132
582
4.60
1.83
Spiking pLMU
60.84
0.132
684
8.68
3.75
dataset (and enough compute power), it may be possible for LMUFormer scale up, and achieve high performance when fine-tuned on downstream tasks. We hypothesize this may not be possible by simply stacking LMUFormer blocks, and may require network architectural optimizations, which is an important research direction.
Moreover, as shown in Table 4, LMUFormer, though yields higher performance compared to all transformer variants, can not surpass the S4 models on LRA dataset, that evaluates model quality under long-context scenarios. Improving the LMUFormer performance for such datasets is also an important and interesting research direction.
A.4 COMPARISON BETWEEN DIFFERENT MODELS ON LRA
For the LRA dataset, we use the same word and position embedding for all the models, replacing the patch embedding block
for LMUFormer. This narrows the gap between LMUFormer and pLMU models on LRA compared to Google Speech Commands as shown in Table 7 Overall, the LMUFormer significantly outperforms pLMU, with comparable results only in Task 'Image" and Task 'Pathfinder..
We also show comparison performance, model size, peak memory, training time, and inference time for the pLMU model, LMUFormer, Spiking LMUFormer and other Transformer variants, using the "Text" subtask as an example. Our results are shown in the Table 8. we can see, our LMU- Former achieves the best performance while maintaining relatively decent training and inference speeds, and requires less memory during training.
A.5 HARDWARE-SOFTW.CN CO-DESIGN
We develop a hardware simulation framework SNNs to estimate the energy, latency, and through- put of our non-spiking and spiking LMUFormer models. We have also incorporated the overhead due to the spike sparsity in our framework, which is minimal with high sparsity as obtained this work.
The total compute energy (CE) the spiking LMUFormer (SpLMUCE) can be estimated as
L Σ
SpLMUCE
DNN Fxnact
(S Fac: + ENN Σ NNN From
1=2
(13)
because the SNN receives full-precision input in the first layer (1=1) without any sparsity. Note that DN Nçom denotes the total number of comparison (thresholding) operations the layer with each operation consuming 1.64pJ energy in our 28nm Kintex-7 FPGA platform for floating point (FP) reperesentation. Also, note that DN denote the total number floating point (MAC or AC) operations in layer 1. Lastly. St denotes the activation sparsity of layer time step and Exp = 0.05pJ denotes the energy overhead due to sparsity, that is incurred in checking whether the binary activation is zero
14

---

Published as a conference paper at ICLR 2024
The CE of the full-precision LMUFormer (LMUcE) is estimated E DNN, Emacc where ignore the energy consumed by the non-linear activation operation (significantly lower compared to thresholding operation).
DNNCE
The compute-efficiency of the spiking LMUFormer stems from two factors: 1) high activation spar- sity, where Σ =055 on average across all the layers, and 2) Use of only AC (1.8p) operations that consume 7.4x lower compared to each MAC (13.32pJ) operation in our FPGA setup for float- ing point (FP) representation. Note that the binary activations can replace the FP multiplications with logical operations, i.e., conditional assignment to 0 with a bank of AND gates. These replace- ments may be realized using existing hardware depending on the compiler and the details of their data paths. Based these two factors, observe that our spiking LMUFormer incurs 27. x lower compute energy compared to LMUFormer on the Google speech commands at iso-parameter.
In contrast, the energy incurred in the memory access of the weights for both non-spiking and spiking LMUFormer depend on the data re-use scheme and the underlying hardware. However, since both the models have almost the same number of trainable parameters, they expected to incur identical memory energy. We also do not expect any additional latency or throughput improvement in SNN, since need to process the identical sequential input, and activation sparsity favors compute energy (and not latency). That said, unlike existing SNNs that incur an additional temporal overhead and suffers from high latency, our SNN re-uses the hidden memory dimension of the LMUformer to incur the temporal dimension. Thus, our spiking LMUFormer can yield higher compute efficiency with no overhead on latency.
15


Reviews:
[{"review_id": "1RmUmEoGQd", "reviewer": "Reviewer_KEVY", "summary": "This paper presents a sequential network that exploits Legendre Memory Units (LMU) module as its temporal computational core. It has been shown that the proposed network, which is called LMUFormer, can achieve a similar performance to transformers with less number of parameters and operations.", "questions": "See my concerns listed as weaknesses.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "1) The accuracy performance of the proposed work is decent and the memory/operation reduction is significant.\n\n2) The paper is also well-written and easy to read and understand.", "weaknesses": "1) The contribution of this work is rather limited as it relies on a previously proposed module (i.e., LMU). It is not also clear why the proposed network yields a better accuracy.\n\n2) My main concern is the performance of such a network when pre-trained. The main advantage of transformers come from its pre-training stage which allows the model to perform well on downstream tasks during fine-tuning. To learn and store those pre-training data, transformers contains numerous parameters. As such, I am not supersized by the results reported in this paper since the transformer used for the comparison was not pre-trained and consequently it is expected to see a better performance from LMUFormer. What would be the performance of LMUFormer on well-known benchmarks such as GLUE? Can LMUFormer be pre-trained?\n\n3) Lack of theoretical analysis and reasoning on the superior performance of LMUFormer.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hBxkiVMo3R", "reviewer": "Reviewer_2bBb", "summary": "This work combines Legendre Memory Units (LMU)  with convolutional patch embedding and convolutional channel mixer in its network design to improve accuracy. It also introduces a spiking version to further improve its inference efficiency. On speech recognition tasks, their results show a significant reduction in both model size and FLOPs compared to the SoTA AST model. It also achieves higher accuracy compared to other low-complexity SNN designs. It also shows it can achieve competitive accuracy with only two-thirds of the seq length due to its sequential processing capability.", "questions": "1. Is there a reason why Am(t) is changed to Am[t-1] from eqn(1) to eqn(2) for discretization?\n2. From eqn (4), it seems the LMU module is composed of mainly matrix-vector operations, which is similar to RNN. Is it correct that we can also parallelize it across the time dimension? can you elaborate on how it can be solved in non iterative way to enable parallelized training? \n3. I'm curious to know how its measured training and inference time compares to that of other linear-time transformers like Linformer on the off-the-shelf hardware.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The addition of the convolutional patch embedding (w/ 1d conv) and convolutional channel mixer to LMU is well explained and justified.\n- The model can be trained in parallel and has the streaming capability during inference. It does not need to wait until the entire input sequence is available, which can reduce the compute complexity and memory requirement.\n-  This transformer design achieves competitive accuracy on a wide range of tasks compared to its counterparts with low compute complexity. It also demonstrates better classification accuracy in long token tasks compared to other linear-complexity attention designs.", "weaknesses": "- The results could be made stronger if the model size and compute complexity comparison can be added to every performance comparison table (Table 1,2,4).\n- Ops represented in the paper for SNN can be a very optimistic proxy for latency performance. It is worth mentioning that SNNs might not be easily accelerated on off-the-shelf hardware like CPUs and GPUs. It might require specialized hardware to demstronate its advantage in latency and energy reduction.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6tFvnxJ2JU", "reviewer": "Reviewer_FTnd", "summary": "This paper proposes a LMU based model that also has a spiking variant. The goal is to explore the accuracy gap between transformer based model with LMU based model with more complex sub-modules.", "questions": "* Does the proposed method suit SNN better and more nature? What about the training cost and time?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* This paper is mostly easy to follow and well organized.\n* Related background and works are discussed and introduced extensively.\n* The intention of exploring this performance gap in function complexity feels intuitive and interesting.\n* Experimental are conducted extensively amongst different models and tasks and comparing both accuracy and computational cost, results also look mostly promising.", "weaknesses": "* In task 2, there is still some accuracy gap between the proposed method and some transformer based methods. Is there any way to compare pLMU on the same task as well as resource usage? It feels the narrative is that accuracy wise on relative complex tasks LMUformer is on par with transformer based method while out-performing pLMU. Besides rMNIST task, it is difficult to tell how much performance gain that the proposed method would offer while (potentially) introducing extra hardware cost.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GjJ4G3QHVw", "reviewer": "Reviewer_4AC7", "summary": "Transformer models are highly accurate but complex and not suited for sequential processing, making them ill-suited for edge devices. RNNs are more suitable for these devices due to their lower complexity and sequential processing capabilities, but they often lag in performance compared to Transformer models. The paper proposes LMUFormer, a model that augments Legendre Memory Units (LMU) with convolutional patch embedding and convolutional channel mixers. This results in a fully-sequential recurrent model that approaches the performance of Transformer models while retaining the ability to process data sequentially.  A spiking version of LMUFormer is also introduced, which brings the benefits of states within the patch embedding and channel mixer modules, further reducing computing complexity.\nThe contribution lies in \n1. New Model Architecture: Introduction of LMUFormer, a novel architecture that combines LMUs with convolutional patch embedding and channel mixers, aiming to provide a balance between high performance and low complexity.\n\n2. Performance: LMUFormer demonstrates impressive performance, particularly on the Speech Commands V2 dataset, showing comparable results to state-of-the-art transformer-based models but with significantly fewer parameters and lower computational complexity.\n\n3. Spiking Version: Presentation of a spiking version of LMUFormer, establishing a new state-of-the-art in low-complexity Spiking Neural Network (SNN) variants with an accuracy of 96.12%, and demonstrating the ability to process real-time data efficiently.\n\n4. Efficiency in Real-Time Processing: The model shows proficiency in real-time data processing, achieving a 32.03% reduction in sequence length with minimal performance decline, highlighting its suitability for streaming applications at the edge.", "questions": "- citation for S4 models?\n\n- Could you add \"how to get 32.03%\" in section 5.5? I assume it is (128-87)/128? similarly to 70x and 140x to help reading", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Originality:**\n- Innovative Model Design: The LMUFormer introduces a unique combination of Legendre Memory Units (LMU) with convolutional patch embedding and channel mixers, creating a novel architecture that stands out in the realm of spiking models and RNNs.\n- Spiking Model Integration: The integration of a spiking version of the LMUFormer adds a layer of originality, as it brings the benefits of states within the patch embedding and channel mixer modules, while also aiming to reduce computing complexity.\n\n**Quality:**\n- Robust Performance: The LMUFormer demonstrates robust performance, especially highlighted by its results on the Speech Commands V2 dataset, where it shows comparable performance to state-of-the-art transformer-based models but with a significant reduction in parameters and computational complexity.\n- Comprehensive Evaluation: The paper includes a thorough evaluation of the architectures on multiple sequence datasets, providing a solid basis for the claims made about the model’s performance and efficiency.\n\n**Clarity:**\n- Well-Structured: The paper is well-structured, with clear sections that logically flow from one to the next, making it easy for readers to follow the development of ideas and understand the proposed model.\n- Detailed Explanations: The authors provide detailed explanations of the LMUFormer architecture, the spiking version of the model, and the motivations behind their design choices, contributing to the overall clarity of the paper.\n\n**Significance:**\n- Addressing Resource Constraints: The LMUFormer addresses a significant challenge in the field of edge computing and streaming applications, where devices are heavily resource-constrained. By providing a model that combines high performance with low complexity and sequential processing capabilities, the paper makes a meaningful contribution to this area.", "weaknesses": "**Addressing Potential Biases:**\nModel Limitations: The paper could be improved by providing a more balanced view, including a discussion of potential limitations or scenarios where the LMUFormer might not perform as well. This would help readers develop a more nuanced understanding of the model’s applicability.\n\n**Enhancing Reproducibility:**\nImplementation Details: Providing more implementation details, including hyperparameters and training procedures, would enhance the reproducibility of the results, contributing to the paper’s overall quality.\n\n**Real-Time Processing Analysis:** \nGiven the focus on streaming applications and real-time data processing, a more detailed analysis of the model’s performance in real-time scenarios, including potential latency issues and how they are addressed, would be valuable.\n\n**Software-Hardware codesign**\nIt would be nice to see a hardware simulation for SNN to learn all aspect of the new model, engergy, latency, throughput and any overhead, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)