Accelerating Linear Recurrent Neural Networks for the Edge with Unstructured Sparsity
Authors: Alessandro Pierro
Steven Abreu
Jonathan Timcheck
Philipp Stratmann
Andreas Wild
Sumit Bam Shrestha
Venue: ICML 2025 poster (2025)
Keywords: Linear RNNs
Sparsity
Pruning
Quantization
Neuromorphic Hardware
Primary area: general_machine_learning->hardware_and_software

Abstract: Linear recurrent neural networks enable powerful long-range sequence modeling with constant memory usage and time-per-token during inference. These architectures hold promise for streaming applications at the edge, but deployment in resource-constrained environments requires hardware-aware optimizations to minimize latency and energy consumption. 
Unstructured sparsity offers a compelling solution, enabling substantial reductions in compute and memory requirements--when accelerated by compatible hardware platforms. 
In this paper, we conduct a scaling study to investigate the Pareto front of performance and efficiency across inference compute budgets.
We find that highly sparse linear RNNs *consistently* achieve better efficiency-performance trade-offs than dense baselines, with $2\times$ less compute and $36$\% less memory at iso-accuracy.
Our models achieve state-of-the-art results on a real-time streaming task for audio denoising.
By quantizing our sparse models to fixed-point arithmetic and deploying them on the Intel Loihi 2 neuromorphic chip for real-time processing, we translate model compression into tangible gains of $42\times$ lower latency and $149\times$ lower energy consumption compared to a dense model on an edge GPU.
Our findings showcase the transformative potential of unstructured sparsity, paving the way for highly efficient recurrent neural networks in real-world, resource-constrained environments.

Accelerating Linear Recurrent Neural Networks for the Edge with Unstructured Sparsity
Alessandro Pierro 12 Steven Abreu 13 Jonathan Timcheck
Philipp Stratmann Andreas Wild
Sumit Bam Shrestha
Abstract
Linear recurrent neural networks enable powerful long-range sequence modeling with constant memory usage and time-per-token during inference. These architectures hold promise for streaming applications at the edge, but deployment in resource-constraines environments requires hardware-aware optimizations to minimize latency and energy consumption. Unstructured sparsity offers compelling solution-when accelerated by compatible hardware platforms. In this paper, we investigate the Pareto front of performance and efficiency across inference compute budgets. We find that highly sparse linear RNNs consistently achieve better trade-offss than dense baselines, with 2x less compute and 36% less memory iso-accuracy. Our models achieve state-of-the-art results on streaming audio denoising task. By quantizing our sparse models to fixed-point arithmetic and deploying them on the Intel Loihi 2 neuromorphic chip, we translate model compression into tangible gains of 42x lower latency and 149x lower energy consumption compared to a dense model on an edge GPU. Our findings showease the transformative potential of unstructured sparsity, paving the way for highly efficient recurrent neural networks in real-world, resource-constrained environments.
https::/gittheii com/ http/Inteellaaa Sparseaarreeaaae
1. Introduction
Dense 85
Linear Recurrent Neural Networks (RNNs) have recently emerged powerful primitives for sequence modeling, both in isolation or hybridized with self-attention, achieving im- pressive results in language modeling (Poli et al., 2024), audio generation (Goel et al., 2022), and genomics (Nguyen et al., 2023), and many other areas. This success has been ignited by advances in initialization, parametrization, and parallelization of these models, which, combined, enabled large-scale training on GPUs (Voelker et al., 2019; Chilkuri et al., 2021; Gu et al., 2020; 2022b; Smith et al., 2023).
Training
bT)
Iterative Pruning
Clean
ReLUfication
Noiay
Quantization
FXP Conversion
Compressed
Intel Loihi chip
Figure I. Model compression and acceleration pipeline for linear RNNs, tailored to the Intel Loihi 2 chip.
At inference time, linear RNNs iteratively compress the in- put sequence into a finite-dimensional representation whose dimensionality does not depend on the sequence length. Their memory requirements remain constant regardless of sequence length, and runtime scales linearly with sequence length. In contrast, transformer architectures (Vaswani et al.,
2017) exhibit linear memory growth and quadratic runtime scaling sequence length increases. This advantageous scaling makes linear RNNs especially well-suited for real- time long-range sequence modeling on edge devices that require low latency, a small form factor, and are subject to weight and power constraints, as common for applications like audio denoising (Timcheck et al., 2023), keyword spot- ting (Warden, 2018), or gerception-and-comee 2023). Although model optimization and compression are essential for enabling efficient edge machine learning by reducing resource demands, their application to accelerate the inference of linear RNNs remains under-explored.
'Equal contribution 'Nurromorphic Computing Lab, Intel Corporation, USA 'Institute of Informatics, L.MU Munich, Ger- many Berrouull Institute CogniGron, University of Groningen, Netherlands. Correspondence to: Alessandro Pierro < <alessan- dro, dro.pierroo@in pierro@itee.com..... intel.com>.
(Lu et al.,
Proceedings of the ndnd International Conference on Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Accelerating Lincar Recurrent Neural Networks for the Edge with Unstructured Sparsity
x N Mayers
diag D
Signoid
BatchNom
Emood
GELUT
Droma
Dropout
Decoder
Uk
diag (A)
Figure Overview of the architecture. Symbols shown defined by equations in Section 2.1.
Linear RNNs are a promising match for neuromorphic pro- cessors, which can efficiently update stateful neurons due tight integration of massively parallel compute and mem- emerging class of brain-inspired hardware architectures, with notable exam- ples like IBM's NorthPole (Modha et al., 2023), SpiNNaker (Mayr et al., 2019). Tianjic (Pei et al., 2019), and In- tel's Loihi 2 (Orchard et al., 2021). Beyond parallelism and compute-memone integration, different neuromorphic processors offer unique sets of further computational fea- tures, including event-driven compute and messaging, low- precision arithmetic, and support for unstructured sparse weight matrices. These sets of features offer unique opportu- nities to optimize and compress linear RNNs for real-world applications.
2. Compressing linear RNNs
2.1. Linear Recurrent Neural Networks
ory. Neuromorphic processors
Recurrent neural networks (RNNs) are a class of neural networks designed for processing sequential data by main- taining hidden states that capture temporal dependencies. Linear RNNs distinguish themselves through their linear dynamics, which enables parallelization over the sequence length and, therefore, efficient training. Previous work has shown- both theoretically (Orvieto et al., 2024) and empiri- cally (Gu et al., 2ha2a)-tha the network's recurrent weight matrix can effectively be diagonalized in the complex do- main without any loss generality or model capacity. We use this diagonal formulation of linear RNNs, such that the network's 's update equations for the state X C and output yı RM given by:
In this work, we explore the potential of unstructured sparsity-in weights and activations-and fixed-point quanti- zation for the compression of linear RNNs and acceleration on neuromorphic hardware illustrated in Figure 1. Specif- ically, we explore four key research questions:
X diag(A) 2 Xk-1 B u
(I)
Y4 = CT X diag D)
(2)
where @ denotes the Hadamard product, us E RR the input sequence, diag(A) () C are the diagonal recurrent weights, B CMxN the input weights, C E CN the output weights, and diag (D) RMM are the residual weights. We follow the S5 model (Smith et al., 2023) for the initialization and parameterization of the linear RNN,
1. Can we train linear RNNs with high synaptic and acti- vation sparsity while retaining high performance?
2. Do highly sparse linear RNNs outperform dense linear RNNs across different inference compute budgets?
Because its linearity, the temporal mixing of the S5 block above is followed by a nonlinear channel mixing block. We use particular variant of the GLU block (Dauphin et al., 2017) where the linear RNN's output yı E R is transformed GLU(U) is an element-wise nonlinear function (we use either the Gaussian error linear unit (GELU) or the Rectified Linear Unit (ReLU)), W E IR MMM is a weight matrix, and o is the sigmoid function. The full model architecture is illustrated in Figure
3. Can fixed-point quantization compress sparse linear RNNs without damaging the network's performance?
4. Can unstructured sparsity and fixed-point quantization be translated into latency and energy advantages on neuromorphic hardware?
(W (W((y+)) & 7(yx) where
We provide definite positive answers to questions and 4, and present positive evidence for questions and
2

---

Accelerating Linear Recurrent Neural Networks for the Edge with Unstructured Sparsity
Kapoho Point
VPX
Alia Point
Hala Point
Neuro cores
Dendrite
Spike
Parallel off-chip
I0 interfaces
10G Ethemet
Edge
Edge/aerospace 16 16chips
Datacenter 128 chips
HPC 1152 chips
Microprocessors
8 chips
Async NoC
Synapses
Accumulator
(a)
(b)
Figure 3. (a) Loihi implements network of neurons, which are processed by neuro-cores and communicate via an asynchronous network-on-chip. Parallel IO and 10 Ethernet interfaces enable a Loihi 2 chip to communicate with other Loihi 2 chips and external hosts, respectively. Embedded micropracessors provide a flexible method of interaction with neuro-core registers, management, and communication. On a neuro-core, each neuron receives spike messages from other neurons via synapses with multiplicative weights wi. and sums them up by one or multiple dendritic accumulators. The input is used by a dendrite to update memory states that are local to the respective neuron. The neuron communicates with other neurons by sending spike messages. (b) Different Loihi 2 systems are available to cover a wide range of applications from the edge to HPC with up to neurons.
2.2. Neuromorphic Computing with Intel Loihi 2
2.3. Evaluating Benefits from Sparsity
Neuromorphic processors mimic computing principles of the brain, which excels in processing sequential data streams with just around 20 W of power. Loihi 2 is the second- generation of Intel's neuromorphic research processor (Or- chard al., 2021) and implements spiking neural network as illustrated in Figure 3. The network is processed by massively parallel compute units, with 120 neuro-cores per chip. The neuro-cores compute and communicate asyn- chronously, but a global algorithmic time step is maintained through a barrier synchronization process. The neuro-cores are co-located with memory and can thus efficiently update local states, simulating up to 8 192 stateful neurons per core. Each neuron can be programmed by the user realize variety of temporal dynamics through assembly code. Input from and output to external hosts and sensors is provided with up to 26 M 32 bit integer messages/s /s (Shrestha et al., 2024b). Loihi 2 can scale to real-world workloads of vari- ous sizes with up to 1B neurons and 128 B synapses, using fully-digital stacked systems shown in Figure 3.
Unstructured sparsity has demonstrated compelling results as an effective model compression technique, serving both as a framework for theoretical analysis of sparsity algo- rithms and as an upper-bound for the gains achievable with constrained forms of sparsity (Liu & Wang, 2023; Mishra et al., 2021; Han et al., 2015). In particular, when com- pared to structured sparsity patterns, like N:M (Mishra et al., 2021) or block-diagonal, it typically attains higher task per- formance or compression rates (Lee et al., 2023). However, the gains of unstructured sparsity have not been realized as the traditional GPU architecture is suited to exploit only block sparsity structures (Liu & Wang. 2023). Additionally, sparse activations complement synaptic sparsity, resulting in fewer operations overall (Mukherji et al., 2024), but GPUs typically cannot take advantage of activation sparsity ei- ther. Realizing the benefits of unstructured sparsity requires suitable hardware architectures (Lie, 2023; Ashby et al., 2019; Zhang et al., 2021). The event-driven neuromorphic architecture of Loihi 2 is inherently suited take advan- tage of the unstructured sparsity in both connections and activities, in particular, when they are extremely sparse, i.e., > 90%. Therefore, we choose to compare the benefits of efficiency gained from sparsity on Loihi 2 with equivalent dense networks on an edge GPU.
The architectural features of Loihi 2 offer unique oppor- tunities to compress and optimize deep learning models. Like GPUs, its neuro-cores benefit from model quantization, as it supports low- www-preciionn arithmetics, 8 bit for synaptic weights and up to 32 bit for spike messages. Unlike GPUs, Loihi 2 is optimized for local computations within neu- rons, common focus of neuromorphic processors. First, it allows fast and efficient updates of neuronal states with recurrent dynamics with minimal data movement, due to its tight compute-memony integration. Second, the fully asynchronous event-driven architecture of Loihi 2 allows it to efficiently process unstructured sparse weight matri- ces. Third, the neuro cores can leverage sparsified activation between neurons, as the asynchronous communication trans- fers only non-zero messages.
Theoretical studies have shown that wider sparse layers out- perform dense layers with the same number of parameters (Golubeva al., 2021; Chang al., 2021). Research has further shown that, in practice, it is better to train larger pree-paramtterized network and prune it to make it leaner compared to training a compact sparse network from start (Frankle & Carbin, 2018; Renda et al., 2020; Chen et al., 2020). There is evidence showing minimal loss in accuracy when the networks are pruned, typically to sparsity levels of 50-80% (Chen al., 2020). However, there is not much
3

---

Accelerating Linear Recurrent Neural Networks for the Edge with Unstructured Sparsity
research on performance at extreme levels of sparsity of 2 90%. We thus ask; Do highly sparse networks achieve superior performance to dense networks when operating under identical inference compute budgets? How does the performance benefit sparsity vary with increased compute budget?
Activity sparsification Sparsifying layer activations pro- vide another means for reducing the compute and on-chip memory requirements during inference. In particular, sparse pre-activations of linear layers can significantly reduce the number of MACs required for the associated matrix-vector multiplication (MVM), if appropriately supported by the hardware backend. On sparse and event-driven accelerators, such as Loihi 2, sparse pre-activations directly translate into MACs savings since the MVM operation is computed as
In Section 3.2, we evaluate the effect of pruning and activity sparsification on haapplly--andccuuuulaa (MACs) opera- tions and task performance for k-family of sparse and densely trained networks where Ksparse [0.5,3.4 3.0], Kaense E [0.25, 1.0] is the width scaling factor of the networks. In linear layers, which account for most of the computation in the S5 architecture, MACs scale linearly with weight and pre-activation sparsity. The detailed MAC calculation reported in Appendix A. Additionally, we benchmark iso-accuracy models on relevant hardware to validate the theoretical gains from sparsity with latency and power mea-
MVM(W, x) = W (1)) (11a,)00
(3)
In contrast, GPU architectures struggle to leverage dynamic sparse activation patterns and have demonstrated gains with more structured activation patterns, and only in memory- bound regimes as in auto-regressive generation with large models (Mirzadeh et al., 2024; Zhang et al., 2024; Shazeer et al., 2017; He, 2024a).
surements in Section 3.3.
Techniques for activation sparsity include top-k (Key et al., 2024), sigma-delta coding (Shrestha et al., 2024a; O' O'Connor & Welling, 2016), sparse mixture-of-exppet (Fedus et al., 2022; He, 2024b) and ReLU-fication (Mirzadeh et al., 2024). We base our methodology on the latter of these. Since ReLU is a fully element-wise operation, it doesn't 't require synchronization across channels which would complicate implementation in compute- such as Loihi 2. Following previous work on transformer models (Mirzadeh et al., 2024), we start from the original dense model with GELU non-linearity, as shown in Figure 2, and apply two modifications. First, we replace the GELU activation with a ReLU, sparsifying pre- arevvaciiations of the linear layer in the GLU block. Second, we insert additional ReLU activations after the residual add in the GLU block and to the real component of the S5 hidden layer, further increasing the pre- mre-activation sparsity of linear operators. Both model surgeries are applied to the pre-trained model at the beginning of the iterative pruning procedure, enabling accuracy recovery from both weight and activation pruning without extra training budget.
2.4. Model Compression
Synaptic pruning Given our focus on edge and low- latency applications, we design our compression pipeline assuming that fine-tuning or re-training of the models is feasible. Following previous work (Mishra et al., 2021), we initialize the parameters from the pre-trained dense models. We adopt iterative magnitude pruning (IMP) which increases sparsity progressively during training and achieves better task performance than one-shot approaches, especially at high sparsity levels (Zhu& & Gupta, 2018; Lee et al., 2023).
integrated platforms,
Specifically, we train for E epochs with T update steps in total. Sparsity starts at S, = 0 at t, = 0 and is increased following a d polynomial schedule (Zhu & Gupta, 2018) and updated three times per epoch as:
(1- 1-44
S. = -S,
11
with fy = 0.75T. Given the total sparsity S, and weights WF E N at time t and position é in the network, we scale the sparsity of for each weight according to the Erdös- Renyi-Kernel (ERK) strategy (Evci et al., 2020; Mocanu et al., 2018) to compute the mask Mf:
Quantization and fixed-point computation Reducing the numerical precision of weights and activations through quantization is an essential way to compress machine learn- ing models, directly leading to reduced memory footprint and faster inference (Gholami et al., 2021). We denote the tensor to be quantized with and the number of bits to with n, such that the quantized tensor X, is defined as:
Ne + M
: 8t 1 No. M M = 1 (IW 2 + of min [TopK Sf
where T. is the calculated threshold for W to reach sparsity of and TTopK(W, k) gives the top-k values from W. In the forward pass, weights are masked as W = M @ W, while the backward pass applies straight- estimation (Bengio et al.. 2013), enabling gradient updates also for masked weights.
X
+ 2x =
| + 2zl
(4)
where |.1 indicates rounding to the nearest integer, Sy is the scale for the given tensor, 2g is the zero point, and As is the corresponding step size. For simplicity, we choose
4

---

Accelerating Linear Recurrent Neural Networks for the Edge with Unstructured Sparsity
5, -2--1 1)(max xx)= and 2s = 0, i.e., we use symmetric quantization based on the absolute maximum.
A single layer of programmable neurons can efficiently fuse many operations on the vector it encodes. This applies to all element-wise operations where each neuron must operate only on its local states. The neuronal layers thus implement ReI.Us, BatchNorm, Hadamard products, residual add, and multiplications of a state vector with a diagonal matrix. Ap- plying this layer fusion, the full S5 architecture only requires one neuron group for the encoder, one for the decoder, and three for each S5 block. The detailed mapping of operations to neuron groups is illustrated in Figure 8,
Post-training quantization (PTQ) applies quantization to pre-trained model without further training. which is com- putationally efficient but may lead to a notable drop in ac- curacy, especially for complex models or tasks (Gholami et al., 2021). Without constraints during training, it has been shown to under-perform on both nonlinear (Wu et al., 2016) and linear RNNs (Abreu et al., 2024). In contrast, quantization-
training (QAT) incorporates quantiza- tion into the training process using mators for the gradients (Bengio et al., 2013), allowing the model to adapt to the reduced precision and typically achieving superior performance retention compared to PTQ (Hubara et al., 2018), which has also shown promising re- sults on linear RNNs such as S4D (Meyer et al., 2024) and S5 (Abreu et al., 2024) on synthetic tasks from the Long Range Arena benchmark (Tay et al., 2021). To demonstrate advantages on hardware, we use static quantization (Gho- lami et al., 2021) using only fixed-point (integer) arithmetic (Wu et al., 2020). Whereas in dynamic quantization, scales 8r are computed dynamically on incoming data (and there- fore requiring floating-point operations), static quantization pre-computes scales for all weights and activations in the neural network and "freezes" these scales so that the net- work can be converted to use only fixed-point arithmetic.
esti-
3. Results
3.1. Experimental Setup
Software We implemented our methodology in JAX 0.4.30, building on top of the original S5 codebase (Smith et al., 2023), with JaxPruner (Lee et al., 2023) for the prun- ing algorithms and the AQT library (Google, 2024) for quantization- training. We implemented static quanti- zation and a fixed-point model ourselves using only JAX.
Audio denoising task We evaluated our approach on the Intel Neuromorphic Deep Noise Suppression Challenge (Timcheck et al., 2023). The objective of the Intel N- DNS Challenge is to enhance the clarity of human speech recorded on single microphone in a noisy environment. The Intel N-DNS Challenge utilizes data from the Microsoft DNS Challenge, encompassing clean human speech audio samples and noise source samples. (Reddy et al., 2020; 2021a;b; Dubey et al., 2024). Clean human speech and noise samples are mixed to produce noisy human speech with a ground truth clean human speech goal.
Following prior work on quantizing linear RNNs (Abreu et al., 2024), we choose bit for all weights, except the diag- onal recurrent diag(A) A) weights which is stored with 16 bit. All activations quantized 16 bit. We denote this quan- tization recipe with W8A16. This is a more compressed quantization scheme than previous work that deployed linear RNN to fixed-point hardware using W8A24 (Meyer et al., 2024). For the linear RNNs that are deployed to the Loihi 2 chip, we combine QAT with sparse training.
To train our models, we used the default Intel N-DNS Chal- lenge training and validation sets, each consisting of 60 000 noisy audio samples of 30s each, and a test set with 12 000 samples. We encoded and decoded each audio sample us- ing the Short-Time Fourier Transform (STFT) and Inverse Short- Time Fourier Transform (iSTFT) (Gröchenig, 2013). Following the N-DNS baseline solution, NsSDNet (Shrestha et al., 2024a), we adopted a 32 ms window length and a 8 ms hop length for the STFT/ISTFT. This resulted in a nominal real-time audio processing latency of 32 ms, which allows ample time (8 ms) for denosing network inference, as 40 ms is the standard for an acceptable latency according to the Mi- crosoft DNS Challenge. We evaluated the denoising quality of our model using the scale-invariant signal-to-noise ratio (SI-SNR)
2.5. Porting S5 to Loihi 2
Running S5 on Loihi requires a range of adjustments, to fully leverage the neuromorphic architecture and to adhere to its constraints. As a result, the S5 network shown in Figure 2 is transformed into network of synapses and neurons for Loihi 2 as illustrated in Figure 8. In general, a state vector of dimension I is encoded by M neurons. Matrix-vector multiplications are hardware accelerated by the synaptic layers, which take a vector of neuron activities, multiply it with the matrix of synaptic weights, and pass the output to the next layer of neurons. Since complex numbers not natively supported on Loihi 2, the complex matrices B and C have been split into two synaptic layers each. Similarly, the complex state Ia is stored by two neuronal states. The remaining operations are performed within the neurons.
|targge 2
SI-SNR = 10 10
(5)
|
Importantly, SI-SNR provides a volume-agnose measure of audio cleanliness relative to the ground truth signal.
5

---

Accelerating Lincar Recurrent Neural Networks for the Edge with Unstructured Sparsity
12
10
10
Test SI-SNR (dB)
15
Test SI-SNR (dB)
15
14
14
Dense w/ GELU Sparse ReLU
Donse
13
13
Sparse
2
5
0.5
1.5
2
Effective MACs (x 105)
Memory Footprint (MB)
Figure Pareto fronts for network audio denoising quality (SI-SNR) as function of effective compute (left) and memory footprint (right) on the Intel N-DNS test set. S5 networks with weight and activation sparsity (green) exhibit a large domain of Pareto optimality versus dense S5 networks (orange). Number annotations enumerate increasing S5 dimensionality configurations, from 500 k to M parameters. Dashed horizontal like marks SI-SNR of Spiking-FullSubbN Net XL, the previous state-of-the model. The horizontal arrows highlight models used for hardware deployment, the diagonal arrows highlight models the same width. See text for details.
3.2. Pareto Front of Performance and Efficiency
100
Norm
SS Out = GLU
We studied the performance- -eeermance--ffcceenyy Pareto front of dense and sparse models across inference compute budgets. Start- ing from the S5 architecture (Smith et al., 2023), we trained family of dense models of increasing size by linearly seal- ing the model dimensions (i.e. model width and size of the SSM hidden state), while keeping the depth fixed three S5 layers. Similarly, trained a family of sparse models, i.e. pruned and ReLU-fied, according to our methodol- ogy discussed above, with 90% of weights pruned by the end of training (further details on the model dimensions are provided Appendix A.2). The results, reported in Figure 4, compare de-now de-noising performance (SI-SNR) and computational efficiency measured by effective MACs and memory footprint (see Appendix A.I).
Pre-activation Sparsity (%)
Dense
KXW Sparse
80
60
40
20
Layer
Layer 2 Model Depth
Layer 3
The results show that sparsification significantly degrades performance when applied to mnder-parametrizee dense models (e.g.. sparsifying dense-3 reduces SI-SNR by 7.3%). However, task performance is recovered with increased model dimensions and the accuracy of dense models is matched larger sparse ones, with fewer MACs and lower memory requirements. This gives empirical support the- oretical work on the capacity of apdrse-and-wide neural networks (Golubeva et al., 2021). For example, sparse-8 model requires 2x less compute and 36% lower memory than the dense-3 model, while achieving the same level of accuracy. Overall, sparse models constitute the Pareto front of task performance and computational efficiency across compute budgets.
Figure 5. Activation sparsity of ReLU blocks across model depth for dense model and a sparse-weight model. The sperse-weight model exhibits significantly lower activation sparsity across layers.
ing out of the box. When compared to Spiking-FullSubbe XL (Hao et al., 2024), the Track I winner of the Intel N-DNS Challenge with 15.2 dB SI-SNR, our sparse- S5 model requires 3.2x less compute and 5.37 lower memory iso-aceuracy. This finding is in line with previous research audio modeling with state space models (Goel al., 2022), and provides additional evidence on the suitability of these architectures for signal processing.
In terms of absolute task performance, we find that the S5 architecture provides state-of-the-a results on audio denois-
Interaction of weight and activation sparsity An inter- esting question is what is the interaction between the two
6

---

Accelerating Linear Recurrent Neural Networks for the Edge with Unstructured Sparsity
types of sparsity, in weights and activations. Figure 5 re- ports the pre-activation sparsity for different layers across the model depth for two ReLU-fied models of the same size (model variant 6), with and without synaptic sparsity. We observe that the synaptic-sparse model exhibits lower acti- vation sparsity across the board, a finding that is consistent across model sizes. In addition, activation sparsity signifi- cantly decreases with model depth, both for dense and sparse models. These phenomena, previously observed in other models (Mukherji et al., 2024), suggest that, during training, the model compensates the reduced information flow caused by pruning with increased levels of activation. Additional research on more advanced activation functions would allow for the optimal allocation MACs, especially those that provide explicit control over sparsity without cross-channel synchronization (e.g., approximate top-k (Key et al., 2024)).
Base
QAT
Memory (kB)
FP32
451.4
Static Quant
451.4
FXP (Sim)
116.7
FXP (Loihi)
116.7
10
12
14
Test SI-SNR (dB)
3.3. Hardware Implementation
Figure 6. Impact of quantization interventions on Test SI-SNR and memory footprint, with and without quantization-awe awaree training, for model variant sparse-6. The results show that the Base model without QAT performs slightly better FP32 than QAT model. but significantly worse in static quantization and fixed-point preci- sion.
Impact of fixed-point conversion Since Loihi 2 only sup- ports fixed-point (FXP) arithmetic, as presented in Section 2, quantized the weights and activations of our model and implemented the network dynamics in FXP arithmetic. The effect of our quantization methodology is presented in Fig- ure 6. Starting from a 32-bit floating-poit (FP32) model, we apply static quantization, which rounds weights and ac- tivations using fixed scales, but still performs the actual Train-
performance on Jetson Orin Nano', which is configuration dense-3 from Figure 4. There exist variety of modes in which to execute model Loihi and Jetson, each ex- hibiting different tradeoffs in terms of latency, throughput, and energy. Therefore, we present different modes for a comprehensive characterization and comparison. We sum- marize our profiling results in Table 1. More details on the different execution modes on Loihi 2 are presented in Appendix A.3.2.
computation in FP32. Notably, Quantization-Awee ing (QAT) is very effective in maintaining test performance (SI-SNR) from FP32 to static quantization, compared to Post-Training Quantization (PTQ). The frozen scales from static quantization are imported into our FXP model imple- mented in JAX, which uses only int32 types and fixed-poine arithmetic to compute the forward pass of the model. We observe further performance degradation in the FXP simu- lation, which we analyze in more detail in Appendix A.3.3. We finally map the FXP model to Loihi 2 and perform inference on the chip, again finding degradation in SI- SNR, which is likely due to subtle differences in the integer arithmetic performed by the FXP simulation and Loihi im- plementation with fused layers. Another source of mismatch that the FXP model in simulation handles overflows by clipping the maximum value, whereas Loihi 2 "wraps around" the value, resulting in sign inversion. The size of the model decreases by about a factor of 4 when transition- ing from FP32 weights to INT8 weights, as shown on the right side of Figure 6.
In real-time, token-by-token processing on single input se- quence, Loihi 2 processes single STFT frame 35x faster and with 1200x less energy than the Jetson Orin Nano. When the Jetson Orin Nano processes "chunks" of multiple time steps, its utilization increases, and energy per token improves. With the largest chunks that fit the real-time re- quirement of latency <8 m sec, Loihi 2 is 42x faster and uses 149x x less energy per token.
In offline processing. when many STFT frames are buffered to process in succession (or in parallel), the energy effi- ciency and throughput of the Jetson Orin Nano improves. Loihi 2 performs offline processing with pipelining (see Appendix A.3.2 for further explanation). When processing single sequences, i.e. batch size b 1, Loihi 2 has 3.7x higher throughput with 8x less energy per sample.
Power and Performance To measure the empirical effi- ciency benefits afforded by the sparse S5 model on neuro- morphic hardware, we profile inference on Loihi 2 using the fixed-point S5 model, in particular, configuration sparse-8 from Figure 4. To compare to conventional hardware, we profile the smallest dense model that achieves equivalent
It is important to note that the Jetson Orin Nano is only fully
OOur W8A16 fixed-point model in JAX does not provide speedup over the FP32 model the Jetson Orin Nano, therefore we profile the FP32 model.

---

Accelerating Linear Recurrent Neural Networks for the Edge with Unstructured Sparsity
Table 1. Power and performance results". The Loihi is running a sparse and quantized S5 model, while the Jetson Orin Nano is running a smaller dense S5 model that reaches similar test performance. All measurements are averaged over 8 random samples from the test set, each containing 3 750 time steps. Gray highlights denote violation of real time constraints for the audio denoising task. Best real-time results underlined.
Mode
Latency (1)
Energy (L)
Throughput (1)
Token-by-token
Intel Loihi 2
Fall-Through
76 μm
13 pJ /tok
13 178 (178tok//
Jetson Orin Nano!
Recurrent 1-step (b 1)
2 2888 μs
15 724 p.
372 tok/s
Jetson Orin N
Recurrent 10-step (b 1)
3.224 us
936 p. /tok
3003 tok/s
Jetson Orin Nano
Recurrent 100-step (h 1)
10 103 ps
626 nJ /tok
9.516 tok/s
Jetson Orin Nano
Recurrent scan (6 1)
236 717 ps
404 μر /tok
15 148 tok/s
Sample-by-sample
Intel Loihi 2
Pipeline
60.58 ms
185.80 mJ/sam
16.58 sam/s
Jetson Orin Nano!
Scan (b- = 1)
233.48 ms
512.60 mJ /sam
4.28 sam/s
Jetson Orin Nano
Scan (b- = maxx
226.53 ms
5.89 mJJ /sain
130.09 sam/
Lothi worklonds churacterized un Oheo Gulch with N3CI revision Loihi chips running NaCore 2.5.8 and NoKemel 0.2.0 with on-chip to unthnotted sequencing of inputs. Researthers interested to run S5 on Loihi 2 can gain access to the software and systems by joining Jutel's Neuromorphic Research Community. Jetson warkloads were
chameteriond on in NVIDIA Jchon Owin Noto 8GB ranning ketpuck 6.2, CUDA 12.4, JAX 0.4.2,, using the MAXN SUPER power made; energy values are compuned bosed on the TOT
power er as reported by jup 4.3.0, The batch size Braas 256 was chosen to be the largest that fits into memory." Performance results are based on testing of Jinuary 2025 and may not reflect all publicly available security updastes; resilis may vary.
2
0.5
phones. For this reason, is interesting investigate the effeect of batch processing energy efficiency and latency for the two hardware architectures. Intel Loihi 2 doesn't 't natively support batching in the sense of processing multiple independent samples through the same model instantiation. However, the parallel inference of independent sequences can be achieved by replicating the model on the chip as many times as required by batch size, thereby obtaining higher throughput through a larger silicon area. We extended the results in Table I to compare the effect of this implemen- tation of batching on a 16-chip Loihi 2 VPX board to the usual hatch processing of the Jetson Orin GPU. The results, reported in Figure 7, show the energy per sample and the total latency for both architectures across batch sizes, from 16. Both hardware backends exhibit similar trend: while total latency remains constant, the energy efficiency improves proportionally with batch size. Loihi 2 remains competitive across batch sizes, showing between 4.43 to 4. 4.72x lower energy per sample and 4.52x lower latency on average. is important to note that since model repli- cas are physically mapped to different cores on Loihi, the resource requirements increase linearly with batch size. For this reason, such batch processing on Loihi is only feasible for small models and small batch sizes.
GPU
Energy
Loihi 2
Time
Energy per Sample (J)
0.4
1.5
Total Lateney (8)
0.3
0.2
0.5
0.1
0
0
10
15
Batch Size
Figure 7. Impact of batching on energy efficiency and latency for Loihi and Jetson Orin Nano. Both platforms exhibit similar trends: energy per sample decreases proportionally with hatch size, while latency remains approximately constant. Loihi maintains competitive performance on both metrics across batch sizes.
utilized when processing 256 sequences in parallel, and at this level, it shows significantly higher throughput while consuming less energy per sample, compared to Loihi 2. We include these results in the last row of Table 1.
Energy at real-time inference rate The latency budget for the neural network component of the audio denoising pipeline, running either on Loihi 2 or on the Jetson, is 8 ms. Our Loihi 2 and Jetson implementations are well below 8ms for online inference. Thus, to estimate the energy consumption in real-time settings, where subsequent tokens
Impact of batch processing While several edge applica- tions typically require batch-one inference, some scenarios can benefit from support for small- batch processing, e.g., de-noising audio streams from multiple on-device miero-
8

---

Accelerating Linear Recurrent Neural Networks for the Edge with Unstructured Sparsity
actually 8 ms apart, we rescale the power as:
Acknowledgments
We acknowledge the support of several colleagues at Intel who enabled our experimental setup on Intel Loihi 2, both in hardware and software. We are also grateful to Prof. Eyke Hüllermeier for feedback early draft of this work.
preal-time
Stattic
+
Ecompute
Pa dynnnmic
8 ms
based on the power measurements in token-by-token pro- cessing. In this setting, Loihi achieves 128 /took while the Jetson achieves 36 528 jJ/ttok for token-by-token pro- cessing and 3 720 puJ jJ/tok /tok when processing chunks of 10 time steps at once. Loihi 2 remains at least 3x more energy efficient than the Jetson Orin Nano.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Limitations Our Jetson Orin Nano implementation is in FP32, while our Loihi 2 implementation is in W8A16. Our fixed-point model in JAX provides no improvements in runtime or energy. More competitive Jetson energy, latency, and throughput could potentially be obtained by developing more optimized quantized implementation.
References
Abreu, S., Pedersen, J. E.,, Heckel, K. M., and Pierro, A. Q- S5: Towards quantized state space models. International Conference on Machine Learning Workshops, 2024.
Ashby, M., Baaij. C., Baldwin, P., Bastiaan, M., Bunting. O., Cairneross, A., Chalmers, C.,, Corrigan, L.,, Davis, S., van Doorn, N., Fowler, J., Hazel, G., Henry, B., Page, D., Shipton, J., and Steenkamp, S. C. Exploiting unstructured sparsity on next-generation datacenter hardware. 2019. URL
4. Diseussion
In this work, we explored the Pareto front of efficiency and performance for a streaming audio processing task, comparing dense and sparsified variants of a linear RNN based on the S5 architecture. We showed that combining activation sparsity and unstructured weight pruning results in significant reduction in compute requirements, up to 3.2x, and memory footprint, 5.7x, without accuracy degra- dation. In addition, we validated these theoretical gains with a accelerraecee implementation on a compute- memory integrated coarse accelerator, the Intel Loihi neu- romorphic chip. When quantized and deployed on Loihi 2, sparse models deliver 42x lower latency and 149x lower energy consumption in token-by-toker processing, com- pared to the iso-accuracy dense models on the Jetson Orin Nano GPU.
Corpus TD: TorpusTD:20930 TD:209928077
Bengio, Y., Léonard, N., and Courville, A. C. Estimating or propagating gradients through stochastic neurons for conditional computation. CoRR, abs/ 3303.333, 2013, URL http: http::///axxxxx//aaaaae 0////3308.333
Chang, X. Li, Y., Oymak, S., and Thrampoulidis, C. Provable Benefits of Overparameterrizatio in Model Compression: From Double Descent to Prun- ing Neural Networks. Proceedings of the AAAI Con- ference on Artificial Intelligence, 35(8):6974-699) May 2021. ISSN 2374-3468. aaai. 1685.. URL
doi: 10.1609/
In conclusion, our work demonstrates that sparse event- driven accelerators, such as neuromorphic processors, can provide state-of-the-arr art accuracy on high- frequency signal processing tasks, with orders of magnitude gains in latency and energy efficiency. This possibility opens up several research directions to further materialize these gains in real- world applications. In particular, future work should investi- gate how the -errennc-ppe Pareto front scales up to larger models and more complex tasks, such as language and multimodal modeling. In this setting, the scalability of multi-chip neuromorphic processors (Kudithipudi al., 2025) and high- frequeeyyy execution could power the grow- ing need for large-scale inference compute (Snell et al., 2024). Finally, improvements to our fixed-point conversion methodology and the use of advanced data types (e.g. FP8), could help close the gap between simulation and hardware deployment.
httexxpppp/.AA//atttccce//////// ber: 8.
Num-
Chen, T., Frankle, J., Chang, S., Liu, S., Zhang, Y., Wang, Z., and Carbin, M. The lottery ticket hypothesis for pre- trained bert networks. Advances in neural information processing systems, 33: 13:15834-15846 2020.
Chilkuri, N., Hunsberger, E., Voelker, A., Malik, G., and Eliasmith, C. Language modeling using Imus: 10x better data efficiency or improved scaling compared to trans- formers. arXiv preprint arXiv:2110.02440,, 2021.
Dauphin, Y. N., Fan, A., Auli, M., and Grangier, D. Language modeling with gated convolutional networks. In Precup, D. and Teh, Y. W. (eds.), Proceedings of the 34th International Conference on

---

Accelerating Lincar Recurrent Neural Networks for the Edge with Unstructured Sparsity
Machine Learning, ICML 2017, Sydney, NSW, Aus- tralia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 933-941. PMLR, 2017. URL http:/ / 270/dauphhe hta.//uupinnaaa.tm.... 7a.html.
Gu, A., Dao, T., Ermon, S., Rudra, A., and Ré, C. Hippo: Recurrent memory with optimal polynomial projections. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL https::/prroceedngg..eeee paper/2020/haret 102f0bb6ffbaac112233750000 6729bb Abstract.html.
Dubey, H., Aazami, A., Gopal, V., Naderi, B., Braun, S., Cutler, R., Ju, A., Zohourian, M., Tang, M., Golestaneh, M., et al. Icassp 2023 deep noise suppression challenge. IEEE Open Journal of Signal Processing, 2024.
Evci, U., Gale, T., Menick, J., Castro, P. S., and Elsen, E. Rigging the Lottery: Making All Tickets Winners. In Proceedings of the 37th International Conference Machine Learning, pp. 2943-2952 PMLR, November iig.eeeeeenngnn. m.mmml r.press/
Gu, A., Goel, K., Gupta, A., and Ré, C. On the parameteri- zation and initialization of diagonal state space models. Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Infor- mation Processing Systems 35: Annual Conference Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022a. URL http: htttt:htappeeeeeeee pager_filee//ppee///22///aa http:/ppppprss...p..o.o e9 eaa2faadee119006ee88833111444449 Abstract-CCofeeeecceett
2020. URL htwps::tp/t..tc...tp w119/evvc22aa.tmll ISSN: 2640-3498.
Fedus, W. Zoph, B., and Shazeer, N. Switch Transform- ers: Scaling to Trillion Parameter Models with Sim- ple and Efficient Sparsity, June 2022. URL http: //arxiv.org/ abs [es].
http://rg//abe/21.035666.5 2atp:///21..31.111 arXiv:2101.03661
Frankle, J. and Carbin, M. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint 1803.0080 2018.
Gu, A., Goel, K., and Ré, C. Efficiently modeling long sequences with structured state spaces. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenRe- view.net, 2022b. URL https: /ttp///eeeeeeee......./e/
Gholami, A. Kim, S., Dong, Z., Yao, Z., Mahoney, M. W., and Keutzer, K. A Survey of Quanti- zation Methods for Efficient Neural Network Infer- ence, June 2021. URL http: /ttt:://arxiie /atrxi..orrg/////.a//// abs/ 21 2103.13630. arXiv:2103.13613 [es].
Han, S., Pool, J., Tran, and Dally, W. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015.
Goel, K., Gu, A., Donahue, C., and Ré, C. It's raw! audio generation with state-space models. In Chaud- huri, K., Jegelka, S., Song, I.., Szepesvári, C., Niu, G., and Sabato, (eds.), International Conference on Ma- chine Learning, ICML 2022, 17-23 July 2022, Balti- more, Maryland, USA, volume 162 of Proceedings of Ma- chine Learning Research, pp. 7616-7633. PMLR, 2022. URL www.httts://// http::sss//pproeeennn....om htwp://ssw........./eee......eee r.pprss/v162// goe122a.html.
Hao, X., Ma, C., Yang, Q., Tan, K. C., and Wu, J. When au- dio denoising meets spiking neural network. 2024 IEEE Conference on Artificial Intelligence (CAI), pp. 1524-1527, 2024. doi: 10. 10.1109/CA159889.20000
He. X. O. Mixture of A million experts. CoRR,
his/2407.04153. 2024a.
doi: 10,48550/
ARXIV 224.7.4074533
URL https://doi.org/
10.4 10.4550/arxiv2447.0111
Golubeva, A., Gur-Ari, G., and Neyshabur, B. Are wider better given the same number of parameters? October 2021. URL https: httpss://ppereeeee http:///opneeeeenee//// Eorum?1d=.2x80ke. 9eF.
He. X. O.
Mixture of A Million Experts, July 2024b.
URL http: / /tt///aaxcaa /ttp/aaiw.oor///oa//////../o.llt abs /btp////////2000555...
arXiv:2407.04153 [cs].
Google. Aqt: Accurate quantized training. https:
Hubara, I., Courbariaux, M., Soudry, D., El-Yaniv, R., and Bengio, Y. Quantized Neural Networks: Training Neural
//github.com/ caaoom/charlii htt.m//haaeeeeeeeeeemmemmmm pestse///roocectttttttt 2024.
Networks with Low Precision Weights and Activations. Journal of Machine Learning Research, 18(187)11-00, 2018. ISSN 1533-7928. URL http: / jawr.org//rr/o//p papers/ /vrrssvll 1 6 htmm.
Gröchenig, K. Foundations of frequueyy analysis.
Springer Science Business Media, 2013.
10

---

Accelerating Lincar Recurrent Neural Networks for the Edge with Unstructured Sparsity
Key, O., Ribar, L.., Cattaneo, A., Hudlass-Galley, L., and Orr, D. Approximate top-k for increased paral-
machine learning. CoRR, (wt.1/1111..2888 2019. URL http://arxiv.orr abs 91 911.02385.
lelism. CoRR, abs/2412.04358,, 2024. doi: 10.48550/ ARXIV.2412.04338. URL dttps://doi.or
Meyer, S. M., Weidel, P., Plank, P., Campos-Macias L., Shrestha, S. B., Stratmann, P., and Richter, M. A di- agonal structured state space model on loihi 2 for ef- ficient streaming sequence processing. arXiv preprint arXiv:2409. /5022, 2024.
Kudithipudi, D., Schuman, C., Vineyard, C. M., Pandit, T., Merkel, C., Kubendran, R., Aimone, J. B., Orchard, G., Mayr, C., Benosman, R., Hays, J., Young, C., Bartolozzi, C., Majumdar, A., Cardwell, S. G., Payvand, M., Buck- ley, S., Kulkarni, S., Gonzalez, H. A., Cauwenberghs, G., Thakur, C. S., Subramoney, A., and Furber, S. Neuro- morphic computing scale. Nature, 637(8844):801-811 January 2025. ISSN 1476-4687. doi: 10.1038/s41586. 024-08253-8. URL https: http://wwwwtt..com/////// articles/s4ie 11586 122408253--
Mirzadeh, S. I., Alizadeh-Vaiie K., Mehta, S., del Mundo, C. C.,, Tuzel, O., Samei, G., Rastegari, M., and Farajtabar, M. ReLU strikes back: Exploiting activation sparsity in large language models. In The Twelfth International Conference on Learning Repre- sentations, 2024, URL https: Forum?id=osoWWXYB0r / //pp//eeeeeeeee...
Lee, J. H., Park, W., Mitchell, N., Pilault, J., Obando- Ceron, J. S., Kim, H., Lee, N., Frantar, E., Long, Y., Yazdanbakhsh, A., Agrawal, S., Subramanian, S., Wang, X., Kao, S., Zhang, X., Gale, T., Bik, A., Han, W., Ferev, M., Han, Z., Kim, H., Dauphin, Y. N., Dziugaite, K., Castro, S., and Evei, U. Jaxpruner: A concise library for sparsity research. CoRR, 1ew23044140822 2023. doi: 10.48550/ARXXI 24082 URL https: //doi.org/1000 hwtp:/.11.4.55111 www.4//55////1.13333.2 200oloorrg0044556//2222222444400
Mishra, A., Latorre, J. A., Pool, J., Stosic, D., Stosic, D., Venkatesh, G.. Yu, C.,, and Micikevicius, P. Ac- celerating sparse deep neural networks. arXiv preprint arXi::2104.08374 2021.
Mocanu, D. C., Mocanu, E., Stone, P., Nguyen, P. H., Gibeseu, M., and Liotta, A. Scalable training of ar- tificial neural networks with adaptive sparse connec- tivity inspired by network science. Nature Commu- nications, 9()):3383, June 2018. ISSN 2041-1723, doi: 10. 10.10388/411447-018--4433 0008--0436-22 dbi.org/10.1038//50
J. and Alvarez, R. On the quantization of recurrent neural networks, January 2021. URL http:/ arxiv.org/ abs/2101.05455 arXiv:2101.0544 [es].
URL https://
Modha, D. S., Akopyan, F., Andreopoulos, A., Appuswamy, R., Arthur, V., Cassidy, A. S., Datta, P., DeBole, M. V.. Esser, S. K., Otero, C. O., Sawada. J., Taba,
Lie, S. Cerebras architecture deep dive: First look in- side the hardware/software co-design for deep learn- ing. IEEE Micro, 43(3):18 30, 2023. doi: 10.1109/ MM.2023.32563844
B., Amir, A., Bablani, D., Carlson, P. J., Flickner, M. D., Gandhasri, R., Garreau, G. J., Ito, M., Klamo, J. L.,, Kusnitz, J. A., McClatchey, N. J., McKinstry, J. L., Nakamura, Y., Nayak, T. K., Risk, W. P., Schle- upen, K., Shaw, B., Sivagnaname, J., Smith, D. F., Terrizzano, I., and Ueda, T. Neural inference at the frontier of energy, space, and time. Science, 382 (6668):329-335, 2023. doi: 10. 1.111//ciencc.aadh URL
Liu, S. and Wang, Z. Ten lessons we have learned in the new "sparseland": A short handbook for sparse neural network researchers. CoRR, abs/2302.02596, 2023. doi: 10.48550/ARXIV WwW.:/AARXXX..22222266666 URL https:
Lu, C., Schroecker, Y., Gu, A., Parisotto, E., Foerster, N., Singh, S., and Behbahani, F. M. P. Structured state space models for in-context reinforcement learning. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances Neural Information Processing Systems 36: Annual Conference Neural Information Processing Systems 2023, NewrIPS 2023, New Orleans, LA, USA, December 10 2023, 2023. URL http::/pppers..ip..... ps.cc/
/tw//////////////oo abs/
10.1126/ 10.1126/scienceee http:/cccennnccchh1....
Mukherji, R., Schöne, M., Nazeer, K. K., Mayr, C., Kappel, D., and Subramoney, A. Weight sparsity complements activity sparsity in neuromorphic language models. arXiv preprint 0oXiv:2405000433 2024.
Nguyen, E., Poli, M., Faizi, M., Thomas, A. W., Wornow, M., Birch-Sykes, C., Massaroli, S., Patel, A., Ra- bideau, C. M., Bengio, Y., Ermon, S., Ré, C., and Baccus, S. Hyenadna: Long-range genomic sequence modeling at single nucleotide resolution. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances Neural Information
paper.fflees/ppee///22///aat 948522880221c33333222213333333 Mbstract-Conferenceee
Mayr, C., Höppner, S., and Furber, S. B. Spinnaker A 10 million core processor system for brain simulation and

---

Accelerating Lincar Recurrent Neural Networks for the Edge with Unstructured Sparsity
Processing Systems 36: Annual Conference on Neu- ral Information Processing Systems 2023, NeurlPS
Reddy, C. K., Dubey, H., Gopal, V., Cutler, R., Braun, S.,
Gamper, H., Aichner, R., and Srinivasan, S. lcassp 2021 deep noise suppression challenge. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6623-6627. IEEE, 2021a.
2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL http:/ sttp:///apprr.nips.co saper_files/ppaerre https/paaaer//22//a/h// 86ab6927ee4ae (Gabf227eeaeebbee227777753333333 Centract-Cooferenceee
Reddy, C. K., Dubey, H., Koishida, K., Nair, A., Gopal, V., Cutler, R., Braun, S., Gamper, H., Aichner, R., and Srinivasan, S. Interspeech 2021 deep noise suppression challenge. arXiv preprint arXh:: 2021b.
O'Connor, P. and Welling, M. Sigma delta quantized net- works. arXiv preprint arXiv:1611.02024 2016.
Orchard, G., Frady, E. P., Rubin, D. B. D., Sanbom, S., Shrestha, S. B., Sommer, F. T., and Davies, M. Efficient neuromorphic signal processing with loihi 2. In IEEE Workshop on Signal Processing Sys- tems, SiPS 2021, Coimbra, Portugal, October 19- 21, 2021, pp.
Renda, A., Frankle, J., and Carbin, M. Comparing rewinding and fine-tuning in neural network pruning, arXiv preprint arXiv:2003.02::8 2020.
Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q. V., Hinton, G. E., and Dean, J. Outrageously large neural networks: The sparsely-gated mixture-of- experts layer. In 5th International Conference on Learn- ing Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenRe- view.net, 2017. URL https: /ppp/reviee.nee// forum id=B1ckkMqlig
IEEE, 2021. doi: 10.1109/
SIPS52927.2021..0000 URL https: //oi.orr////////// 10.1 00.100//1PP52297.2222...
Orvieto, A., De, S., Gulcehre, C., Pascanu, R., and Smith, S. L. Universality of linear recurrences followed by non- linear projections: Finite-width guarantees and benefits of complex eigenvalues. In ICML, 2024. URL https: // //oppnneevew..eer...eeee
Shrestha, S. B., Timcheck, J., Frady, P., Campos-Macias, L., and Davies, M. Efficient video and audio processing with loihi 2. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 13481-13485. IEEE, 2024a.
Pei, J., Deng, L., Song. S., Zhao, M., Zhang, Y., Wu, S., Wang, G., Zou, Z., Wu, Z., He, W., Chen, F., Deng, N., Wu, S., Wang. Y., Wu, Y., Yang. Z., Ma, C., Li, G., Han, W., Li, H., Wu, H., Zhao, R., Xie, Y., and Shi, L. To- wards artificial general intelligence with hybrid tianjic chip architecture. Nature, 572(7767):106-114 Aug 2019. ISSN 1476-4687. doi: 10. 10.10088/411886-019-4422 URL htttps://oo..oog111110000 1424-8.
Shrestha, B., Timcheck, Frady, P., Campos-Macias, L., and Davies, M. Efficient Video and Audio Processing with Loihi 2. In ICASSP 2024 2024 IEEE Internu- tional Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 13481-13485, April 2024b. doi: 10.1109/ICASSP444455 2024. 2024..00000224..0 URL https:://eeexp: http://eeeeeplorreeeeeerer//bbbe document/10448003.
Pierro, A. and Abreu, S. Mamba-ptq: Outlier channels in re- current large language models. International Conference on Machine Learning Workshops, 2024.
Smith, J. T. H., Warrington, A., and Linderman, S. W. Simplified state space layers for sequence modeling. In The Eleventh International Conference on Learn- ing Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https:/ spenreviewwnet//pffP..eee A.W
Poli, M., Thomas, A. W., Nguyen, E., Ponnusamy, P., Deiseroth, B., Kersting, K., Suzuki, T., Hie, B., Ermon, S., Ré, C., Zhang, C., and Massaroli, Mechanistic design and scaling of hybrid architectures. In Forty- Machine Learning. ICML 2024, Vienna, Austria, July 21-27, 2024. OpenRe- view.net, 2024. URL https: /ttLtt/s///peeeeeieeo/o/ wpt.e/vieeeeeeeetttt.teee forum? ddd.pDDDGGddd..
first International Conference
Snell, C., Lee, J., Xu, K., and Kumar, A. Scaling LLM test-time compute optimally can be more effective than scaling model parameters. CoRR, abs/2408 00000000334,.. 2024. doi: 10.4 10.44850//RXII.24440 www.://....0..33.... 03314. URL https:// do..orgg10..855//rzzz.....
Reddy, C. K., Gopal, V., Cutler, R., Beyrami, E.,, Cheng, R., Dubey, H., Matusevych, S., Aichner, R., Aazami, A., Braun, S., et al. The interspeech 2020 deep noise suppres- sion challenge: Datasets, subjective testing framework, and challenge results. arXiv preprint arXiv:2005.1398) 2020.
Tay, Y., Dehghani, M., Abnar, S., Shen, Y., Bahri, D., Pham, P., Rao, J., Yang, L., Ruder, S., and Metzler, D. Long range arena : A benchmark for efficient trans- formers. In 9th International Conference on Learn- ing Representations, ICLR 2021, Virtual Event, Austria,
12

---

Accelerating Lincar Recurrent Neural Networks for the Edge with Unstructured Sparsity
May 3-7, 2021. OpenReview.net, 2021. URL https:
Zhang, J.-F,, Lee, C.-E., Liu, C., Shao, Y. S., Keckler, S. W., and Zhang, Z.. Snap: An efficient sparse neural acceler- ation processor for unstructured sparse deep neural net- work inference. JEEE Journal of Solid-State Circuits, 56 (2):636-677, 2021. doi: 10. 10.100995C..2220.3444333
Timcheck, J., Shrestha, S. B., Ben Dayan Rubin, D., Kupry- janow, A., Orchard, G., Pindor, L., Shea, T., and Davies, The intel neuromorphic dns challenge. Neuro- morphic Computing and Engineering, 33))034005,, aug 2023. doi: 10. 10.1008/2633444436acce7333 URL https: //tpp/do..rgg/.....1.. -3388///2663486/6/6// ace737.
Zhang. 7.,, Song, Y., Yu, G., Han, X., Lin, Y., Xiao, C.,, Song, C., Liu, Z., Mi, Z., and Sun, M. Relu² wins: Discovering efficient activation functions for sparse 1lms, 2024, URL https: /ttaps:/arxiinn /aaaaaaxivvooroo///eee/// abs 240.2/3844..4444
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, I.,, Gomez, A. N., Kaiser, I.., and Polosukhin, I. Attention is all you need. In Guyon, I., von Luxburg, U., Bengio, S., Wallach, H. M., Fergus, R., Vishwanathan, V. N., and Garnett, R. (eds.), Advances Neural Information Processing Systems 30: Annual Conference Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA. USA, pp. 1998-60088 2017. URL. htttp:://proceedings.. paper/2017/hash// fffs223334dde2111b03eeeeeeeeee Abstract.html.
Zhu, M. and Gupta, S.
To prune, not prune: Ex- ploring the efficacy of pruning for model compression. In 6th International Conference tations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3. 2018, Workshop Track Proceedings. OpenRe- view.net, 2018. URL https: http:///eeeeeeeeet.... Forum?id=Syl Forum?id=Sy11IDKRM
Learning Represen-
Voelker, A., Kajić, I., and Fliasmith, C. Legendre memory units: Continuous-time representation in recurrent neural networks. In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché- Buce F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL http:::/proceeiinsseeeei..ss paeer.fieee/aaeer////////L// 95225bbbreeaalee5a78848222222223 Paper.pdff
Warden, Speech commands:
dataset for limited-vocabulary speech recognition. CoRR, abs/1804.03209, 2018. URL http://arxiv.or/ abs/1804.03209.
Wu, H., Judd, P., Zhang, X., Isaev, M., and Mi- cikevicius, Integer Quantization for Deep Learn- ing Inference: Principles and Empirical Evalua- tion, April 2020. URL http: /ww.i.wwwioo///////.////.../..or /abs/ 2004.09602. arXiv:2004.09602 [cs, stat].
Wu, Y., Schuster, M.. Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., Klingner, J., Shah, A., Johnson, M., Liu, X., Kaiser,
L., Gouws, S., Kato, Y., Kudo, T., Kazawa, H.. Stevens, K., Kurian, G., Patil, N., Wang, W., Young, C., Smith, J.,
Riesa, J., Rudnick, A., Vinyals, O., Corrado, G., Hughes, M., and Dean, J. Google's Neural Machine Transla- tion System: Bridging the Gap between Human and Machine Translation, October 2016. URL http: arxiv.org/ abs abb./603..814... arXiv:1609.08144 [es].
13

---

The final linear projection for Accelerating Lincar Recurrent Neural Networks for the Edge with Unstructured Sparsity
A. Supplemental Material
Gated Linear Unit (GLU); The computation for the
GLU involves matrix multiplications for the dense weight matrix, followed by an element-wise multipli- cation:
A.1. Effective MACs computation for SS architecture
In this section, we detail the computation of effective multiply-acemmulate operations (MACs) for different com- ponents of the S5 architecture. The total MAC count pro- vides estimate of the computational cost associated with each stage of the model. Below, we outline the individual contributions from key components of the architecture. The effective MACs for all model sizes-sparse and dense-in Figure Figure are calculated based on the formulas below, summed over the entire network structure.
N wgt model GLU + Mooddel
(10)
Classification Head:
classification:
Noodel N Noutput 4
(wgt ject head pre_hread pre.hruut
(11)
The regression head follows the same computation as the classification head:
Regression Head:
Notation:
N Modell N Noatput (
pwyt pact beaa pre_hread
(12)
Ninput: Input dimension
Numerical operations such as the inverse square-root, sig- moid function, and others, are ignored from our MAC cal- culations, as is commonly done when calculating the MACs or floating point operations (FLOPs) of machine learning models (Evci et al., 2020).
NModde:: Model dimension for activations outside of the linear RNN.
Nosmi Dimension of the linear RNN's hidden state.
Noutpu: Output dimension (equal to the number of classes for classification)
A.2. Experimental Details
Wgt
Density of weights for
Model architecture Our linear RNN is based on the S5 architecture (Smith et al., 2023), as described in Section Section 2. We use the following dimensions for our base model with width scaling & 1 (i.e. configuration 4 in Figure 4). We use three layers, the recurrent state vector x, E 256 we use a model dimension of 192. Both input and output have dimension 257. The width scaling factors k; scale the model and recurrent state dimension linearly. In Figure 4, we report results for k-family of sparse and densely trained networks where ksparse E [0.5, 3.0], kaense E [0.25, 1.0].
pet. 2
Density of activations for x
where the density d is calculated from the sparsity
d =
1=
Breakdown of MAC Calculation per Component:
The MACs for the encoder depend on the input dimension, model size, and scale linearly with activation and weight densities:5
Encoder:
Training recipe We trained all models for 50 epochs with the Adam optimizer. The parameters of the SSM block were updated with initial learning rate 0.002, while the rest of the architecture used initial learning rate 0.008 and weight decay 0.04. All learning rates used cosine annealing and no warmup epochs. The dropout was set to 0.1.
Weg ( Pinput Nanp N Vmndee encoder
input
Batch Normalization (BatchNorm): A lightweight operation, requiring only element-wise scaling, leading to:
Nmudel
(7)
A.3. Additional Results
SS Hidden Layer:
The hidden state update for the S5 model involves both matrix multiplications and element-wise operations:
1.3.1. KEYWORD SPOTTING
We extended our experiments by applying the proposed scaling protocol the keyword spotting task of the Speech- Commands V2-35 dataset (Warden, 2018). The results, reported in Figure 9, exhibit a similar trend to that observed on the N-DNS dataset. Sparse models are more efficient while reaching the same level of accuracy. However, further scaling of the sparse model family would be required to compare against dense models at higher accuracy.
2N madee Nom de
jugt
gact sam
+
4N
(8)
SSM Output Layer: Computes the output transforma- tion of the linear RNN:
2. 2Nesm N d wigt pact
N Ninodelld jact
ssmm
14

---

Accelerating Lincar Recurrent Neural Networks for the Edge with Unstructured Sparsity
W Weight matrix
Recurrent state
Input A ccumulator
Ethernet
Activation funcs., regularization, normalization
Loihi neuron
Spike output
Host
X Ntayers
Seep (
Siggnau
W
BatchNomm
Encodar
ReLU
ReLU
ReLU
Decoder
diag(A)
imag B
imag x
imag (C)
ReLU
real
ReLD
real(
real
diag(A)
Figure 8. Diagram of S5 as implemented on Loihi 2. To leverage the neuromorphic hardware architecture, several adjustments are made in comparison the original model shown Figure First, complex numbers are split into real complex components for processing. Second, ReLUs are introduced to increase activation sparsity. Third, multiple element-wise operations are fused into single neuromorphic neurons. Symbols are shown as defined in Section
A.3.2. LOTHT EXECUTION MODE
static quantization step. Further investigations show that the mismatch between hidden activations is highest for the hidden states X₃ of the linear RNN and its outputs Yk. see Figure 11. This mismatch increases approximately linearly with model depth, indicating that quantization errors ac- cumulate as information propagates through the network layers. This linear escalation of errors underscores a critical challenge in fixed-point quantization of recurrent models (Wu al., 2016; Abreu et al., 2024; Li & Alvarez, 2021: 1; Pierro & Abreu, 2024). Consequently, ensuring the fidelity of deeper Linear RNNs on fixed-point neuromorphic hard- may require advanced quantization techniques mitigation strategies to preserve the network's 's temporal dy- namics and memory capacity effectively.
Loihi 2's asynchronous architecture allows to trade off between throughput and latency, as illustrated in Figure 10a. For optimal throughput, new input provided every time step and forwarded through the neuronal layers in pipelined mode. For optimal latency, only once the previous input has been processed by, or fallen through, the network as fast as possible. The pipelined and fall-through mode can be balanced by changing the rate of new input, to match the throughput of given input stream while minimizing its processing latency.
input is injected
As audio denoising is typically deployed in realtime in online fashion where STFT input frame in processed at time, fall-through mode is appropriate, one desires corresponding output STFT frame immediately.
We see that Loihi 2 processes a single STFT frame 35x faster and with 1200x less energy than the Jetson Orin Nano by-toen Loihi Fall- Through and Jetson Orin Nano Recurrent 1-step (b=1) in Table 1).
A.3.3. BOXED-POITY MODEL MISMATCH
The mismatch in Figure 6 indicates that fixed-point imple- mentation in JAX does not perfectly match the original FP32 model when using the scales computed through our
15

---

Accelerating Lincar Recurrent Neural Networks for the Edge with Unstructured Sparsity
97
96.5
Test SI-SNR (dB)
96
95.5
Dense GELU
Sparse w/ ReLU
95
0
2
3
4
5
Effective MACs (x10*)
Figure Pareto fronts for S5 network test accuracy a function of effective compute on SpeechCommands V2-35 keyword spot- ting task. networks with weight activation sparsity (green) exhibit a domain of Pareto optimality versus dense S5 networks (orange). Number annotations enumerate incrcasing S5 dimen- sionality configurations. Further scaling of the sparse architectures would be required to compare with the dense models at higher accuracy.
16

---

Accelerating Lincar Recurrent Neural Networks for the Edge with Unstructured Sparsity
Pipelined mode
25
Throughpul
Fall-Through
Pipelined
Input
Layer 0
20
Layer 2
Time per Step (us)
15
Layer
Latency
Fall-through mode
10
Throughput
Imput
5
Layer 0
Layer 2
0
10
20
30
40
50
Layer n
Latency
Step
Figure 10. (a) Loihi offers processing modes that optimize either throughput latency. In the pipelined mode, new data point inserted each time step, to use all processing cores and maximize the throughput-an at the expense of latency because equal time bins data points is only provided once the last data point has been fully processed
l1 ... enforced. In the fall-through mode, with minimum latency. Only single neuronal layer is active at any step as data travels through the network. The time per step is thus minimized traffic reduced and potentially complex neuronal layers not updated. (b) Comparison of execution mode and time per step.
Average by block for fixed-point model
127
107
10
0.00
-0.25
S100Y 3000%
1300%
100%
32"
700%
1011
Figure 11. Lase analysis of mismatch between the point model JAX against the base model using floating-pott point weights and activations. The left and right side show the same data with lincar y-axis and log y-axis, respectively. The top panels show the mean absolute error N Σ. las xi| for all components of the model while the bottom panels show the mean relative error N N (4) se (J).....N au [x; xf1/ (as].. For further explanation, see text.
17


Reviews:
[{"review_id": "RrXnMoQ2V7", "reviewer": "Reviewer_VTiJ", "summary": "The paper presents a method to accelerate the computations of linear Recurrent Neural Networks (RNNs) using unstructured sparsity for edge computing applications. This work is motivated by a case study showing that highly sparse linear RNNs achieve superior efficiency-performance trade-offs compared to dense baselines. The paper particularly highlights the deployment of sparse linear RNNs on the Intel Loihi 2 neuromorphic processor, where quantized models demonstrate significant reductions in latency (~42×) and energy consumption (~149×) compared to edge GPUs.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We gratefully thank the reviewer for their effort and feedback, and we would like to address their questions. We will be happy to hear and address any further feedback.\n\n### Generalization to different accelerators\nReviewers VTiJ and jRFp raised the need for a discussion about the generalizability of our methodology and benchmarking to other hardware platforms (e.g., SpiNNaker 2 and IBM NorthPole). While we adopted a hardware-aware approach focused on Loihi 2, we believe that platforms with similar feature sets can benefit from it. However, since most neuromorphic processors are currently research prototypes, they don’t provide public access (like IBM NorthPole) and/or lack a high-level programming framework that would allow the transfer of models from one platform to another. Following the approach proposed in NeuroBench [1], we will publicly release our training pipeline and checkpoints to enable the implementation on other neuromorphic processors by the relevant experts.\n\nWe further address the generalizability discussion in the following paragraph, which we plan to add to Section 4 in the camera-ready version.\n\n*While the proposed hardware-aware methodology is tailored to leverage the Loihi-specific feature set, we believe that platforms with similar characteristics could potentially benefit from the results we presented. Neuromorphic processors such as SpiNNaker 2 [2] and IBM NorthPole [3], which presents similar architecture patterns to Loihi 2, would greatly benefit from our methodology on activation and weight sparsity. In addition, other platforms with tight compute-memory integration, such as Cerebras Wafer-Scale Engine (WSE-3) [4], provide support for unstructured sparsity, even if targeting datacenter-scale applications.*\n\n### Generalization to different tasks\nReviewers VTiJ and jRFp noted that extending our benchmarking to other datasets would strengthen our claims and the relevance of our work for the broader edge inference community. Starting from the S5 baseline [5] and following our methodology, we are running experiments on the keyword spotting SpeechCommands V2-35 (SC35) dataset [6], which provides a common real-time use case for the edge. The preliminary results, available at https://figshare.com/s/3d0ba8e3f515535871ba, compare the training curves for a narrow dense and wider sparse model (90% sparse weights and ReLU activations) with similar inference MACs. The plot shows that the sparse model (still under training) is on track to match or exceed the accuracy of the dense counterpart. Similar experiments are underway across inference compute budgets, and we plan to add such results in the same form as Figure 4 to the camera-ready version. Since our methodology seems to generalize without modification to SC35, we expect it to also generalize to other sequence modeling applications commonly solved with linear RNNs, such as bio-marker signal monitoring [7], time series forecasting [8], or action recognition [9].\n\n#### References\n1. Yik, Jason, et al. \"The neurobench framework for benchmarking neuromorphic computing algorithms and systems.\" Nature Communications 16.1 (2025): 1545.\n2. Mayr, Christian, Sebastian Hoeppner, and Steve Furber. \"SpiNNaker 2: A 10 million core processor system for brain simulation and machine learning-keynote presentation.\" Communicating Process Architectures 2017 & 2018. IOS Press, 2019. 277-280.\n3. Modha, Dharmendra S., et al. \"Neural inference at the frontier of energy, space, and time.\" Science 382.6668 (2023): 329-335.\n4. Lie, Sean. \"Cerebras architecture deep dive: First look inside the hardware/software co-design for deep learning.\" IEEE Micro 43.3 (2023): 18-30.\n5. Smith, Jimmy TH, Andrew Warrington, and Scott W. Linderman. \"Simplified state space layers for sequence modeling.\" arXiv preprint arXiv:2208.04933 (2022).\n6. Warden, Pete. \"Speech commands: A dataset for limited-vocabulary speech recognition.\" arXiv preprint arXiv:1804.03209 (2018).\n7. Pimentel, Marco AF, et al. \"Toward a robust estimation of respiratory rate from pulse oximeters.\" IEEE Transactions on Biomedical Engineering 64.8 (2016): 1914-1923.\n8. Schirmer, Mona, et al. \"Modeling irregular time series with continuous recurrent units.\" International conference on machine learning. PMLR, 2022.\n9. Kuehne, Hildegard, et al. \"HMDB: a large video database for human motion recognition.\" 2011 International conference on computer vision. IEEE, 2011.", "claims_and_evidence": "The paper supports its claims about the efficiency and performance benefits of unstructured sparsity in linear RNNs by reporting its implementation results from the Intel Loihi 2 chip. The results demonstrate significant reductions in latency (42×) and energy consumption (149×) compared to an edge GPU, supported by benchmarks on real-world tasks like audio denoising.", "methods_and_evaluation_criteria": "Yes, the proposed methods and evaluation criteria align well with the problem of accelerating linear RNNs for edge applications. The Intel N-DNS Challenge benchmark is a good test case for audio denoising, and the comparison between sparse vs. dense models on Loihi 2 vs. Jetson Orin Nano provides is interesting.", "theoretical_claims": "There is not much theory in this work.", "experimental_designs_or_analyses": "Yes, the experimental design appears OK specially in evaluating sparsity effects, quantization, and hardware acceleration on Loihi 2 vs. Jetson Orin Nano. The Pareto front analysis shows interesting trade-offs between accuracy, compute efficiency, and memory.", "relation_to_broader_scientific_literature": "The paper builds on prior works in sparse models and neuromorphic computing. The main broader impact of this work is its validation on neuromorphic hardware for streaming tasks, pushing the field toward practical edge AI deployment.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n\n-- Low-latency, memory-efficient implementation of AI models is highly important for real-time streaming applications, which motivates the research conducted in this paper.\n\n-- The authors empirically validate their claims by studying Pareto efficiency trade-offs across different compute budgets.\n\n-- The integration of neuromorphic processors for accelerating sparse models is very interesting. It provides practical results which supports their simulation results.\n\n\nWeaknesses:\n\n-- While Loihi 2 is well-suited for sparse and event-driven computations, it remains unclear how these optimizations would translate to conventional customized hardware. How does it compare to customized accelerators for sparse or quantized models?\n\n-- The focus of this work is on audio denoising. How does the proposed method could generalize to other domains (e.g., NLP and vision applications).", "other_comments_or_suggestions": "N/A", "questions_for_authors": "See my points listed as weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XZhmGsZbU8", "reviewer": "Reviewer_jRFp", "summary": "This paper explores unstructured sparsity in linear recurrent neural networks (RNNs) to improve efficiency in edge AI applications, particularly when deployed on neuromorphic hardware (Intel Loihi 2). The authors examine various model compression techniques and conduct a scaling study to determine the Pareto front of performance vs. efficiency. The study highlights unstructured sparsity as a viable strategy for real-time processing in low-power edge devices.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We gratefully thank the reviewer for their effort and feedback. We would like to address their questions, and we’d appreciate any raise in score if our arguments convince them. We will be happy to hear and address any further feedback.\n\n### Generalization to different accelerators\nPlease refer to our response to Reviewer VTiJ.\n\n### Generalization to different tasks\nPlease refer to our response to Reviewer VTiJ.\n\n### Impact of batch processing\nWe appreciate the reviewer’s question regarding the impact of batching on our benchmarking results. We address it in the following paragraph, which we plan to include in Section 3.3 of the camera-ready version.\n\n*Although edge applications are typically thought of as single-batch applications, some workloads at the edge require small-batch inference, e.g., de-noising audio streams from multiple on-device microphones. For this reason, it is important to investigate how batch processing affects latency and energy efficiency for the two hardware architectures. While Intel Loihi 2 doesn’t natively support batching in the sense of processing multiple independent samples though the same model instantiation, the parallel inference of independent sequences can be obtained by replicating the model on the chip as many times as required by batch size. We extended the results in Table 1 to compare the effect of this implementation of batching to the usual batch processing of the Jetson Orin GPU. Figure X (available at https://figshare.com/s/4d6035c9a2c4739d7201) shows total latency and energy per sample across batch sizes, from 1 to 32. The results demonstrate that our approach on Loihi 2 maintains its large latency advantage while the energy efficiency gain is only maintained for the small-batch regime (below 8 samples), which is typical of edge inference applications. As expected, batch processing improves energy efficiency for the GPU, since the cost of data movement associated with loading the model is offset by the parallelized evaluation of multiple samples.*\n\n### Novelty and originality of the work\nWe thank the reviewer for pointing out that we missed to express the novelty and originality of our work. We also appreciate their statement that our work “can be a good reference for the deployment of linear RNN models on edge devices”. In addition, we would argue that our paper has two further main original/novel contributions.\n\nFirst, this paper spearheads the idea that neuromorphic processors are an ideal platform for the emerging class of linear RNNs. This is particularly due to the tight integration of massively parallel compute and memory in neuromorphic hardware, which can efficiently update stateful recurrent neurons. Neuromorphic processors are typically designed for low-latency processing of sequentially incoming sensory signals, and can thus particularly benefit from the advantageous scaling trends of linear RNNs with long sequences. Our present paper is the first peer-reviewed publication that explores the combination of model compression techniques required to exploit the synergy between neuromorphic processors and linear RNNs.\n\nSecond, while we agree that we applied off-the-shelf compression techniques, we would argue that our paper combines them into a novel training recipe necessary to leverage the specific features typical for neuromorphic computing for an optimized deployment of linear RNNs. We would like to kindly point out that, according to the 2025 ICML reviewer guidelines, “Originality need not mean wholly novel methods. It may mean a novel combination of existing methods to solve the task at hand, a novel dataset, or a new way of framing tasks or evaluating performance so as to match the needs of the user.” Our new training and deployment recipe has proven to bring tangible performance and energy advantages on real hardware for a real-world task that requires low-latency, low-power execution, audio denoising. We will release this new training pipeline upon acceptance so that the community can extend it or apply it to additional applications.\n\nWe will update the introduction in the camera-ready version accordingly.", "claims_and_evidence": "The claim of the findings in this work can generalize to edge AI beyond neuromorphic hardware is questionable. Only one neuromorphic chip (Loihi 2) and one task (audio denoising) was tested.", "methods_and_evaluation_criteria": "It makes sense that the performance is measured using Scale-Invariant Signal-to-Noise Ratio (SI-SNR), compute (MACs), memory usage, latency, and energy consumption. Results are compared with dense models running FP32 on a Jetson Orin Nano. However, no evaluation on other neuromorphic processors (e.g., SpiNNaker 2, IBM NorthPole). Only one downstream task is evaluated.", "theoretical_claims": "No theoretical claims are described in the paper.", "experimental_designs_or_analyses": "See \"Methods And Evaluation Criteria\".", "relation_to_broader_scientific_literature": "The paper builds on prior work in structured state-space models (S5), model compression, and neuromorphic hardware. The experimental results in this work can be good references for the deployment of linear RNN models on edge devices.", "essential_references_not_discussed": "I am not an expert in either linear recurrent neural networks or neuromorphic computing, so I cannot assess whether any relevant prior findings are missing from the paper.", "other_strengths_and_weaknesses": "Strengths:\n1. The investigation of deep learning model compression in neuromorphic computing has substantial real-world impact. \n2. The on-device experiment results are very beneficial for the community regarding edge AI deployment.\n\nWeaknesses:\n1. This work is more like an experimental report rather than an academic paper with solid intellectual contribution. The model and the compression methods are all off-the-shelf.\n2. The profiling of model compression on neuromorphic hardware is only conducted on a single device model (Intel Loihi 2), which weakly support the conclusion from the experimental results.\n3. Only a single task (audio denoising) is evaluated. A broader range of tasks need to be assessed to sufficiently support the conclusion for the general edge AI applications.", "other_comments_or_suggestions": "As a conclusion, I suggest the authors to make a further step in designing original methodology of model compression techniques for neuromorphic hardware in edge. The generalizability of the current results require more comprehensive experiments. The authors should also reconsider if they want to position this work specifically for \"neuromorphic hardware\" or even more general edge computating, which should be reflected correctly in the paper title.", "questions_for_authors": "1. Why was only Intel Loihi 2 used? Would the results generalize to other neuromorphic chips like SpiNNaker 2 or IBM NorthPole?\n2. Would the observed efficiency trends hold for other real-time tasks (e.g., NLP, speech recognition, or perception)?\n3. How does batch size impact efficiency? Could Loihi 2 be optimized for batch processing?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G78P2Q6HpC", "reviewer": "Reviewer_btkY", "summary": "This paper explores the efficiency-performance trade-offs of sparse linear RNNs through a scaling study. The models achieve SOTA results in real-time audio denoising. By quantizing and deploying them on the Intel Loihi 2 neuromorphic chip, the work significantly reduces latency and energy consumption compared to a dense model on an edge GPU.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We gratefully thank the reviewer for their effort and feedback. We would like to address their questions, and we’d appreciate any raise in score if our arguments convince them. We will be happy to hear and address any further feedback.\n\n### Clarification on Figure 6\nWe regret the suboptimal presentation of Figure 6 and appreciate the reviewer’s nudge to clarify. We confirm that FPX is a typo and should read FXP, for fixed-point. We will replace the caption with the following in the camera-ready version:\n\n*Base: trained a 32-bit floating-point model and applied post-training quantization without additional quantization-aware fine-tuning. QAT: trained the model with quantization-aware training using W8A16 quantization for the forward pass, as straight-through estimators for the backward pass. The results show that the Base model without QAT performs slightly better in FP32 than the QAT model, but significantly worse in static quantization and fixed-point precision. The model shown here is the sparse-6 variant, see Figure 4.*\n\n### Previous work on deployment of linear RNNs to neuromorphic devices\nSince the now broad interest in linear RNNs is relatively recent, there is limited previous work on deployment of this architecture to neuromorphic devices, and, to the best of our knowledge, no previous work on hardware-aware model compression in this context. Previous work on Loihi 2 demonstrated the implementation of S4D [1], a state space model based on the Legendre Delay Network [2], and a MatMul-free LLM [3]. \nWhile S4D could potentially target the same application domains as S5, Ref. [1] only reports benchmarks on simple synthetic tasks (sMNIST and sCIFAR) and without model compression. For this reason, a direct performance comparison is not possible. Ref. [2] implemented a state space model with a spiking neural network on a neuromorphic processor – also without model compression techniques. The MatMul-free LLM in Ref. [3] uses ternary weight matrices for model compression but doesn’t leverage sparsity advantages from the neuromorphic chip on which it is deployed, thereby making the comparison to our present work difficult.\nWe are not aware of any further deployment of linear RNNs on neuromorphic hardware.\n\n#### References\n1. Meyer, Svea Marie, et al. \"A Diagonal Structured State Space Model on Loihi 2 for Efficient Streaming Sequence Processing.\" arXiv preprint arXiv:2409.15022 (2024).\n2. Gaurav, Ramashish, Terrence C. Stewart, and Yang Yi. \"Legendre-SNN on Loihi-2: Evaluation and Insights.\" NeurIPS 2024 Workshop Machine Learning with new Compute Paradigms.\n3. Abreu, Steven, et al. \"Neuromorphic Principles for Efficient Large Language Models on Intel Loihi 2.\" arXiv preprint arXiv:2503.18002 (2025).", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes, they are correct.", "experimental_designs_or_analyses": "The presentation of Figure 6 lacks clarity. The experimental conditions for Base and QAT are not very clear. Please provide a detailed explanation of what Base represents in each experiment.", "relation_to_broader_scientific_literature": "This work emphasizes the suitability of linear RNNs for real-time, long-range sequence modeling on edge devices, while noting that their compression and acceleration remain underexplored. However, compression and acceleration of linear RNNs have not been thoroughly explored. Additionally, the paper notes that linear RNNs are a promising match for neuromorphic processors. The study explores sparsification and quantization of linear RNNs and deploys the compressed models on the Intel Loihi 2 neuromorphic chip for real-time processing", "essential_references_not_discussed": "No missing essential references were identified.", "other_strengths_and_weaknesses": "I think the paper is reasonably clear and well-supported by sufficient evidence. However, its organizational structure could be improved. For example, the section on Compressing Linear RNNs includes a significant amount of related work and background information, which could be streamlined without disrupting the logical flow. Additionally, the discussion of contributions is relatively weak and could be expanded. Furthermore, the section on hardware deployment could provide more details to enhance clarity.", "other_comments_or_suggestions": "Is FPX in Figure 6 actually intended to be FXP, meaning fixed-point?", "questions_for_authors": "Could you provide additional related works on deploying linear RNNs to neuromorphic devices, along with performance comparisons?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6NaSpUlCsD", "reviewer": "Reviewer_EUuh", "summary": "This paper explores compression techniques for linear RNNs, including unstructured sparsity and fixed-point quantization, and evaluates their acceleration on neuromorphic hardware. The study investigates the trade-offs between latency, energy, and accuracy compared to dense RNNs by introducing these compression techniques. The results demonstrate that highly sparse linear RNNs achieve improved efficiency-performance trade-offs, with 2× less compute and 36% less memory at iso-accuracy. Additionally, quantizing the sparse models and deploying them on the Intel Loihi 2 neuromorphic chip yields significant speed and energy improvements over edge GPUs.\n\n## Update after rebuttal\n\n(Was previously in official comment not visible by the authors)\nThanks for the clarification. It would be beneficial to see the extended results and ablation study in the appendix. For now I will keep the score as 3.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We gratefully thank the reviewer for their effort and feedback. We would like to address their questions, and we’d appreciate any raise in score if our arguments convince them. We will be happy to hear and address any further feedback.\n\n### Generalization to different architectures: preliminary results on HGRN\nTo expand our results to different linear RNN architectures beyond S5, we can present some preliminary results on training a 370M parameter language model based on the HGRN architecture [1]. Using the same iterative magnitude pruning setup as described in our paper, we trained the HGRN model on the FineWeb dataset for ~5B tokens with different target sparsity ratios from 0% to 90%. The training loss for the 90% sparse model is ~9% higher than for the dense model, see the Table below. Additional details are available in the Figure available at https://figshare.com/s/a8cd30c07af56f4ddbbf. These findings confirm a similar trend as what we reported in our paper on real-time sequence modeling tasks. More work is underway to scale up the language modeling experiments, and verify their validity on real-world datasets and benchmarks. \n\n|                        | Dense | 50%   | 80%   | 90%   |\n|------------------------|-------|-------|-------|-------|\n| Training loss          | 2.68  | 2.72  | 2.82  | 2.90  |\n| Relative loss change   | 0%    | +1.5% | +5.3% | +8.3% |\n| MACs (10^6)            | 341   | 187   | 95    | 64    |\n| Relative MAC change    | 0%    | -45%  | -72%  | -81%  |\n\n### Ablation study on quantization schemes and sparsity levels\nWe agree with the reviewer that exploring the large design space of quantization schemes and sparsity distributions would provide additional insights in this domain. On quantization, previous work on S5 [2] showed that, compared to W8A8, lower precision significantly impacted accuracy. Since Loihi’s message format doesn’t provide any performance advantage for 8-bit versus 16-bit activations, we selected W8A16 which preserves higher accuracy. Regarding sparsity, we selected a 90% target after some initial experimentation, as it resulted in a good trade-off between impact on task accuracy and MACs reduction. We could add an ablation study on sparsity levels to Appendix A, if the Reviewer finds it helpful for readers. We would also acknowledge these directions for future work by updating the last sentences of Section 4 with the following:\n\n*Finally, further exploration of quantization schemes and sparsification techniques could offer deeper insights into optimal model design for different hardware platforms. In particular, leveraging advanced data types for quantization (e.g., FP8) and adopting a more fine-grained selection of sparsity levels, potentially guided by iterative hardware profiling, are promising directions for future research.*\n\n#### References\n1. Zhen Qin, Songlin Yang, Yiran Zhong, “Hierarchically Gated Recurrent Neural Network for Sequence Modeling”, NeurIPS 2023 Spotlight.\n2. Abreu, Steven, et al. \"Q-S5: Towards quantized state space models.\" arXiv preprint arXiv:2406.09477 (2024).", "claims_and_evidence": "The paper claims that sparse linear RNNs provide better efficiency-performance trade-offs than dense baselines, with notable reductions in computation and memory usage. Furthermore, it states that quantized sparse models on Intel Loihi 2 outperform edge GPUs in terms of speed and energy efficiency. These claims are supported by experimental results, which show improvements in denoising quality while maintaining computational efficiency. The evidence presented in the paper generally aligns with these claims.", "methods_and_evaluation_criteria": "The proposed method is evaluated using the Intel Neuromorphic Deep Noise Suppression Challenge, which focuses on human speech denoising. The dataset is derived from the Microsoft DNS Challenge, containing clean human speech and noise source samples. The denoising quality is assessed using the scale-invariant signal-to-noise ratio (SI-SNR), which is an appropriate metric for this application. Given the embedded system context, the chosen evaluation settings are relevant and reasonable.", "theoretical_claims": "The paper does not make any theoretical claims.", "experimental_designs_or_analyses": "The experimental design is based on the S5 model built with JAX. JAXPruner is used for pruning the network to introduce unstructured sparsity, while the AQT library is used for quantization-aware training. The final network is deployed and evaluated on the Intel Loihi 2 neuromorphic chip. The experimental design appears valid, leveraging appropriate tools and methodologies to assess the impact of sparsity and quantization.", "relation_to_broader_scientific_literature": "The paper provides a sufficient literature review covering key topics such as linear RNNs, sparsity, model compression, and neuromorphic computing. The citations are relevant and provide necessary context for understanding the contributions of the work.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n* The paper successfully demonstrates an improved Pareto optimal front, achieving better denoising quality for the same multiply-accumulate operations (MAC) or reducing MAC while maintaining denoising quality.\n* The results show competitive performance compared to an edge GPU (Jetson Orin Nano) in terms of latency and energy efficiency.\nWeaknesses:\n* The study focuses primarily on a single design point: the S5 model on the Intel Loihi 2 chip. It does not explore how different variants of linear RNNs are affected by sparsity and quantization.\n* The paper examines model dimension versus denoising quality but does not sufficiently discuss other critical design parameters such as sparsity ratio and quantization format (beyond W8A16).", "other_comments_or_suggestions": "* It would be beneficial to evaluate how different linear RNN variants respond to sparsity and quantization to generalize the findings further.\n* Additional analysis of various sparse ratios and quantization formats could provide deeper insights into trade-offs and optimal configurations.", "questions_for_authors": "* Have you considered evaluating the impact of sparsity on other linear RNN architectures beyond the S5 model?\n* Can you provide insights into how different quantization formats (other than W8A16) might affect the performance and efficiency of the model?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper studies the Pareto frontier between performance and efficiency for unstructured sparsity in recurrent neural networks at the edge. Using the Intel Loihi 2 neuromorphic chip and quantization of the studied models, this translates even to an 149x lower energy consumption than dense neural networks. All reviewers found the work insightful, with possible real-world impact. After the rebuttal period, they unanimously lean towards the paper acceptance. Consequently, the AC suggests acceptance and recommends to the authors to include all rebuttal discussions and results in the final version of the paper.
Decision: Accept (poster)