Advancing Neuromorphic Computing Algorithms and Systems with NeuroBench

Venue: NeurIPS_cc_2024_Workshop_MLNCP
Published: 2024-09-11 23:49:32

Abstract: Neuromorphic research aims to approach the capabilities and efficiency of the brain by developing algorithm and hardware architectures which adopt mechanical features of biological computing. As improvements in computing speeds slow and new workloads scale towards untenable costs in conventional approaches, neuromorphic computing offers a novel paradigm for neural network intelligence and hardware architectures which may realize a next-generation of computing performance. Such a paradigm optimizes for different goals than conventional computing systems, and therefore novel benchmark assessment and methods are necessary for technological progress. In this article, we present NeuroBench, a benchmark framework for neuromorphic computing which has engaged over 100 researchers from more than 50 institutions across academia and industry. NeuroBench provides novel benchmark tasks and common, extensible tools for evaluation, unifying the space of neuromorphic research. Results from NeuroBench benchmarks and challenges highlight the strengths of neuromorphic algorithms and systems for sparse, event-driven, and energy-efficient machine learning.

Main paper: 
 introduction: 
 The brain, a computer evolved by nature, boasts intelligence capabilities and a minuscule energy consumption that are both far outside the realm of existing computing technologies. Approximating between consumed calories and mechanical power, the brain operates using a power budget in the order of 20W [1] , which could not begin to power any general-purpose AI system. For reference, one NVIDIA H100 has a thermal design power (TDP) of 700W [2] . As the energy expenditure of conventional AI computing accelerates without foreseeable end [3] , it is prudent to study techniques towards more energy-efficient AI.Neuromorphic research aims to approach biological capability and efficiency by developing algorithms and hardware systems which incorporate mechanical features of biological computation, such as sparsity, decentralized parallelism, and event-driven operation. A key algorithmic approach in neuromorphic AI research is the spiking neural network (SNN). SNNs feature stateful, binaryactivation, sparsely-activated neuron models, similar to biological neurons and unlike conventional artificial neural network (ANN) neuron models, which are typically stateless, continuous-activation, and densely-activated. Such features enable SNNs to be used for AI tasks with far lower effective operational costs than densely-calculated ANNs, while maintained comparable fidelity. Algorithmic research in SNNs continues to close the accuracy gap with ANNs across domains such as computer vision [4] and language processing [5] , demonstrating high task accuracy and theoretical efficiency. However, so far, SNNs have not yet been widely recognized on their potential for capable, highly efficient AI. This is because fundamentally, SNNs cannot realize theoretical efficiency on the con- ventional hardware systems that power current AI, such as GPUs. Such conventional systems are specialized to full-utilization for dense compute operation, but to realize efficiency utilizing the highly dynamic sparsity in SNNs, a different approach is needed. Thus, new types of neuromorphic hardware architectures (e.g., Intel Loihi [6] , SpiNNaker [7] , Synsense Xylo [8] , BrainScaleS [9] , to name a few) are being developed for co-designed algorithm+hardware systems which can exploit the sparse, event-driven features of neuromorphic algorithms for efficient deployed performance.In order for neuromorphic research to advance and realize next-gen performance and efficiency, standardized benchmarks must be used to objectively quantify and compare promising solutions. The neuromorphic approach represents a full-stack paradigm, which in turn warrants specialized benchmark tasks and measurement methodology. Classic deep learning benchmarks such as MNIST [10] ,CIFAR [11] , and ImageNet [12] have been commonly applied to neuromorphic research. However, the tasks focus on offline, batch processing of image frames, while neuromorphic algorithms and hardware are geared more towards real-time operation on natively temporal data, such as video or event-camera data.Furthermore, when measuring benchmark performance, conventional benchmarks will have a primary, often exclusive, focus on algorithmic correctness (accuracy) and system runtime [13, 14, 15] . In contrast, neuromorphic approaches are designed with efficiency concerns as a primary objective and must be benchmarked accordingly. Conventional benchmarks may use measures such as floatingpoint operations (FLOPs) to quantify compute costs, but this metric is geared towards dense compute throughput, and not sparse or event-based operation.While different benchmark tasks are used which better align with neuromorphic characteristics such as sparse, event-based time series data (e.g. N-MNIST [16] , Spiking Heidelberg Digits [17] , and DVS Gesture [18] ), standardized evaluation methodology for analyzing compute costs at the algorithmic and system levels are necessary. A complete benchmarking framework must unite benchmark tasks, which concern task accuracy, with measurements that account for the final efficiency and cost of a solution.To this end, this article presents recent advancements with the NeuroBench benchmark framework [19] . NeuroBench is a collaboratively-designed effort from over 100 participants across over 50 At a high level, NeuroBench aims to inclusively apply for benchmarking of novel neuromorphic solutions, actionably enable researchers to implement benchmarks, and iteratively expand with extensible and practical design principles.This article overviews the NeuroBench framework and recent benchmarks which have been demonstrated. Further up-to-date information can be found at the project website: neurobench.ai. Predicting fingertip velocity from cortical recordings.Autoregressive modeling of chaotic functions.Table 1 : NeuroBench algorithm track novel benchmarks. 
the neurobench framework: 
 Although neuromorphic hardware designs have been developed and even are becoming viable commercial products, hardware platforms still lack standardized system interfaces and are not available off-the-shelf for research. As such, current neuromorphic algorithm research, such as machine learning with SNNs, is conducted for the most part on conventional platforms like GPUs, which are non-optimal for final deployment. Steadily, neuromorphic algorithms are being tuned and deployed for neuromorphic hardware, forming complete systems.To facilitate and bridge between research, the NeuroBench framework is composed of both an algorithm track and a system track of benchmarks (Figure 1 ). The algorithm track is intended for agile, hardware-independent evaluation, and defines general compute complexity metrics in order to analyze solution costs. Meanwhile, the system track is for fully deployed solutions, defining standard protocols to measure real-world speed and efficiency of neuromorphic hardware platforms.Between the two tracks, features of successful existing solutions will influence the requirements and configurations of future research. Furthermore, the interactions between algorithm track complexity metrics and deployed measurements from the system track will inform algorithmic profiling and expedite prototyping and exploration.The algorithm track establishes primary metrics which are generally relevant to different types of solutions, including ANNs and SNNs. Firstly, there are correctness metrics, which measure the quality of the model predictions on the particular task. Next, there are complexity metrics, which measure the compute costs of the algorithm, profiling it to offer intuition into deployed performance and efficiency on specialized neuromorphic hardware. The complexity metrics include:• Footprint, accounting for data buffering, neuron states, weights [19] , and the four lower results are from the recent BioCAS Grand Challenge [27] .Results are highlighted from the NeuroBench non-human primate motor decoding task in Table 2 . The task is to decode motor cortical signals from monkeys who are engaged in reaching tasks -given the neural data sequence, predict the sequence of fingertip velocities. The task lends to BMI applications, which may be subject to extreme area and energy constraints. The NeuroBench baselines (Table upper rows) demonstrated how a small SNN outperforms a small ANN, achieving higher R 2 (0.5805 vs 0.5755) while operating with fewer effective non-zero operations (413 vs 4970), which further are accumulations and would not require multiplication hardware.Recently, this task was used as a Grand Challenge with the IEEE BioCAS conference [27] , inviting researchers to compete on achieving high correctness with low operational complexity. The four winning submissions (Table lower rows) cover both recurrent ANN (GRU) approaches and recurrent SNNs. The training effectiveness to produce higher R 2 is impressive (up to 0.6982), and even more notable are the performance and low footprint and operations of the large and small SNNs (0.6604 with only 304 effective synaptic operations). The NeuroBench algorithm track benchmark and harness has provided a platform to highlight state-of-the-art techniques for neuromorphic algorithms, compare them against conventional approaches, and motivate further research progress.While the NeuroBench algorithm track is meant for hardware-independent profiling and prototyping, the system track targets benchmarking full neuromorphic platforms. As the development of systems involves far more engineering effort including algorithm, hardware, and software development, the initial focus of the system track is to provide a unified platform for which system developers can showcase their results head-to-head, in contrast to providing tools and methods for profiling as in the algorithm track. Deployed metrics are compared between novel neuromorphic designs, as well as against conventional platform baselines, in order to demonstrate where advantages and strengths lie. The system track also offers a framework for documenting the methods used for designing and measuring different architectures. By transparently sharing and documenting, the system track intends to drive neuromorphic hardware towards mature standardization.The system track benchmarks assess platforms at the task level, as opposed to operation or kernel level micro-benchmarking. Thus, analagous to the correctness and complexity metrics in the algorithm track, the system track also measures task correctness and performance in terms of latency or throughput, and efficiency in terms of energy expenditure and power consumption.While the metrics are straightforward, fairly measuring varying hardware platforms presents a large challenge. For one, neuromorphic platforms are implemented at various solution scales, from single-chip, ultra-efficient edge devices [28] up to multi-board server machines containing hundreds of chips [29] . Furthermore, neuromorphic devices often utilize mixed-signal technology [8] [9] , which have significantly different characteristics than fully-digital platforms. Also, the state of many neuromorphic systems may not be a mature, standalone machine, and thus development and experimental hardware may populate the system which shouldn't reasonably be included in measurements. All of the challenges make a singular, fully-consistent measurement methodology that everyone can strictly apply currently infeasible.In order to actionably benchmark neuromorphic systems, in the absence of consistency the Neu-roBench system track instead favors transparency. Based on features of existing neuromorphic systems, the track offers guidelines for measurement protocols and reporting documentation to contextualize system track results. Transparent documentation provides the foundation for a shared methodology between varying systems, and it allows for wide system exploration with holistic analysis. Using the current generation of reports, future generations of system design will have development guides to improve system consistency, lending to both system maturation and standardization across the field. As more results are collected with both the algorithm and system benchmark tracks, neuromorphic 193 researchers will be able to find task leaderboards, providing a target for new design metrics and  
 
 Appendix:  



Reviews: review: title: title: fundamental benchmark framework with huge effort
main_review: main_review: Quality: The overall quality should be better than the average line. This work filled the gap on the benchmark perspective for neuromorphic computing, which can be taken as a elephant in the room.

Clarity: The article was written with idiomatic English to convey the information. The structure is logical and rational. Readers can position to specific sub-topic with minor effort.

Originality: This benchmark framework is a original huge work engaged with over 100 researchers from more than 50 institutions. It is observed that part of the figures and tables were taken from their pre-release in arxiv (2304.04640).

Significance: This article clearly stated the necessity of the proposed benchmark framework, which brings additional value, such as showcase of performance difference between ANN and SNN solutions for the same task, transparent task leaderboard, and a platform to share and extend on each other's work, for the community of neuromorphic computing.

Comment 1: For the collected tasks in the algorithm track, is there a way to display the task difficulty with a figure of merit? Algorithms or models can be sequentially challenged with more and more difficult tasks to evaluate their scalability. 

Comment 2: Table 1 and 3 are wider than the text width
title: title: Work in Progress with Limited Significance
main_review: main_review: The paper addresses an important issue relevant to the workshop which is a lack of standardized methods and benchmarks for neuromorphic hardware implementing spiking neural networks. The authors approach this issue in a systematic way and correctly recognize challenges such as the need to distinguish between algorithmic properties (e.g., memory requirements) and the properties of hardware implementations that ultimately determine the performance of neuromorphic systems. However, the described work is clearly in progress and, at this point, it is questionable whether the described methods significantly advance the state-of-the-art or are useful to serve the intended purpose of providing standard benchmarking methodology for neuromorphic hardware in general.

In my opinion, the authors should either reduce the scope of their work (e.g., to characterizing implementations based on PyTorch and digital neuromorphic hardware) or attempt to develop evaluation metrics that are generally applicable and defer publication until the project, specifically the "system track", has matured beyond statements of intent. 

Constructive criticism:
* The manuscript only discusses evaluating inference. However, the bottleneck of model size in ML is given by the energy and memory demands during training. The authors should discuss or develop benchmarks or metrics for the costs of model training and how these might differ between classical and neuromorphic implementations. 
* The "algorithm track" seems to be geared towards PyTorch-based SNN implementations. These typically are discrete-time and formally equivalent to recurrent binary neural networks/multi-layer Perceptrons. Authors should discuss how this blurs the line between traditional machine learning and neuromorphic computing.
* The "algorithm track" seems to be inapplicable to any neuromorphic algorithm implementation that is not based on PyTorch. The authors should discuss how analog or mixed-signal hardware could fit into the described framework. Can the provided "code harness" be used with SpiNNaker2, Loihi, BrainScaleS, ... or would this require re-implementing the models using PyTorch? 
* Leaving aside the "code harness" that automatically computes complexity metrics for PyTorch implementations, what is the novelty or scientific contribution of the algorithm track? The described metrics in 2.1.1 are standard and authors should consider constructing metrics that capture a trade-off between model performance and computational complexity.
* ANNs such as LLMs are typically quantized to lower precision arithmetic for inference and GPUs such as the H100 have native support for, for example, FP8 MAC. Authors should take this into account and discuss how this impacts how meaningful metrics such as the number of effective MAC operations are.
* The "system track" seems to currently only consist of guidelines encouraging transparency when profiling neuromorphic hardware. While such guidelines may be useful, the authors should describe them in the manuscript instead of a general mention.
* Why are the benchmark data sets different for both tracks? For example, it seems highly relevant to characterize task-level performance for event-based vision.
* GPUs (e.g., the mentioned H100) and other ML accelerators (e.g., Cerebras) have features to accelerate sparse matrix multiplication. Authors however claim "Such conventional systems are specialized to full-utilization for dense compute operation". Authors should discuss how neuromorphic hardware differs from accelerated, sparse operations on traditional hardware.
* Authors should mention the specific contributions of "over 100 researchers". Readers may suspect that the majority of these researchers did not make a substantial contribution to the project and feel that this number has no bearing on the relevance or validity of the presented research.
* Authors should either explain what they mean by "agile analytical models" (l. 205) or avoid the use of such vague phrases.
* Authors should support claims such as "Many benchmarks have shown that neuromorphic designs can beat conventional technology" (l. 190-191) using citations.