Spike-based Neuromorphic Model for Sound Source Localization
Authors: Dehao Zhang
Shuai Wang
Ammar Belatreche
Wenjie Wei
Yichen Xiao
Haorui Zheng
Zijian Zhou
Malu Zhang
Yang Yang
Venue: NeurIPS 2024 poster (2024)
Keywords: Spiking Neural Networks
Resonate-and-Fire Neurons
Bio-inspired Neuromorphic System
Sound Source Localization
Primary area: neuroscience_and_cognitive_science
TL;DR: Inspired by the superior biologic Sound Source Localization (SSL) capability,  we propose a spike-based neuromorphic SSL model, achieving SOTA performance and enhanced robustness.

Abstract: Biological systems possess remarkable sound source localization (SSL) capabilities that are critical for survival in complex environments. This ability arises from the collaboration between the auditory periphery, which encodes sound as precisely timed spikes, and the auditory cortex, which performs spike-based computations. Inspired by these biological mechanisms, we propose a novel neuromorphic SSL framework that integrates spike-based neural encoding and computation. The framework employs Resonate-and-Fire (RF) neurons with a phase-locking coding (RF-PLC) method to achieve energy-efficient audio processing. The RF-PLC method leverages the resonance properties of RF neurons to efficiently convert audio signals to time-frequency representation and encode interaural time difference (ITD) cues into discriminative spike patterns. In addition, biological adaptations like frequency band selectivity and short-term memory effectively filter out many environmental noises, enhancing SSL capabilities in real-world settings. Inspired by these adaptations, we propose a spike-driven multi-auditory attention (MAA) module that significantly improves both the accuracy and robustness of the proposed SSL framework. Extensive experimentation demonstrates that our SSL framework achieves state-of-the-art accuracy in SSL tasks. Furthermore, it shows exceptional noise robustness and maintains high accuracy even at very low signal-to-noise ratios. By mimicking biological hearing, this neuromorphic approach contributes to the development of high-performance and explainable artificial intelligence systems capable of superior performance in real-world environments.

Spike-based Neuromorphic Model for Sound Source Localization
Dehao Zhang Shuai Wang' Ammar Belatreche", Wenjie Wei', Yichen Xiao',
Haorui Zheng', Zijian Zhou', Malu Zhang' Yang Yang' University of Electronic Science and Technology of China
Northumbria University
Peking University
{zhangdh, wangshuai1718 www.ueesst....eee edu. en, maluzhangouestc..dd cn
Abstract
Biological systems possess remarkable sound source localization (SSL) capabilities that are critical for survival in complex environments. This ability arises from the collaboration between the auditory periphery, which encodes sound as precisely timed spikes, and the auditory cortex, which performs spike-based computations. Inspired by these biological mechanisms, framework that integrates spike-based neural encoding and computation. The framework employs Resonate-and aeronate-an-Feee -nd--Fir Fire (RF) neurons with phase-locking coding (RF-PLC) method to achieve energy-efficient audio processing. The RF-PLC method leverages the resonance properties of RF neurons to efficiently convert audio signals to time-frequency representation and encode interaural time difference (ITD) cues into discriminative spike patterns. In addition, biological adaptations like frequency band selectivity and short-term memory effectively filter out many environmental noises, enhaneing SSL capabilities in real-world settings. Inspired by these adaptations, we propose spike-driven multi-auditory attention (MAA) module that significantly improves both the accuracy and robustness of the proposed SSL framework. Extensive experimentation demonstrates that our SSL framework achieves state-of-the-we art accuracy in SSL tasks. Furthermore, it shows exceptional noise robustness and maintains high accuracy even at very low signal-to-noise noise ratios. By mimicking biological hearing, this neuromorphic approach contributes to the development of nigh-performance and explainable artificial intelligence systems capable of superior performance in real-world environments.
propose novel neuromorphic SSL
Introduction
Sound source localization (SSL) [1] [43 critical skill for mammals that enables them identify and locate external auditory stimuli. This skill plays a vital role in survival behaviors like prey detection and predator evasion. Over decades of scientific exploration
48 SSI. has evolved
from purely biological concept to a sophisticated technology with wide range of applications
across various fields 10 39 Today, SSL methods are finding increasing use in areas like security monitoring [12] robotic navigation 37 and autonomous driving [13] 36).
Early SSL approaches rely hand-crafted analysis of speech signals from multiple receivers. While offering basic ability to localize sound sources, these methods suffer from limitations in accuracy and robustness. The emergence domains lead researchers to explore their application in SSL tasks, achieving significant performance improvements (64 67]. However, DNN-based approaches face two key challenges. Firstly, DNNs
Deep Neural Networks (DNNs) and their success in various
Equal contribution, "ooresponniig author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

achieving high SSL accuracy often require substantial computational resources, leading to increased energy consumption. Secondly, DNNs struggle learn the intricate relationships between localization behaviors and noisy environmental constraints. These limitations hinder the development of portable, edge-based SSL models [16] for real-world environments.
Recently, Spiking Neural Networks (SNNs) [4 20 34). inspired by brain neural architectures, have gained significant attention for their
simulation of neural systems. Spiking neurons [33] simulate the information transmission mechanism of biological neurons, computing only upon the arrival of imput spikes and remaining silent otherwise
[53]. Such an event-driven mechanism results in sparser information transmission [60] 61], hence reducing computational costs [6] 69]. Therefore, the SNN- based SSL models enable a more energy-efficient emulation of biological SSL SSL model that achieves localization in real audio signals. Chen al. 7 improve the model's performance through hybrid encoding method, achieving competitive results with less energy consumption. Although these examples achieve edge-friendly SSI. ability, limitations still exist in neural encoding efficiency and robustness under noisy conditions.
processes. Pan et al. 40) propose
In terms of neural encoding, most methods still rely on Fourier Transform (FT) 12 [5 to encode ITD [ present in the received audio signals into spike trains for processing by back-end SSL model. However, FT operations involve many multiply-accumulate (MAC) computations and require significant computational resources [58] which hinders our goal of developing energy-efficient SSL models. In terms of robustness, the superior localization ability in biology not only relies on ITD cues but also on various auditory mechanisms [2 [54] such as frequency preferences, short-term memory, etc. Frequency preference significantly mitigates the impact of complex environments on localization accuracy [3 62), and auditory short-term memory effectively filters out irrelevant noise 49 (5),
focusing on important auditory signals. However, most SNN-based solutions 3 primarily focus on ITD cues, with little attention to multiple auditory mechanisms, Therefore, investigating more energy-efficient and robust SNN-based SSL models remains pressing challenge to address.
(a) Binaural SSL Tasks
(b) ITD Cues Capture
(c) Mammals Auditory Mechanism
Monery Rosaniken
Difteraan Praquracy
MSO
10
10
20
30
40
50
Sound Localization
Basilar Memthrane
Frequency Prefermnce
Memory
(d) Input Signal
(e) Front-end ITD Encoding Method
(f) Back-end SSL Model
Toonatuee SSS pait-ooaaad
Original Matni
Masked Manix
Left ITD Cues
Right ITD Cues
MAA Meaal::
FSS
Fining Detection Neuron
Sälent Detection Neurons
Delay
Memory
Interaction
Au
Aximuth:45
Hadamard
R1-PLC method
RF Nourons
Product
Figure A apike-based SSL framework inspired by biological auditory localization. (a) Schematic of binaural SSL tasks. (d) Simulation of the SSL tasks. The upper section illustrates two key processes involved in mammalian localization: (b) the Basilar Membrane and Medial Superior Olive (MSO) collaborate to capture ITD cues; (c) multiple auditory attention mechanisms further process these ITD cues for precise localization. The lower section presents our spike-based SSL framework, comprising two main components: (e) a front-end ITD encoding method employing RF and detection neurons; (f) back-end SSL model utilizing multi- aulti-auditory attention.
In this paper, we propose a novel SNN-based SSL framework, which primarily comprises an ITD encoding front-end method and biomimetic localization back-end model. As illustrated in Fig we introduce a phase-locking coding (RF-PLC) method using Resonate-and-Fiis (RF) neurons 38 and detection neurons 40. It simulates the frequency band decomposition function of the basilar membrane and captures ITD cues, respectively. Furthermore, we introduce novel back-end SSL model based on multi multi-auditory attention (MAA) that integrates frequency preferences and short-term

---

memory characteristics. This approach significantly improves both the accuracy and robustness of localization. Extensive experiment on the HRTF [57], Single Words [30,, and SLoClas 442 datasets demonstrates that our SNN-based model achieves otate-of-the-art performance. Moreover, evaluation in noisy environments reveals the model's enhanced adaptability to real-world conditions. The work introduces the following key contributions:
The RF-PLC method leverages the resonance properties of RF neurons to perform energy-efficient auditory time-freqeen transformations, avoiding the high resource costs of FT operations. Additionally, it utilizes a phase-locking loop and ITD detection neurons to encode auditory ITD cues into spike patterns directly, ensuring the fully spike-driven nature of the entire SSL framework.
Spike-based ITD encoding:
Biologically inspired attention: The MAA module incorporates knowledge of biological auditory frequency preferences and short-term memory characteristics. Frequency prefer- ences effectively mask the ITD information of irrelevant frequency bands and spatial regions, while short-term memory focuses on the interaction of information across adjacent time steps. This combination enhances the robustness of the SNN model in noisy environments.
State-of-the-art performance with reduced complexity: By integrating these methods, we present SNN- SSL framework that achieves state-of-the- art performance while reducing energy consumption to existing works. Additionally, extensive experimentation demonstrates that our system exhibits superior robustness in noisy environments.
2 Related Work
2.1 ITD Cues for SSL Tasks
To develop biologically inspired models for SSL tasks, researchers have drawn upon the auditory
localization mechanisms observed in mammals 23 25 The cues of ITD are recognized as critical for these models [29 44 47 ITD refers to the temporal disparity in sound arrival between the ears. Specifically, when a sound source is closer to the listener's ' right side, audio reaches the right ear sooner than the left. The Jeffress model [ and BiSoLaNN (56) encode ITD cues into spike trains and corroborate their biological credibility through experiments approaches primarily focus on the localization of pure tones, which significantly differs from the time-varying audio signal in daily life. Substantially, some researchers (7] 40) have utilized complex FT operations to obtain the phase information of audio signals. However, FT operations require substantial computational resources and pose significant challenges when implementing systems on edge devices with limited computational capabilities. Therefore, the exploration of low-power ITD encoding methods become pressing direction to pursue.
barn owls (5). But these
2.2 Biological Adaptation in Auditory System
In the field of auditory science, frequency preference and short-term memory characteristics are essential for understanding auditory processing. Numerous studies [2 52 63 have demonstrated that biological auditory systems exhibit heightened sensitivity to specific frequency ranges, such as 20-20 000 Hz in humans, with other species like bats and blue whales adapted to different ranges. Further research [5T] has revealed tonotopic maps and variations in frequency tuning across regions, underscoring the importance of frequency selectivity in hearing. Electrophysiologee experiments [31 confirmed that inner hair cells on the basilar membrane of the cochlea exhibit significant differences in response to various frequency bands.
These studies underscore the irreplaceable role of frequency band preference in auditory decision- making. Additionally, compared to visual short-term memory, auditory short-term memory [2] has shorter retention span. Nonetheless, it is essential for real-time integration and coherent environmental perception. Simultaneously, some researchers (50) propose that neurofeedback training targeting auditory short-term memory can significantly enhance selective attention to auditory signals in noisy environments. Moreover, Zhong et al. [70] suggested that auditory short-term memory can highlight sound source characteristics under reverberant conditions, reducing interference from other sources. However, current SSL methods mainly focus on ITD cues, neglecting these well-established biological mechanisms. Therefore, the effective integration of diverse auditory attention mechanisms within SSL tasks to enhance robustness remains significant ongoing challenge.
3

---

3 Method
In this section, we introduce our spike-based SSL framework, which includes front-end ITD encoding method and a back-end localization model. For the front-end ITD encoding, we propose the energy-efficient RF-PLC method, which uses RF neurons capture ITD cues and detection neurons to convert these encoded cues into spike patterns. For the back-end localization model, we take inspiration from biological adaptation and propose the MAA mechanism enhance the model's localization performance and robustness.
3.1 RF Phase-locking Coding: A Direct Font-end ITD Encoding Method
Due to the physical separation of the ears, sound waves arrive each with slightly different timing. leads to differences in the initial phase information between the two audio channels. Pan et al. [40] propose Multi-Tone' Phase Coding (MTPC) method that utilizes this information exploit ITD cues. However, this approach relies on computationally expensive FT operations and introduces an additional phase transformation step during processing. To overcome these limitations, we propose the RF-PLC method, leveraging RF neurons' resonance filtering and periodic decay properties. This approach effectively eliminates the need for energy-intensie FT and phase transformation processes. Subsequently, a set of detection neurons with varying delays is employed to efficiently encode the ITD cues from different microphones into spike patterns.
Ampliude
Boy JOnZ
40HZ
1.0
125
140
Timestep
20
60
Timeslkp
Menheanee Vollage
LIF Model
SPIKE
Memberse Volcgge
Model
SPIKE
Part Zgr
oo:: TON1Z
40:40HH
60
80
100
120
Timestep
Timestop
(a)
(b)
Figure 2: Properties of spiking neuron models. (a) Responses of LIF and RF neurons to an identical input spike train. We can observe distinct patterns in both membrane voltage accumulation and spiking behavior between the two neuron models. (b) The frequency-selective properties RF neurons. RF neurons with resonant frequency of 10 Hz (w 10) have significantly stronger response at 10 Hz compared to the response at Hz.
The first step of model processes the audio to capture ITD cues. We segment the audio into y1 based on the smallest durations by the human car. These segments are then encoded by specialized RF neurons (38) tuned to different frequency bands. The dynamics of these RF neurons can be described as follows:
ZRF [t] Xe iwAt ZRF [t 1] + + [0]
where represents the resonant frequency of the neuron, indicating the number of radians it progresses per second. serves the dampening factor, which causes the oscillation to decay exponentially. AL represents the sampling rate, which is set to 1. 7[1] denotes the audio input. Zar(4) can be reformulated as æ + iy C. A detailed process can found in Appendix. A The real component of ZRF reflects the current-like behavior of the neuron, capturing the dynamics of voltage-gate and synaptic currents. The imaginary component y of [ serves as a voltage-like variable.
Based on Eq. D we depict the spiking behavior of the RF neuron in Fig. 2(a) and summarize its characteristics in two aspects. Firstly, the complex form of the RF neuron's dynamics enables it to capture the phase information in a specific frequency band w, termed resonance filtering. Secondly, the dampening factor allows it exhibit periodic decay characteristics when there is input.
By leveraging RF neurons' resonance filtering and periodic decay properties, we encode input signals into ITD cues efficiently and effectively. To better describe the RF-PLC process, decompose the

---

dynamics of the RF neurons into two stages: a silent stage and a spike stage. The silent stage is utilized to decompose audio information into distinct frequency components and store this data in the state of the RF. The spike stage then oscillates phase information, effectively converting it into ITD cues through phase-locking mechanisms. These stages can be described as follows:
(ewwss ZRF(T 1| + [[t), Silent Stage,
ZRF [t]
(2)
Aelwar ZRElt - 1],
Spike Stage.
In the silent stage, RF neurons with distinct w values selectively respond to specific resonant frequencies. As illustrated in Fig 2(b) when the frequency wi of the audio input 7[t] closely matches the RF neuron's 's resonant frequency a significant increase in its membrane potential occurs. Conversely, misalignment between these frequencies leads slower accumulation of membrane potential. This characteristic offers an energy-efficient alternative to the computationally expensive FT operations. The result of the silent stage can be interpreted as analogous to the initial phase information of each pure sinusoidal component within the audio signal.
C
Silent Stage
Spike Stage
Membeane Voltage
T4
TD cues
F2
f
V
Original Phase
Phase Diegram
Phase Diagram
6
Plaaee Locking
Locked
Spike
Spike
T4 +
Silent Stuge
Differenc RF
Sikeat Dctacioo
Desection
ITD eaes
(a)
(b)
Figure 3: The proposed the RF-PLC method. (a) ITD cues capture: during the silent stage, RF neurons replace FT by responding to input signals. In the spike stage, the RF neurons' first oscillatory peak time is encoded as their spike firing time through phase-locking loop. (b) The coincidence detection network: detection neurons directly encode ITD cues by analyzing the spike timings of multiple RF neurons from two receivers and generating spikes after a specific time delay.
In the spike stage, we introduce PLC method that ensures the RF neuron fires a spike only at specific phase. Specifically, the spike firing time flock is defined as the special state when the real part of the RF neuron state reaches zero and the imaginary part reaches its maximum (ZRe[fock]] This precise spike timing can be directly utilized as an ITD cue, with details validated in Appendix B Notably, the periodic decay characteristic of RF neurons guarantees that only one spike is generated using our PLC method, ensuring the efficiency of ITD encoding.
0 iymax).
As illustrated in Fig. 3(a) we provide schematic representation for obtaining ITD cues from input audio. During the silent stage, RF neurons receive audio signals and convert them into phase information of pure tones at different frequencies. During the spike stage, the PL.C method leverages this phase information to determine whether the RF neuron fires a spike. The precise spike timing of the RF neuron serves as the ITD cue. Compared to traditional FT-based methods that rely multiple network layers, our RF-PLC significantly reduces computational costs and offers a more biologically plausible representation of ITD cues.
The final step of the RF-PLC method involves detection neurons that convert spike times (also, ITD cues) into spike patterns. As illustrated in Fig. 3(b) a series of detection neurons are used in each band, with each neuron tuned to a specific delay preference (from Ti Tw)). These detection neurons then encode the overall ITD cues for the audio signal. Interestingly, similar symmetrical detection structures have been observed mammalian auditory pathways [55,, which support the biological plausibility of our approach.
5

---

3.2 MAA: Multi-auditory Attention Mechanism for Back-end SNN-based SSL Model
After encoding audio signals into spike patterns, we construct back-end SNN-based model to process this encoded information for SSL tasks. The SNN-based SSL model is built based on the Leaky Integrate-and-
(LIF) neuron due to its computation efficiency. The LIF model receives the resultant current and accumulates membrane potential which is used to compare with the threshold to determine whether to generate the spike. Its dynamic can be described as follows:
H[t] + X[t + 1). S -|
Ult (
(3)
Via);
(4)
Hit [[ Stt + 1] ( 1] Sit + 1J)-
(5)
At each time step + 1, the spatial input current X + 1] is obtained through convolution and linear layers. This current integrates with the previous temporal input H [t] to update the membrane
potential U + 1]. The Heaviside step function ( determines whether the binary spike Sit 1] is generated by comparing the membrane voltage with the threshold ViA-
If there is spike emission, [t] is reset to the resting potential Vreset: otherwise, Utt + 1] decays with a time constant and directly feeds into Hit + 1]. We denote the LIF spiking neuron layer as SN(.), which takes X[t + 1] input and produces the spike tensor Sit + 1] as output. Existing back-end SNN-based SSL models only rely on simple convolutional and fully connected layers for localization, without considering biological adaptation mechanisms such as frequency band selectivity and short-term memory. This leaves substantial room for improvement in localization performance. Therefore, draw on these biological mechanisms propose the computationally efficient MAA.
In the field of SNNs, there have been some studies on attention mechanisms 19 65 71. However,
these methods almost rely heavily squeeze-an gqueeze-and-excinge and-excitation operations, which introduce additional MAC operations. Therefore, we propose a novel spike-driven MAA mechanism that comprises a Trequency-spatiie joint attention module and a short-term memory structure. The former enhances networks' focus on critical ITD cues within key frequency bands. The latter strengthens the model's memory for wise decisions across timeframes. Notably, our MAA module is tailored for SSL tasks and achieves the best trade-off between performance and efficiency.
Input
Input
Conv
87
B.N
12
Cat/Stt
Bast
57
BaradCast
Ingui
(a) Spatial Attention SNNs
(b) Time Attention
(c) FSJA module (ours)
(d) ST-M (ours)
Figure 4: Comparing MAA with spiking attention methods. (a) In SNNs, CA/SA 66 uses MAC- based broadcasting operations. (b) TA [65] efficiently focuses on temporal sequences but struggles with streaming data. (c) FSJA adopts binary attention map as an alternative to MAC-based broad-
casting, enhancing computational sparsity and masking noise (white blocks). (d) ST-M incorporates ITD cues within a streaming context, significantly reducing the model's 's computational resource.
3.2.1 Frequency-Spatial Joint Attention
To enhance adaptive learning and selection of preferred frequency in our SNN-based SSL model, propose a spike-driven FSJA module. For each time step, the output of the RF-PLC method is

---

defined as X E RCxxx where F and S respectively represent the number of RF and detection neurons, and C' denotes the microphone array. The FSJA module can be expressed as follows:
s
1 2 ==SN( FXS (11==1
Xi,lt), Attrs (2) SN (ConvBN(2)), FSJA Auys (2).X(t) (6)
where ConvBN is convolution operations with a 3 x kernel and batch normalization. The matrix Z is defined as the average of [[]] across the F dimension and S dimension and the spike results after passing through the LIF neuron. Due to the attention map of FSJA module is in binary spike form, effectively concentrates spike information specific frequency and spatial dimensions.
To further demonstrate the difference between our FSJA module and previous SNN-based attention methods across frequency and spatial dimensions, we show the difference between them. As shown in Fig. 4[a] the existing SNN-based attention module relies on full-precision values. Although the broadcast operation is spike-driven computational paradigm, it introduces additional MAC operations for the next layer. As shown in Fig. 4[c) it effectively avoids MAC-base broadcasting operations which substantially improves energy efficiency. Additionally, our method effectively masks the ITD information of irrelevant frequency bands and spatial regions, substantially boosting the SSL model's robustness in noisy environments.
3.2.2 Short-term Memory Structure
The auditory short-term memory characteristic enables sustained perception of SSL processing, yet few studies have focused on this aspect. Although the membrane potential accumulation of spiking neurons partially reflects this mechanism, its simplified mathematical expression is insufficient for describing short-term memory adequately. Therefore, we develop an innovative ST-M structure that emphasizes the interaction of information across adjacent time steps to enhance the neuronal memory capacity. The structure can be represented as:
In[t] = SN (ConvBN (X(t)), ST(E] SN (a ConvBN (ST 1]) + (1 (1-a) a) In(t)).
where Tn[t) represents the preliminary feature extraction of the input X(t). and ST(() denotes the enhanced memory unit, with a serving as the hyperparameter that balances adjacent time steps. Our ST-M architecture is asynchronous, processing information frame-by-frame rather than employing time-dimension attention, thereby significantly reducing the computational resources required by the network and facilitating the direct processing of streaming audio information. Additionally, by balancing the memory residual at time f -1 with the input information at time t. our approach facilitates memory interaction between adjacent time steps. This processing paradigm that relies on input from adjacent time steps pays more attention to short-term memory, thereby granting our model improved localization robustness in dynamic environments.
To avoid the MAC operations present in a ConvBN (ST(t- 1)), we integrate a into the firing threshold of SN, which can be expressed by the following formula:
( ConvBB (ST 1J)
1 1- In(tt]
ST SN'
(8)
Here, SN' denotes a layer of spiking neurons with a threshold of Vin/a. Due ST(() and X(!) being binary spikes and ConvBN can be fused during inference, Eq. contains no MAC operations which ensure low power consumption in inference. Compared with attention in the temporal dimension, the ST-M structure demonstrates asynchronous inference and low wow-power computational characteristics. As shown in Fig. 4(b) and Fig, 4(d) TA methods require the processing of all temporal information and rely on full-precision attention representation, whereas our ST-M structure utilizes only the spike information from adjacent time steps and features spike-driven computation. This ensures the model can perform inference in a low-power manner.
Combining the FSJA and ST-M modules, we propose a spike-driven MAA mechanism, with its insertion location detailed in the Appendix. [D] The proposed MAA mechanism leverages the FSJA module to effectively filter noise, and the ST-M module to strengthen the model's memory for wise decisions across timeframes. As a result, our biologically inspired MAA significantly improves the localization accuracy and robustness of our SNN-based back-end network model.
7

---

Experiments
In this section, we evaluate our proposed spike-based SSL framework performance on three datasets: the HRTF [57]. Single Words [30], and SLoClas dataset [42]. Moreover, we examine its energy efficiency and robustness through extensive ablation studies and noise addition experiments on the SLoClas dataset.
4.1 Comparison with SOTA Models
The HRTF dataset and Single Word cxamined utilizing 2-channel audio at singular frequency, with a minimum angular resolution of 10". In contrast, the SLoClas dataset comprises 4-channel audio in real-world scenarios, with higher resolution of 5 Consequently, the SLoClas dataset presents a higher level of challenge and more closely resembles real-world scenarios. We report in detail the Mean Absolute Error (MAB) and the classification accuracy (Acc.) [7], defined shows:
N
1
MAE(*)
=
Σ
10;
0,,
N
=
(9)
1
Acc.(%)
Σ (10: 01 <n), N
where d, represents the estimated azimuth angle, and denotes the ground truth azimuth angle of the sample i. MAE quantifies the deviation between predicted and true angles, where a lower value means superior performance. Moreover, Acc quantifies the similarity between predicted angles and actual output angles. The 17 is set differently across datasets to align with their specific characteristics: For the HRTF and Single Word datasets, 1j set at 5 while for the SLoClas dataset, is set 2.5 In this manner, 7) is rounded to the nearest increment corresponding to the minimum localization resolution when calculating classification accuracy.
Table Comparison of sound source localization systems.
DoA
Dataset
Methods
Type
Param (M)
T
MAE())
Acc(%)
LSO 57
SNN
74.56%
HRTF
MNTB 157
SNN
97.38%
Our works
SNN
1.64M
99.84%
Single
MSO/LSO 30
SNN
96.30%
Word
Our works
SNN
1.64M
99.63%
GCC-PHAT
ANN
4.17M
4.390
86.94%
SELDnet
ANN
1.68M
1.78
88.24%
EINV2
ANN
1.63M
0,98
94.64%
SRP-DNN
66
ANN
1.64M
0.96
94.12%
FN-SSL
59
ANN
1.68M
0.63*
95.40%
SLoClas
MTPC-CSNN
40
SNN
1.61M
1.23
93.95%
MTPC-CSNN
40
SNN
1.61M
1.02
94.72%
MTPC-RSNN
40
SNN
1.67M
51
1,48
94.30%
Hybrid Coding
SNN
1.61M
4.37
0.60
95.61%
Our works
SNN
1.64M
0.33
0.02
96.40% % +0.3%
As shown in Table I our model achieves SOTA accuracy among similarly sized models but also significantly reduces MAE metrics. Specifically, our model achieves an accuracy of 99.84% and 99.63% on the HRTF datasets and Single Words, respectively. Additionally, on the challenging SLoClas dataset, our model achieves a MAE of 0.33 and an accuracy of approximately 96.4%, while the number of the model parameter is only 1 1.64M. It represents nearly 50% improvement in the MAE metric compared to the current SOTA performance of SNN-based models. The localization precision of our model is also competitive compared to other recently introduced ANN models.

---

4.2
Ablation Study
To assess the efficiency of the proposed RF-PLC method and MAA module, we conduct a series of ablation studies. Specifically, we compare our ITD extraction approach in the RF-PLC method with the established FT-based method used in previous work [40]. As depicted in Fig our method achieves an accuracy nearly identical to that of the conventional approaches, with an error rate of only 1%. Furthermore, prior research has demonstrated that RF neurons exhibit significantly lower energy consumption compared to FT operations, particularly when implemented on neuromorphic hardware [17 [87 45). These findings validate the effectiveness of our RF-PLC method in achieving highly efficient and accurate ITD encoding scheme.
SANWOWA C
40
20
RE-TD
10
FI-ID
TCJA
100
200
300
400
500
Trequency
(a)
(b)
Figure 5: (a) RF-PLF achieves results similar to FT-ITD, highlighting the benefit of avoiding FT operations ITD encoding. (b) Attention mechanism. MAA's binary attention map effectively filters noise and avoids energy-intensive MAC-based broadcasting compared to other spiking attention.
The effectiveness of the MAA module is demonstrated in the model's localization performance. As shown in Table. 2 both the FSJA module and ST-M structure components individually enhance the performance of the back-end SSL model, and their combination yields even superior results. In addition, compared to attention mechanisms such as TA and TCJA, our attention method employs a fully spike-driven computational paradigm. This characteristic allows MAA method to maintain an energy consumption of 9.58uJ, representing an increase 8.49% compared to the work Moreover, Fig ((b) illustrates the binary nature of the MAA attention map. This design effectively avoids the energy-intedd inergy-intensive broadcasting operations typically associated with MAC units. Details on the energy consumption calculations provided in Appendix. [ and Appendix. E This capability substantially improves the model's robustness, which is discussed in the following section.
Table 2: Ablation study
DoA
Methods
Spike- Deike-Driven
Param (M)
Power (uJ)
MAE (9)
Acc %
Baseline
40
/
1.61M
8.83
1.23
93.95%
TA
66
X
1.62M
15.37
0.65%=0..05
93.37% +1.2%
TCIA
X
1.68M
15.34
0.47%++0.0%*
93.45%
+1.0%
ST-M
1.62M
8.99
0.45"
95.67%
+0.5%
FSJA
1.63M
9.42
0.49 +0.02
95.95%
+0.6%
MAA
1.64M
9.58
0.339
96.4%%+0.3%%
4.3 Robustness Experiments
To assess the robustness of our proposed spike-based SSL framework, we evaluate the distribution of MAE under varying signal-to-noise ratio (SNR) conditions. a metric used to measure the level of noise present in the input signal. It can be defined follows:

---

Paigmal
SNR (dB) = 10 10g10
(10)
Puoise
where Paignal represents the power of the signal and Paoise denotes the power of the noise. A lower SNR indicates higher proportion of noise. Specifically, incorporate noise from the NOISEX-92 database into audio recordings from different microphone channels. Detailed information about the noise addition process and the experimental setup described the Appendix
As shown in Fig (()) we visualize the encoding results of the RF-PLC method under various SNR conditions better understand the input form of our back-end model. In addition, also present the
distribution of recognition over 360 As shown in 6ig) and Fig [(c) the MTPC method is more likely predict certain angles, especially in terms of information in the noise direction; however,
method is not significantly affected by noise information. It indicates that our model exhibits higher stability. As SNR increases, our method demonstrates higher recognition accuracy. This indicates that our model effectively suppresses noise in specific frequency bands, thereby preventing significant variations in recognition results due increased noise. The results provide strong evidence of the model's 's superior generalization and robustness when applied to complex real-world scenarios.
504B
90"
0d11
2MRR
20dB
135
Sanple
190
180
Sumple
225
335
225"
335'
270'
270'
(b)
(c)
Figure 6: Performance under varying SNR levels. (a) Impact of SNR ITD Encoding: at dB, it challenging to intuitively discern the direction of the sound source from the encoding results. (b) and (c) Different distribution of MAE over 360 in MTPC (40] and our model. Our model achieves enhanced noise resistance and improved localization stability.
5 Conclusion
Inspired by the efficiency of biological auditory systems, this work proposes a novel spike-based based SSL framework. The core components the RF-PLC method and the MAA module. The RF-PLC method leverages the resonance properties of RF neurons to bypass computationally expensive FT operations. utilizes phase-locking loop and ITD detection neurons efficiently encode ITD from the audio signal into spike trains. Furthermore, the study incorporates insights from auditory biology, including frequency preferences and short-term memory characteristics. By designing a fully spike-driven MAA module, our SNN-based SSL model effectively filters irrelevant environmental noise in the frequency domain while temporally focusing on specific auditory content. This approach achieves superior performance, robustness. and interpretability. significantly advancing the field of neuromorphic SSL research. It establishes a new benchmark for the development of SSL techniques. Future work will investigate the deployment of this model on neuromorphic hardware platforms.
6 Acknowledgments
This work was supported in part by the National Natural Science Foundation of China under grant U20B2063, 62220106008. and 62106038, the Sichuan Science and Technology Program under Grant 2024NSFTD0034 and 202399
10

---

References
[1] Sharath Adavanne, Archontis Politis, Joonas Nikunen, and Tuomas Virtanen. Sound event localization and detection of overlapping sources using convolutional recurrent neural networks. IEEE Journal of Selected Topics in Signal Processing, 13(1):34-48, 2018.
[2] Amogh Agrawal, Aayush Ankit, and Kaushik Roy. Spare: Spiking neural network acceleration using rom-embedded rams as in-memory-computation primitives. IEEE Transactions on Computers, 68(8): 190-1200, 2019.
[3] Go Ashida and Catherine E Carr. Sound localization: Jeffress and beyond. Current opinion in neurobiology, (1(5:7445-711 2011.
[4] Yin Cao, Turab Iqbal, Qiuqiang Kong, Fengyan An, Wenwu Wang, and Mark D Plumbley. An improved event-independent network for polyphonic sound event localization and detection. In ICASSP 1021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 885-889, IEEE, 2021.
[5] Catherine Carr and Masakazu Konishi. Axonal delay lines for time measurement in the owl's brainstem. Proceedings of the National Academy of Sciences, 85(21):8311 1113311-11.11 1988.
[6] Stefano Caviglia, Maurizio Valle, and Chiara Bartolozzi. Asynchronous, wvent-driven readout of posfet devices for tactile sensing. In 2014 IEEE International Symposium on Circuits and Systems (ISCAS), pages 2648-2651 IEEE, 2014. doi:
[7] Xinyi Chen, Qu Yang, Jibin Wu, Haizhou Li, and Kay Chen Tan. A hybrid neural coding approach for pattern recognition with spiking neural networks. IEEE Transactions on Pattern Analysis & Machine Intelligence, 46(05):3064-35) 3078, 2024.
[8] Yi Chen, Hong Qu, Malu Zhang, and Yuchen Wang. Deep spiking neural network with neural oscillation and spike-phase information. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 7073-7083 2021.
[9] James Chrobak and Gyorgy Buzsáki. High-frequency oscillations in the output networks the hippocampal-entorhine axis of the freely behaving rat. Journal of neuroscience, 16(9):3056- 3066, 1996.
[10] Maximo Cobos, Fabio Antonacci, Anastasios Alexandridis, Athanasios Mouchtaris, Bowon Lee, et al. A survey of sound source localization methods in wireless acoustic sensor networks. Wireless Communications and Mobile Computing, 2017, 2017.
[11] Dhwani Desai and Ninad Mehendale. A review on sound source localization systems. Archives of Computational Methods in Engineering, 29(7):4631-4642, 2022.
[12] Ha Manh Do, Karla Conn Welch, and Weihua Sheng. Soham: A sound- based human activity monitoring framework for home service robots. IEEE Transactions on Automation Science and Engineering. 19(3):2369-235 2383, 2021.
[13] Chuang Gan, Yiwei Zhang, Jiajun Wu, Boqing Gong, and Joshua B Tenenbaum. Look, listen, and act: Towards audio-visual embodied navigation. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 9701-9707. IEEE, 2020.
[14] Wulfram Gerstner and Werner M Kistler. Spiking neuron models: Single neurons, populations, plasticity. Cambridge university press, 2002.
[15] Benedikt Grothe, Michael Pecka, and David McAlpine, Mechanisms of sound localization in mammals. Physiological reviews, 90(3):983-1012, 2010.
[16) Pierre-Amaury Grumiaux, Srdan Kitić, Laurent Girin, and Alexandre Guérin. A survey of sound source localization with deep learning methods. The Journal of the Acoustical Society of America, 152(1):10-151, 2022.
[17] Yufei Guo, Yuanpei Chen, Xiaode Liu, Weihang Peng, Yuhan Zhang, Xuhui Huang, and Zhe Ma. Ternary spike: Learning ternary spikes for spiking neural networks. arXiv preprint arXiv:2312.06372, 2023.

---

[18] Mark Horowitz. 1.1 computing's s energy problem (and what we can do about it). In 2014 IEEE international solid-stch state circuits conference digest of technical papers (ISSCC), pages 10-14. IEFF, 2014. doi: 10.1109/iscc22211.6777773
[19] Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excirttin networks. In Proceedings the conference on computer vision and pattern recognilion, pages 7132-7141, 2018.
the IEEE
[20] Eugene M Izhikevich. Simple model of spiking neurons. IEEE Transactions neural networks, 569651156- 2003.
[21] Pingping Jiang, Christopher Kent, and Jonathan Rossiter. Towards sensory substitution and augmentation: Mapping visual distance to audio and tactile frequency. Plos one, 1933)e0229213, 2024.
[22] Naim Kapucu and Vener Garayev. Collaborative decision decision-making in emergency and disaster management. International Journal of Public Administration, 34(6))366-377 376,,375222 2011.
[23] Monika Körtje, Timo Stöver, Uwe Baumann, and Tobias Weissgerber. Impact of processing- lateney induced interaural delay and level discrepancy on sensitivity to interaural level differ- ences in cochlear implant users. European Archives of Ott--Otto----a- Pho-Rhino-Larynolooy) 280(12):5241-
5249, 2023.
[24] Souvik Kundu, Massoud Pedram, and Peter A Beerel. Hire-snn: Harnessing the inherent robustness of energy-efficient deep spiking neural networks by training with crafted input noise. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5209-52115 5218, 2021.
[25] Bernhard Laback. Contextual lateralization based on interaural level differences is preshaped by the auditory periphery and predominantly immune against sequential segregation. Trends in Hearing, 27:233121657 171988, 2023.
[26] Peter Lakatos, Ankoor S Shah, Kevin H Knuth, Istvan Ulbert, George Karmos, and Charles E Schroeder. An oscillatory hierarchy controlling neuronal excitability and stimulus processing in the auditory cortex. Journal of neurophysiology. 1403):19041111,, 2005.
[27] Muhammad Usman Liaquat, Hafiz Suliman Munawar, Amna Rahman, Zakria Qadir, Abbas Kouzani, and MA Parvez Mahmud. Sound localization for ad-hoc microphone arrays. Energies, 14(12):3446, 2021.
[28] Jim-Shih Liaw and T.W. Berger. Robust speech recognition with dynamic synapses. In 1998 IEEE International Joint Conference Neural Networks Proceedings. IEEE World Congress on Computational Intelligence (Cat. No. No.98CH36227), volume 3, pages 2175-2179 vol.3, 1998.
[29] Hong Liu, Yongheng Sun, Ge Yang, and Yang Chen. Binaural sound source localization based on weighted template matching. CAAI Transactions on Intelligence Technology, 6(2):214-223, 2021.
[30] Jindong Liu, David Perez-Gonzalez, Adrian Rees, Harry Erwin, and Stefan Wermter. A biologically inspired spiking neural network model of the auditory midbrain for sound source localisation. Neurocomputing, TA(1-3):129-133) 2010.
[31] Fran Lapez-Caballero and Carles Escera. Binaural beat: a failure to enhance eeg power and emotional arousal. Frontiers in human neuroscience, 11:557, 2017.
[32] Alexandra Annemaric Ludwig. Sylvia Meuret, Michael Fuchs, and Arne Emsst Sound localization in single-sided deaf participants provided with cochlear implant. Frontiers psychology, 12:753339, 2021.
Battmer, Mare Schönwiesner.
[33] Wolfgang Maass. Networks of spiking neurons: the third generation of neural network models. Neural networks, 1699)1155,, 1997.
[34] Timothée Masquelier, Rudy Guyonneau, and Simon Thorpe. Spike timing dependent plasticity finds the start of repeating patterns in continuous spike trains. PloS one, 3(1):e1377, 2008.
12

---

[35] Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. arXiv preprint arXiv:1611006440, 2016.
[36] Dylan Moore, Rebecca Currano, and David Sirkin. Sound decisions: How synthetic motor sounds improve autonomous pediess-e interactions. In 12th International Conference on Automolive User Interfaces and Interactive Vehicular Applications, pages 94-103, 2020.
[37] Fangli Ning, Jiahao Song, Jinglong Hu, and Juan Wei. Sound source localization of non- synchronous measurements beamforming with block hermitian matrix completion. Mechanical Systems and Signal Processing, 147:1071 18, 2021.
[38] Garrick Orchard, E Paxon Frady, Daniel Ben Dayan Rubin, Sophia Sanborn, Sumit Bam Shrestha, Friedrich T Sommer, and Mike Davies. Efficient neuromorphic signal processing with loihi 2. In 2021 IEEE Workshop on Signal Processing Systems (SiPS), pages 254-255 IEEE, 2021.
[39] Takashi Oya, Shohei Iwase, Ryota Natsume, Takahiro Itazuri, Shugo Yamaguchi, and Shigeo Morishima. Do we need sound for sound source localization? In Proceedings of the Asian Conference on Computer Vision, 2020.
[40] Zihan Pan, Malu Zhang, Jibin Wu, Jiadong Wang, and Haizhou Li. Multi-tone phase coding of interaural time difference for sound source localization with spiking neural networks. JEEE/ACM Transactions on Audio, Speech, and Language Processing. 29:2656-2670 2670, 2021.
[41] Xinyuan Qian, Alessio Brutti, Oswald Lanz, Maurizio Omologo, and Andrea Cavallaro. Audio- visual tracking of concurrent speakers. IEEE Transactions on Multimedia, 24:942-9999 954, 2021.
[42] Xinyuan Qian, Bidisha Sharma, Amine EI Abridi, and Haizhou Li. Sloclas: A database for joint sound localization and classification. In 202/ 24th Conference of the Oriental COCOSDA International Committee for the Co-ordination and Standardisation of Speech Databases and Assessment Techniques (O-COCOSDA), pages 123-133, 2021.
[43] Michael Risoud, J-N Hanson, Fanny Gauvrit, C Renard, P-E Lemesre, N-X Bonne, and Christophe Vincent. Sound source localization. European annals of otorhinolaryngology head and neck diseases, 135(4):259-264 2018.
[44] Zahra Roozbehi, Ajit Narayanan, Mahsa Mohaghegh, and Bamaneh-Alsadas Saeedinia. Dynamic-structured reservoir spiking neural network in sound localization. IEEE Access, 2024.
[45] Bodo Rueckauer, Iulia-Alexandra Lungu, Yuhuang Hu, Michael Pfeiffer, and Shih-Chii Liu. Conversion continuous-valuue deep networks to efficient event-driven networks for image classification. Frontiers in neuroscience, 11:682, 2017. doi: 21.38//iiins.217700666
[46] Daniel Schmid, Timo Oess, and Heiko Neumann. Listen to the brain-auditory sound source localization in neuromorphic computing architectures. Sensors, 23(9):451,, 2023.
[47] Thorben Schoepe, Daniel Gutierrez- Galan,, Juan P Mominguez-Morales, Hugh Greatorex, Angel Jimenez-Fernandez, Alejandro Linares-Barrance and Elisabetta Chicca. Closed-loop sound source localization in neuromorphic systems. Neuromorphic Computing and Engineering, 3(2):024009, 2023.
[48] Arda Senocak, Tae-Hyun Oh, Junsik Kim, Ming-Hsuan Yang, and Kweon. Learning to localize sound source in visual scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1358-4366, 2018
[49] Shihab A Shamma, Mounya Elhilali, and Christophe Micheyl. Temporal coherence and attention auditory scene analysis. Trends in neuroxciences, 34(3):114-))) 14--13,, 2011.
[50] Hwan Shim, Leah Gibbs, Karsyn Rush, Jusung Ham, Subong Kim, Sungyoung Kim, and Inyong Choi. Neural mechanisms related to the enhanced auditory selective attention following neurofeedback training: Focusing cortical oscillations. Applied sciences, 13(14):8499, 2023.
13

---

[51] Joel S Snyder, Melissa K Gregg, David M Weintraub, and Claude Alain. Attention, awareness, and the perception of auditory scenes. Frontiers in psychology, 3:15, 2012.
[52] Paul Tarwireyi, Alfredo Terzoli, and Matthew O Adigun. Using multi-audio feature fusion for android malware detection. Computers & Security, 131:103282, 2023.
[53] Amirhossein Tavanaci, Masoud Ghodrati, Saced Reza Kheradpisheh, Timothée Masquelier, and Anthony Maida. Deep learning in spiking neural networks. Neural networks, 11 111:47-63, 2019.
[54] Franziska Trede and Joy Higgs. Collaborative decision making. Clinical reasoning in the health professions, pages 43,54, 2008.
[55] Bernhard Vogginger, Felix Kreutz, Javier López-Randulfe, Chen Liu, Robin Dietrich, Hector A Gonzalez, Daniel Scholz, Nico Reeb, Daniel Auge, Julian Hille, et al. Automotive radar processing with spiking neural networks: Concepts and challenges. Frontiers in neuroscience, 16:851774, 2022.
[56] Kyriakos Voutsas and Jürgen Adamy. A biologically inspired spiking neural network for sound source lateralization. IEEE transactions on Neural Networks,
1799, 2007.
[57] Julie A Wall, Liam J McDaid, Liam P Maguire, and Thomas M McGinnity. Spiking neural network model of sound localization using the interaural intensity difference. IEEE transactions on neural networks and learning systems, 2374):577-586 2012.
[58] Shuai Wang, Dehao Zhang, Ammar Belatreche, Yichen Xiao, Hongyu Qing, Wenjie We, Malu Zhang, and Yang Yang. Ternary spike-based neuromorphic signal processing system. arXiv preprint arXiv:2407.053J1 2024.
[59] Yabo Wang, Bing Yang, and Xiaofei Li. Fn-ssl: Full-band and narrow-ban fusion for sound source localization. arXiv preprint (xX.x:2305//61/// 2023.
[60] Wenjie Wei, Malu Zhang, Hong Qu, Ammar Belatreche, Jian Zhang, and Hong Chen. Temporal- coded spiking neural networks with dynamic firing threshold: Learning with event-driven hackpropagation. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 10552-10562, 2023.
[61] Wenjie Wei, Malu Zhang. Jilin Zhang, Ammar Belatreche, Jibin Wu, Zijing Xu, Xuerui Qiu, Hong Chen, Yang Yang. and Haizhou Li. Event-driven learning for spiking neural networks. arXiv preprint arXiv:2403.cm 00270, 2024.
[62] John H Wittig Jr. Anthony I Jang. John B Cocjin, Sara K Inati, and Kareem A Zaghloul. Attention improves memory by suppressing spiking-neuron activity in the human anterior temporal lobe. Nature Neuroscience, 21(6):808-88 -80, 2018.
[63] Zhimin Xu, Huicui Xin, Yuren Weng, and Guang Li. Hydrogeological study in tongchuan city using the fuquuenyyy magnetotelluric method. Magnetochemistry, 9():332, 2023.
[64] Bing Yang, Hong Liu, and Xiaofci Li. Srp-dnn: Learning direct-path phase difference for multi- ple moving sound source localization. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 721-725. IEEE, 2022.
[65] Man Yao, Huanhuan Gao, Guangshe Zhao, Dingheng Wang, Yihan Lin, Zhaoxu Yang, and Guoqi Li. Temporal-wise attention spiking neural networks for event streams classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10221- 10230, 2021. doi: 10.1109ficc44892222221.1 ,1000
[66] Man Yao, Guangshe Zhao, Hengyu Zhang, Yifan Hu, Lei Deng, Yonghong Tian, Bo Xu, and Guoqi Li. Attention spiking neural networks. IEEE transactions on pattern analysis and machine intelligence, 2023.
[67] Masahiro Yasuda, Yuma Koizumi, Shoichiro Saito, Hisashi Uematsu, and Keisuke Imoto. Sound event localization based on sound intensity vector refined by dan-based denoising and source separation. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 651-655. IEEE, 2020.
14

---

[68] Bojian Yin, Federico Corradi, and Sander M Bohté. Accurate and efficient domaio classification with adaptive spiking recurrent neural networks, Nature Machine Intelligence. 3(10):905-913 913, 2021.
[69] Malu Zhang, Jiadong Wang, Jibin Wu, Ammar Belatreche, Burin Amornpaisannon, Zhixuan Zhang, Venkata Pavan Kumar Miriyala, Hong Qu, Yansong Chua, Trevor E Carlson, et al. Rectified linear postsynaptic potential function for backpropagation in deep spiking neural networks. IEEE transactions on neural networks and learning systems, 33(5):1447-1958, 2021.
[70] Xiaoli Zhong, Zihui Yang, Shengfeng Yu, Hao Song, and Zhenghui Gu. Comparison of sound location variations in free and reverberant fields: An event-related potential study. The Journal of the Acoustical Society of America, 148(1):LL4-ELL1, 2020.
[71] Rui-Jie Zhu, Malu Zhang, Qihang Zhao, Haoyu Deng, Yule Duan, and Liang-Jian Deng. Teja- snn: Temporal-channel joint attention for spiking neural networks. IEEE Transactions on Neural Networks and Learning Systems, 2024.
15

---

A RF Neurons for Energy Efficient FT alternative
Silent Stape
Spiking Stage
af ariigec
W HAM
Video Timestop
Ingust Signal Frame
Phase-Locked Leap
WwMMMMM MM
Origan Phase
Amplinic
Stent Plase
Real of
Plase-ockted Leop
Tene(rx))
Figure 7: RF neurons serve as an energy-efficient alternative to FT, accumulating membrane potential during silent phases to substitute for FT, and directly mapping phases during spike phases.
Lemma 1. Nyquist Theorem
The Nyquist Theorem is pivotal for the sampling process in converting analog signals to digital signals. It stipulates that avoid aliasing, the sampling frequency must be least twice the maximum frequency component present in the analog signal. This criterion ensures that the reconstructed digital signal closely approximates the original analog signal without distortion.
Lemma 2. Fourier Transform (FT)
Consider audio sequence [211, 22, Fr) sampled at frequency fs. The Fourier Transform (FT) facilitates the conversion from time domain to frequency domain, computed as:
2n N
12T nk )) N
FIK Σ
x|n]e-
s/n]] Σ
cos
nk
isin
(11)
where N the number of discrete samples used in the FT. The complex vector Fik] quantifies the spectral components varying frequencies. Utilizing Lemma 1, these components represent sinusoidal signals decomposed at frequencies indexed by k. scaled by ft- For each component Fik] = ag + ih. the corresponding time-domain signal can be described by:
(bk (tr)
yk (t) Va by sin 2T "N"
kt + tan
(12)
aa
Proof: Assume a series of RF neurons, each with resonant frequency of w = [-0 x 2x -1 2T -2 2n N ..... -N 1) N initially set to zero. When exposed to real-time input audio
2
N
the response of the th neuron time is given by the recursive update:
1RF x|t] Ae iun ZRF: [t 1]
rtt + uw At (zelt 1] + iw ZRF [t 2))
(13)
T
T
Σ Am stltt n]= Σ x" (cos (tw) - isin (nw) ælt n).
n=1
This recursive filtering mimics discrete Fourier transform when X = 1. Moreover, RF neurons can be efficiently implemented on neuromorphic hardware like the Loihi2 chip, facilitating low-power and high-speed computations.
16

---

B Initial Oscillation Peak of RF Neurons as ITD Cue
Lemma 1. Neural Phase Coder
Inspired by observations of specific mechanisms within the auditory and visual cortices [9 26 biological evidence are expressed through phase coding found. can encode the pure tone audio into precie--tminng spike. Regarding the results of the decomposition:
UKL
bb Ok
Vr. A1 sin fit ft + tan
.R = A2 sin 2 (2 + tan
(14)
Vr. and VR represent the single-tone sinusoidal signals arriving at the left and right ears. Subsequently, the first peak time encoded into spike time as the arrival time of the sound.
π 2
tan -1
Okz. (iRg
tR
π 2
tan -1
(15)
2T
where ff represents the frequency of the sinusoidal signal. Thus, ITD can be expressed as t t-- Proof: When RF neurons (from both ears) enter the spike stage, their initial states are represented as ake + ibk, for the left ear and 0kn + ibkn for the right car. Here, L indicates the left ear, R denotes the right car, and K refers to the RF neuron associated with the intrinsic frequency fR- Subsequently, the state will decay oscillations over time. To facilitate understanding, we calculate the real and imaginary parts in discrete manner:
Okz [t] azz [t 1] cos fz. + bkz. sis (2 fk),
(16)
bkz [1] = -ax 1] sin (2 fa) + bug cos (2 fx)),
With the RF-PLC method we propose, we can directly spike timing acquisition. Specifically, RF neurons will fire spike at as, [fm, = O. bk, [x,, max(ba,)) In the phase space, this state represented the first peak of time aligns with the neural phase coder:
bkg. [mazl
Plocked tan
Phase-locking Loop
alkz [m
2
(17)
RF tnp fnn = 1t tR. RF-based ITD encoding
where tnz satisfies Olocked ZF [tyz. and fnR satisfies Blocked ZRF [tn.] Due to the discrete form, our ITD encodings are not identical to those Lemma 1. The error is in the difference between the audio's sampling rate and the actual frequency. It results in our inability accurately obtain the first peak time.
Specifically, as Lemma demonstrates, to obtain the spike timing for pure tone audio of different frequencies, the phase coding model requires leveraging Eq compute the audio's ITD cues. However, with the RF-PLC method we propose, it can only iterate according to the audio's sampling rate. The difference between them is dependent on the sampling rate Ss, which can be represented as f/ For the dataset SLoClas that we tested, its errors is only approximately 1%.
C Experiment Detail
We primarily validated the accuracy and robustness our proposed method on the SLoClas dataset. It utilizes 4-channel microphone array to collect data on RWCP sound scenes to ensure SNR of 40 to 50dB. It is comprised of ten distinct categories of ambient sounds: bells, bottles, buzzers, cymbals, horns, metal, particles, phones, rings, and whistles. Each category includes approximately 100 instances, providing a diverse and comprehensive set of audio samples.
Additional, to evaluate the robustness of our method, we need to construct source localization information under various complex scenarios. Specifically, we can introduce different types of noise into the audio. The noise sounds are sourced from the NOISEX-92 database, which contains recordings of various types of real-world noise. We add noise audio to each microphone channel to simulate noise coming from different directions.
17

---

ComplezVideo, = (i);(n) + Anoise(n), 1,2,3,4.
(18)
In this setup, mic, where = 1, 2, 3, 4, represents the 4-channel signals recorded by the micro- phone array. Complex Video, denotes the multi-channel data with noise. The noise data consists of randomly selected audio clips from noise database. À is the scaling factor used to adjust the audio to specific SNR ratio. A smaller SNR indicates a stronger noise presence in the audio which is more similar to real environment.
Table 3: Experimental configuration of the sound localization task.
Attributes
Setup
1. Data preprocessing:
Sampling rate (Hz) Frame length (ms)
16000
170
Frame stride (ms)
170
RF neurons n Number of Microphones
512
2. RF-PLC setting:
CQT frequency range (Hz)
[0, 8800]
(ms)
0.0625
Frequency channels N
40
Coincidence detector N,
51
Microphone pairs C
3. SNN Hyperparameter:
0.75
Timestep
Epochs Batch size
300
128
Optimizer
Adam
Base learning rate
le-3
Learning rate decay
Cosine
Weight decay
5e-3
D Model Structure
The overall network architecture of our SNN-based SSL model is illustrated in Fig featuring comprehensive system design tailored for sound localization. The architecture consists of two main components: front-end RF-PLC method and back-end MAA-based localization decision network.
In the RF-PLC method, we utilize 512 RF neurons with widths w that increase incrementally from to 8000Hz. These neurons are strategically deployed as an alternative to the traditional FT operation, optimizing the model for energy efficiency and computational speed. Additionally, we utilize cochlear filter bank following the Constant Q Transform (CQT) which is the most commonly used and easily implemented cochlear filter bank, to extract auditory features of appropriate dimensions. This encoding approach not only mimics the cochlear filtering process but also enhances the temporal dynamics of sound processing. Additionally, N detection neurons are engaged to characterize the ITD with delays ranging from -25T to 25r. Based the audio sample ratio. the value of 7 is set at 0.0625 ms. Furthermore, our model utilizes data from four microphones to compute ITDs between each pair, resulting in six distinct sets of ITD cues. As result, each 170 ms speech frame is encoded into X E NCxNXN, capturing rich array of spatial and temporal information. Details of these parameters be found in Table 3 This setup ensures detailed and dynamic spatial representation of auditory scenes.
In the back-end decision network, we propose fully spike-based model, which is illustrated in Fig. g We will validate our module within the SSL models [ [4) consisting of convolutional and MI. layers. To enhance the performance of the SSL model, augment this basic structure with our MAA
18

---

module. This module is designed to emulate biological auditory processing by focusing on frequency band preference and short-term memory capabilities. Compared with traditional spatio-temporal
spike attention techniques 65 71. our module markedly improves the system's s computational efficiency, as shown in Table. Additionally, to better illustrate the performance of our module, we present comparison with various modules. Details of these networks are provided in Table. | This enhancement allows for faster and more accurate sound localization, demonstrating the potential of model in real-world applications.
SLoclass Dataset
Mulit-auditory Attention Block
?
Max Pooling
Max Pooling
Classification Layer
RF-PLC Encoding
Conv 2D
Cony 2D
BN
MAA
BN
FSIA
ST-M
Conv
ST
Ing
BN
Conv
Splic Neuron
Elament
1--
Element-Wie
Product
Figure 8: The structure of the Spike-based Neuromorphic Sound Source Localization. It includes an RF-PLC encoding method and back-end classification model based on SNNs.
Table 4: Detail Network
Stage
Output Size
Baseline
Baseline + MAA
Baseline others
Conv x 3, stride
Stage
12 25 20
BatchNorm
MaxPooling 2 x 2, stride 2
Conv 3 x 3. stride BatchNorm
Stage
24 x
MAA module
other Attention
MaxPooling x 2,stride 2
Conv x 3, stride BatchNorm
Stage
48 x 6 x
MAA module
other Attention
MaxPooling 2.stride
Classifier
1x1x1
360-FC
E
Energy Cost
To describe the energy consumption calculations in the ablation experiments, we introduce a theoreti- cal energy estimation method for the proposed attention mechanism. Compared to the ANN model, the energy consumption caleulation of the spiking version requires information the timesteps (T) and spike firing rates (R). The spike firing rate is defined as the proportion of non-zero elements in the spike tensor. Since our proposed MAA method is spike-driven, we only need evaluate the model's 's FLOPs, along with T and R, to estimate the theoretical energy consumption of our methods.
In ANN 35 the FLOPs for the n-th Cony layer expressed
Conv (knn) hnn Wn Cn-1 Cx.
(19)
19

---

where kn denotes the kernel size, (ha, wn) specifies the dimensions of the output feature map, and Ca-1 1 and On represent the numbers of input and output channels, respectively. The FLOPs of the m-th MLP layer in ANNs are:
MLP im Own
(20)
where im and Om represent the input and output dimensions of the m-th MLP layer.
Refer on previous research [24 68). we assume that all computational data are implemented using
45nm technology for b2-bit Moating-poinl calculations, with ÉMAC 4.6pJ and E 0.9 pJ.
In Table 3 we further present the details of different models. Referring to 66 71,, the attention matrix is derived using the sigmoid function, which results in the network input for the subsequent layer being non-spiking. Consequently, during the energy consumption calculation, this component is computed using the energy consumption of MAC operations, leading to significant increase in energy consumption within the network. Due to the unique properties the MAA attention matrix, our method does not introduce additional floating-poin operations.
F Limitation
The limitations of this study include the lack of deployment on edge devices. Furthermore, the limited availability of datasets for SSL tasks restricts the validation of our model across a broader array of datasets. Future research will seek to overcome these challenges by amassing a more extensive model edge devices more effectively ascertain the
collection of datasets and implementing efficacy of our approach. The experimental results reported herein are reproducible, with detailed descriptions of the model architectures and hyperparameter configurations available in Appendix. Additionally, our code will be made available on subsequent after review.
G Supplementary Ablation Experiments
To further validate that the enhancement in our model's performance is indeed due to the effective implementation of band selection and short-term attention mechanisms, rather than increased parameter count. We designed Conv2d layer module with the same amount of parameters, ensuring all other parameters remained consistent. Specially,
Adean Absoluce Eiror
LocilMAA
MAA
50.4
Localization Accurncy
Tecelie--T CT. M
4-
MAE
MAE
0.62
0.33
1-
50
100 Epoch
150
200
50
100
150
200
MAE(")
Acc.(%)
Lspoch
(b)
(c)
Figure 9: Supplemental ablation experiments. In these experiments, the Local module is configured with the same number of parameters as our proposed models to ensure a fair comparison: (a) A comparison of MAE.(") and Acc.(%) between our proposed MAA and LocalMAA. (b) A comparison of the MAE between our proposed ST-M and LocalST-M. (c) A comparison of the MAE between FSJA and LocalFSJA.
As illustrated in Fig. 9 LocalMAA refers network with the same number of parameters as our proposed MAA. It can be observed that our MAA method achieves a lower MAE()) and higher Acc(%). Furthermore, band preference and short-term memory structure significantly enhance the network's 's localization performance.
compared the MAE of different modules, finding that our frequency
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our abstract and introduction clearly describe our contribution, the algorithm, and the experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We mentioned our limitation in Appendix. F
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
21

---

Justification: We mentioned our Theory Assumptions and Proofs in Section 3.1 and 3.2 and Appendix. and respectively.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
and D we provide a detailed description of our model architecture and present all the training details, including dataset processing methods and hyperparameter settings.
Justification: In the Appendix, C
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
22

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We mentioned our data the Appendix, and code in supplemental material, respectively.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ https/cccccuclcccuueeeeeeeeeeeee Coaee//dddeSebmmmcma ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide the full details in Appendix, C and D
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We mentioned using random seeds to repeat at least 5 times to calculate the average in Table. and 2
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
23

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We mentioned in the Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: This paper strictly adheres the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
This work is foundational research and not tied to particular applications.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Yes, the paper properly credits the creators or original owners of assets (e.g..oode, data, models) and explicitly mentions and respects the relevant licenses and terms of use.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
25

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new assets introduced in this article.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "uJ9Z5snufw", "reviewer": "Reviewer_PWNz", "summary": "The paper presents a novel neuromorphic model for sound source localization (SSL) inspired by biological auditory systems. The model integrates spike-based neural encoding and computation, employing Resonate-and-Fire (RF) neurons with a phase-locking coding (RF-PLC) method. The RF-PLC method leverages the resonance properties of RF neurons to efficiently convert audio signals to time-frequency representations and encode interaural time difference (ITD) cues into discriminative spike patterns. Additionally, the model incorporates a spike-driven multi-auditory attention (MAA) module inspired by biological adaptations, which enhances SSL capability in noisy environments. The authors demonstrate that their model achieves state-of-the-art accuracy and robustness in real-world conditions.", "questions": "The introduction provides a clear motivation and context for the research. However, it could benefit from a more detailed discussion of existing SSL models and their limitations. The methodology is well-described, but some parts are overly technical and could be simplified for clarity. Diagrams illustrating the RF-PLC method and MAA module would be helpful.The results section presents compelling evidence of the model's performance. Including statistical significance tests and error bars in the figures would add rigor to the findings. The discussion touches on the implications of the results but lacks depth in exploring future work and potential applications. More emphasis on the practical applications of the model would be beneficial.", "limitations": "See questions.", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "Biological Inspiration: The model is well-motivated by biological mechanisms, which is a strong point given the effectiveness of biological systems in sound localization. Energy Efficiency: The use of RF neurons and phase-locking coding is innovative and contributes to the energy efficiency of the model. Robustness: The introduction of the MAA module significantly improves robustness and accuracy in noisy environments. Performance: The experimental results are promising, showing that the model achieves state-of-the-art accuracy and maintains high performance even at low signal-to-noise ratios.", "weaknesses": "Comparative Analysis: While the paper claims state-of-the-art performance, the comparative analysis with existing models is not comprehensive. The authors should provide more detailed comparisons and discuss why their model outperforms others. Complexity: The model's complexity and computational requirements are not adequately addressed. It's important to understand the trade-offs between performance gains and computational costs. Generalization: The experiments are promising but limited in scope. Additional tests across various datasets and real-world scenarios are needed to validate the generalizability of the model. Biological Plausibility: The biological plausibility of some components, such as the MAA module, is not thoroughly discussed. More insights into how closely these components mimic biological processes would strengthen the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your recognition of the innovative aspects and motivation of our work. In response to the weaknesses and questions, we will provide further detailed explanations:\n\n# W1: Compare Analysis\n**A**: Following your suggestion , we rigorously evaluated our work against previous SOTA ANN-based and SNN-based models in terms of both Acc and MAE. As demonstrated in the table, our model all achieved SOTA performance.\n|Methods|Type|MAE ($\\degree$)|Acc (%)|\n|:-:|:-:|:-:|:-:|\n|GCC-PHAT-CNN [1]|ANN|4.39$\\degree$|86.94%|\n|SELDnet [2]|ANN|1.78$\\degree$|88.24%|\n|EINV2 [3]|ANN|0.98$\\degree$ |94.64%|\n|SRP-DNN [4]|ANN|0.96$\\degree$|94.12%|\n|MTPC-CSNN [5]|SNN|1.23$\\degree$|93.95%|\n|MTPC-RSNN [5]|SNN|1.48$\\degree$|94.30%|\n|Hybrid Coding [6]|SNN|0.60$\\degree$|95.61%|\n|**Ours**|SNN|**$0.33\\degree$**|**96.40%**|\n\nWe attribute the superior performance of our model to the following two key parts:\n- **RF-PLC method**: RF neurons are strategically deployed to precisely isolate ITD cues, effectively eliminating auditory information that does not contribute to localization. This provides a reliable data foundation for high-performance localization.\n- **MAA module**: The MAA module draws on biological auditory principles of frequency band preference and short-term memory. The font enhances networks' focus on critical ITD cues within key frequency bands. The latter strengthens the model's memory for informed decisions across timeframes.\n\nOverall, our SSL model effectively integrates the proposed RF-PLC method and MAA module, resulting in demonstrably superior performance.\n\n# W2: Trade-offs between Performance and Computational Costs\n**A**: As per your suggestion, we conducted a detailed comparison across various SSL models. As shown in the following table, our model achieves competitive performance while reducing the model size by **1/10**. The results demonstrate that our model achieves the **best trade-offs** between performance and computational overhead.\n|Model|Param (M)|MAE ($\\degree$)|Acc (%)|\n|:-:|:-:|:-:|:-:|\n|Hybrid Coding [1]|1.61M|0.60$\\degree$|95.61%|\n|MTPC-RSNN [2]|1.67M|1.48$\\degree$|94.30%|\n|Spike-Driven V2 [3]|15.1M|0.25$\\degree$|97.10%|\n|**Ours**|1.64M|0.33$\\degree$|96.40%| \n\nReference:\n\n[1] A hybrid neural coding approach for pattern recognition with spiking neural networks. In: TPAMI (2024).\n\n[2] Multi-tone phase coding of interaural time difference for sound source localization with spiking neural networks. In: TASLP (2021).\n\n[3] Spike-driven Transformer V2: Meta Spiking Neural Network Architecture Inspiring the Design of Next-generation Neuromorphic Chips. In: ICLR (2024).\n\n# W3: Generalization\n**A**: As per your suggestion, we evaluated our model using the HRTF, Single Words, and SLoClas datasets, assembled from both simulations and real-world data collections. As detailed in the subsequent table, our model demonstrates SOTA performance across these datasets, thereby validating its generalizability.\n\n|Dataset|Previous SOTA Acc (%)|Ours Acc (%)|Azimuth Range|Resolution|\n|:-:|:-:|:-:|:-:|:-:|\n|HRTF [1]|97.38%|**99.84%**|$-60\\degree\\sim60\\degree$|10$\\degree$|\n|Single Words [2]|96.30%|**99.63%**|$-45\\degree\\sim45\\degree$|10$\\degree$|\n|**SLoClas [3]**|95.61%|**96.90%**|$-180\\degree\\sim180\\degree$|5$\\degree$|\n\nReference:\n\n[1] Spiking neural network model of sound localization using the interaural intensity difference. In: TNNLS (2012).\n\n[2] A biologically inspired spiking neural network model of the auditory midbrain for sound source localisation. In: Neurocomputing (2010).\n\n[3] A hybrid neural coding approach for pattern recognition with spiking neural networks. In: TPAMI (2024).\n\n# W4: Biological Plausibility  \n**A**: The biological plausibility of our model is demonstrated in two key aspects:  \n- **RF-PLC method**: This method utilizes RF spiking neurons and phase-locking mechanisms, both of which are biologically plausible. Neuroscientific evidence demonstrates that RF neurons, a specific type of biological neuron, selectively respond to particular input frequencies and efficiently process oscillatory signals [1]. Furthermore, phase-locking mechanisms are prevalent in the biological auditory system and facilitate the capture and encoding of temporally correlated information by neurons [2].\n- **MAA module**: The MAA module emphasizes frequency band preference and short-term memory. Neuroscientific research indicates the critical role of frequency selectivity in auditory processing [3]. Additionally, biological studies demonstrate that short-term memory enhances selective auditory attention in noisy environments [4].\n\nReference:\n\n[1] Dendritic channelopathies contribute to neocortical and sensory hyperexcitability in Fmr1−/y mice. In: NAT NEUROSCI (2014).\n\n[2] An oscillator model better predicts cortical entrainment to music. In PANS (2019).\n\n[3] Temporal coherence and attention in auditory scene analysis. In: TRENDS NEUROSCI (2011).\n\n[4] Attention improves memory by suppressing spiking-neuron activity in the human anterior temporal lobe. In: NAT NEUROSCI (2018).\n# Q1: Suggestions for Improving Writing\n**A**: Thank you for your constructive feedback on our manuscript. We will carefully consider your recommendations to simplify technical descriptions and enhance discussions of limitations relative to existing SSL models. \n# Q2: Experimental Reproducibility and Future Discussion\n**A**: As shown in the following table, all experiments were conducted **at least 5 times** to ensure reliability.Furthermore, **the code has been uploaded to the supplementary materials to enhance the reproducibility**. In future research, we aim to deploy and test our model on Loihi 2  or Tianjic, ultimately applying it to intelligent robots.\n|SNR (dB)|MAE ($\\degree$)|Acc (%)|\n|:-:|:-:|:-:|\n|50 dB (Low)|$0.33\\degree \\pm 0.02\\degree$|$96.40$\\% $\\pm$ $0.30$\\%|\n|20 dB (Middle)|$0.43\\degree \\pm 0.03\\degree$|$95.60$\\% $\\pm$ $0.23$\\%|\n|0 dB (High)|$0.54\\degree \\pm 0.04\\degree$|$94.90$\\% $\\pm$ $0.15$\\%|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZMsNxb7QIt", "reviewer": "Reviewer_2syp", "summary": "This work proposes a SNN-based model for SSL. To achieve efficient processing of raw\n speech signals, they introduce a phase-locking coding (RF-PLC) method using Resonate-and-Fire (RF)\n neurons and detection neurons.", "questions": "1. The improvements of performance compared with current SOTA methods are quite limited. And the paper did not compred with Spike-driven V2 and other SOTA methods. Hence the contribution on model performance improvements  is unclear.\n\n2. The novelty of proposed attention mechanisms  is limited compared to existing  attention methods for SNN.\n\n3. The experiments results should be conducted multiple trials. All source code could be provided to show the reproducability.", "limitations": "See mentioned above.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "This work proposes a SNN-based model for SSL. To achieve efficient processing of raw\n speech signals, they introduce a phase-locking coding (RF-PLC) method using Resonate-and-Fire (RF)\n neurons and detection neurons.", "weaknesses": "1. The improvements of performance compared with current SOTA methods are quite limited. And the paper did not compred with Spike-driven V2 and other SOTA methods. Hence the contribution on model performance improvements  is unclear.\n\n2. The novelty of proposed attention mechanisms  is limited compared to existing  attention methods for SNN.\n\n3. The experiments results should be conducted multiple trials. The source code could be provided to show the reproducability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your review comments. In response to the issues you have raised, we offer the following replies:\n\n# Q1: Limited Improvement\n- **Limited Improvement**: In SSL tasks, Acc and MAE are the most important metrics. As shown in the following table, extensive comparative experiments have demonstrated that our model not only achieves SOTA accuracy among similarly sized models but also reduces MAE by approximately **50%**. \n\n|Methods|Type|Model Size|MAE ($\\degree$)|Acc (%)|\n|:-:|:-:|:-:|:-:|:-:|\n|GCC-PHAT-CNN [1]|ANN|4.17M|4.39$\\degree$|86.94%|\n|SELDnet [2]|ANN|1.68M|1.78$\\degree$|88.24%|\n|EINV2 [3]|ANN|1.63M|0.98$\\degree$ |94.64%|\n|SRP-DNN [4]|ANN|1.64M|0.96$\\degree$|94.12%|\n|MTPC-CSNN [5]|SNN|1.61M|1.23$\\degree$|93.95%|\n|MTPC-RSNN [5]|SNN|1.67M|1.48$\\degree$|94.30%|\n|Hybrid Coding [6]|SNN|1.61M|0.60$\\degree$|95.61%|\n|**Ours**|SNN|1.64M|**$0.33\\degree$**|**96.40%**|\n \nReference can be found in Table.1 of the submitted manuscript.\n\n- **Compared with Spike-Driven V2 Model**:   As per your suggestion,  we replaced the back-end model with Spike-driven V1 and V2 models [1,2] for the ablation study.  As shown in the following table, our model achieves competitive performance while reducing the model size by **1/10**. Therefore, it achieves the **best trade-offs** between performance and computational costs.\n\n| Model | Param (M) | MAE ($\\degree$) |\n|:-----:|:---------:|:----------:|\n| Spike-Driven [1] | 16.81M | $0.27\\degree \\pm 0.05\\degree$ |\n| Spike-Driven V2 [2] | 15.1M | $0.25\\degree \\pm 0.04\\degree$ |\n|  **Ours**  | **1.64M** | $0.33\\degree \\pm 0.02\\degree$ |\n\nBy the way,  spike-driven methods are generic and high-performing, but they rely on extensive parameters. In contrast, our model is specifically designed for SSL tasks, fully considering the characteristics of ITD pulse sequences, thus it is both lightweight and efficient.\n\nReference:\n\n[1] Spike-driven Transformer. In: NeurIPS (2023).\n\n[2] Spike-driven Transformer V2: Meta Spiking Neural Network Architecture Inspiring the Design of Next-generation Neuromorphic Chips. In: ICLR (2024).\n\n# Q2: Novelty of Attention Mechanisms\n\n**A**: The novelty of our MAA module lies in three key aspects:\n- **MAC-free Computational Paradigm:** Compared with those attention methods [1,2,3] in SNNs, our methods achieve MAC-free attention computation paradigm. Therefore, it possesses significant energy efficiency advantages.\n\n- **Task-Specific Design:**  Our methods are designed to work well with the specific patterns of ITD spike trains from the RF-PLC method. This helps the front-end and back-end of the SSL model work better together, making the system both efficient and effective.\n\n- **Biological Plausibility:** The MAA incorporates frequency band preference and short-term memory, both grounded in strong biological evidence. Neuroscientific evidence indicates that frequency tuning varies across regions, highlighting the critical role of frequency selectivity in auditory processing [4]. Additionally, biological studies demonstrate that auditory short-term memory significantly enhances selective attention to auditory signals in noisy environments [5].\n\nReference:\n\n[1] Temporal-wise attention spiking neural networks for event streams classification. In: ICCV (2021).\n\n[2] TCJA-SNN: Temporal-channel joint attention for spiking neural networks. In: TNNLS (2024).\n\n[3] Attention spiking neural networks. In: TPAMI (2023).\n\n[4] Temporal coherence and attention in auditory scene analysis. In: TRENDS NEUROSCI (2011).\n\n[5] Attention improves memory by suppressing spiking-neuron activity in the human anterior temporal lobe. In: Nature Neuroscience (2018). \n# Q3: Experiment Detail\n\n**A**:  As per your suggestions, all experiments in this paper were conducted **at least 5 times** to ensure reliability. For example, the follow table shows the robustness performance of our methods in five independent experiments.  In addition, **we have uploaded all the code to the supplementary materials to help others replicate our results more easily**. \n\n|SNR (dB)|MAE ($\\degree$)|Acc (%)|\n|:-:|:-:|:-:|\n|50 dB (Low)|$0.33\\degree \\pm 0.02\\degree$|$96.40$\\% $\\pm$ $0.30$\\%|\n|20 dB (Middle)|$0.43\\degree \\pm 0.03\\degree$|$95.60$\\% $\\pm$ $0.23$\\%|\n|0 dB (High)|$0.54\\degree \\pm 0.04\\degree$|$94.90$\\% $\\pm$ $0.15$\\%|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r277BDjxrl", "reviewer": "Reviewer_ajui", "summary": "This paper introduces a spike-based neuromorphic model designed for sound source localization (SSL), capitalizing on the inherent properties of Resonate-and-Fire (RF) neurons. By encoding sound via phase-locking to leverage the resonance characteristics of these neurons, the model efficiently represents interaural time differences (ITD), facilitating precise sound localization. The inclusion of a multi-auditory attention (MAA) module, inspired by biological auditory systems, enhances the model's performance in noisy conditions by focusing on relevant sound frequencies and temporal sequences. Extensive experimental results demonstrate the model's state-of-the-art accuracy and robustness against noise, presenting an advance in neuromorphic SSL applications.", "questions": "__Q1: Training Time Cost__. I am curious about the time cost during the training phase. SNNs are known for their training overhead, even though they theoretically have lower power consumption during inference. It might be beneficial for the paper to discuss the additional overhead introduced by the three modules during training, perhaps including this as a metric in the ablation experiments.\n\n__Q2: Suitability of Neuromorphic Systems for SSL Tasks__. Why are neuromorphic systems more suitable for SSL tasks? From Table 1, it appears that the proposed SNN-based solution significantly outperforms ANN-based implementations, which is contrary to the common understanding that ANNs generally lead in performance compared to SNNs. I would appreciate insights from the authors on this discrepancy.", "limitations": "See Weaknesses and Questions.", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "__Comprehensive and Detailed Presentation__: The scheme is presented in full, with detailed descriptions of all modules' design thoughts and operational logics. Ablation experiments validate the effectiveness of each individual improvement.\n\n__Clarity and Empirical Validation__: The paper is well-written and clear, with the experimental section effectively demonstrating key metrics like Parameters, Power, and DoA. These metrics align well with the paper's goals of optimizing energy consumption and performance.", "weaknesses": "__Limited Dataset__: The experiments are conducted on a single dataset, which does not validate the universality of the proposed solution.\n\n__Incremental Costs in Module Stacking__: The design seems incremental by stacking modules that introduce additional costs to improve performance. Specifically, the introduction of additional Parameters and Power in the ablation studies (Table 2) appears to contradict the paper's motivation to reduce computing complexity and energy consumption. The trade-off between these aspects may need clarification.\n\n__Novelty__: The application of SNNs for SSL seems to have been first introduced in reference [38]. Thus, the contributions of this paper are optimizations based on [38], and the novelty needs further clarification.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for recognizing the strengths and quality contributions of our paper. In response to the questions you raised, we are providing further details and insights to clarify the points mentioned in your review.\n# Q1: Generalization\n**A**: As per your suggestion, we evaluated our model using the HRTF, Single Words, and SLoClas datasets, assembled from both simulations and real-world data collections. As detailed in the subsequent table, our model demonstrates SOTA performance across all the three datasets, thereby validating its generalizability.\n\n|Dataset|Previous SOTA Acc (%)|Ours Acc (%)|Azimuth Range|Resolution|\n|:-:|:-:|:-:|:-:|:-:|\n|HRTF [1]|97.38%|**99.84%**|$-60\\degree\\sim60\\degree$|10$\\degree$|\n|Single Words [2]|96.30%|**99.63%**|$-45\\degree\\sim45\\degree$|10$\\degree$|\n|**SLoClas [3]**|95.61%|**96.90%**|$-180\\degree\\sim180\\degree$|5$\\degree$|\n\nReference:\n\n[1] Spiking neural network model of sound localization using the interaural intensity difference. In: TNNLS (2012).\n\n[2] A biologically inspired spiking neural network model of the auditory midbrain for sound source localisation. In: Neurocomputing (2010).\n\n[3] A hybrid neural coding approach for pattern recognition with spiking neural networks. In: TPAMI (2024).\n\n# Q2: Incremental Costs in Module Stacking\n\n**A**:  We agree with the importance of balancing performance and computational costs. As per your suggestion, we conducted a detailed comparison across various SSL models. As shown in the following table, our model achieves competitive performance while reducing the model size by 1/10. The results demonstrate that our model achieves the **best trade-offs** between performance and computational overhead.\n\n|Model|Param (M)|MAE ($\\degree$)|Acc (%)|\n|:-:|:-:|:-:|:-:|\n|Hybrid Coding [1]|1.61M|0.60$\\degree$|95.61%|\n|MTPC-RSNN [2]|1.67M|1.48$\\degree$|94.30%|\n|Spike-Driven V2 [3]|15.1M|0.25$\\degree$|97.10%|\n|**Ours**|1.64M|0.33$\\degree$|96.40%| \n\nReference:\n\n[1] A hybrid neural coding approach for pattern recognition with spiking neural networks. In: TPAMI (2024).\n\n[2] Multi-tone phase coding of interaural time difference for sound source localization with spiking neural networks. In: TASLP (2021).\n\n[3] Spike-driven Transformer V2: Meta Spiking Neural Network Architecture Inspiring the Design of Next-generation Neuromorphic Chips. In: ICLR (2024).\n\n# Q3: Novelty of This Work\n**A**: Compared to the work referenced in [38], our study demonstrates innovation primarily in two aspects:\n- **Energy-efficient ITD Encoding:** Reference [38] uses FT operations during encoding, which needs massive computing resources. Our method innovatively employs RF spike neurons and phase-locking mechanisms to directly encode raw speech signals into sparse ITD spike trains. This approach is not only biologically plausible but also energy-efficient.\n- **Enhanced Performance in SSL Backend Networks:** The backend network in [38] is limited to convolutional architectures, which do not adequately consider the dependencies and preferences between the frequency and time domains of sound signals, resulting in suboptimal localization performance. In contrast, our MAA module innovatively incorporates the frequency band preferences and temporal interactions found in biological auditory systems. Our ablation studies robustly validate this enhancement.\n# Q4: Training Cost\n**A**: Based on your suggestion, we tested the training time overhead on ANN and SNN backends with identical architectures. Detailed experimental details are as follows:\n- **ANN**: Training was conducted on a single NVIDIA RTX 3090 (24G) with a batch size of 16. Each epoch took approximately **30s**, and the model required around 300 epochs to complete training.\n- **SNN**: Training was conducted on a single NVIDIA RTX 3090 (24G) with a batch size of 16 and a time step of 4. Each epoch took approximately **60s**, and the model required around 200 epochs to complete training. \n\nAs you produced, SNNs indeed require longer training times compared to ANNs. Efficiently training SNNs continues to be an unresolved issue within the field. Numerous studies have focused on mitigating the training challenges associated with SNNs through the development of efficient algorithms [1] and the implementation of dedicated hardware acceleration platforms [2,3].\n\nReference:\n\n[1] Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting. In: ICLR (2022).\n\n[2] Towards artificial general intelligence with hybrid Tianjic chip architecture. In: Nature (2019).\n\n[3] 22.6 anp-i: A 28nm 1.5 pj/sop asynchronous spiking neural network processor enabling sub-o. 1 μj/sample on-chip learning for edge-ai applications. In: ISSCC (2023).\n\n# Q5: Why Our SSL Model Performance Better\n**A**: This is a core highlight of our work. We attribute the superior performance of our model to the following three key parts:\n- **RF-PLC method**: RF neurons are strategically deployed to precisely isolate ITD cues, effectively eliminating auditory information that does not contribute to localization. This provides a reliable data foundation for high-performance localization.\n- **MAA module**: The MAA module draws on biological auditory principles of frequency band preference and short-term memory. The font enhances networks' focus on critical ITD cues within key frequency bands. The latter strengthens the model's memory for informed decisions across timeframes. \n- **Advantage of SNNs in SSL task**:  Information in SNNs is encoded within temporal spike trains, allowing these networks to directly and effectively capture temporal features. Given the high time resolution required for SSL tasks, SNNs are particularly well-suited for processing temporal audio signals.\n\nOverall, the proposed SSL model well-orchestrates the proposed RF-PLC method, MAA module and the inherent advantage of SNNs, and this is the main reason for its high performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "17EkFSQaNe", "reviewer": "Reviewer_zZvi", "summary": "This paper introduces a spike-based neuromorphic model for sound source localization. It utilizes the RF-PLC methods for auditory-like spectral analysis and encoding. Additionally, it is supported by the MAA module, which simulates attention mechanisms in specific biological frequency bands. These technologies are integrated to enhance the efficiency and accuracy of SSL Tasks.", "questions": "Please refer to the weaknesses section.", "limitations": "Please refer to the weaknesses section.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper describes the RF-PLC method and the MAA module, both of which together replicate the mechanisms of the mammalian auditory pathway, enhancing performance in SSL Tasks.\n2. The method is technically sound, with low computational energy consumption, facilitating deployment on neuromorphic computing platforms. In addition,the mathematical proofs are rigorous.\n3. The experimental results are comprehensive and reliable, effectively validating the method's effectiveness and robustness.", "weaknesses": "1. Regarding the design of the loss function: In practical settings, there should be no significant difference between the model outputs at 355° and 5°. It is worth designing the loss function to avoid this issue effectively. I hope the authors can provide a detailed discussion of the loss function.\n2. Concerning robustness validation of localization: To verify the model's accuracy under noisy and complex conditions, the paper involves adding noise to pure audio before classification. I hope the authors can provide a detailed discussion on how the noise dataset was constructed and why it was structured in this way.\n3. Validity of experimental methods: The paper appears to be tested only on the SLoClas dataset, which may not sufficiently demonstrate the effectiveness of the proposed method. It is hoped that the authors will test the method on more datasets. If no other public datasets are available, did the authors use any other methods to validate the effectiveness of the proposed approach?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your valuable feedback and for taking the time to read our paper. We hope the following responses will address your questions.\n\n# W1: How to Design Loss Function\n\n**A**: To achieve a $1\\degree$ resolution in SSL tasks, the output layer comprises 360 neurons, each representing a distinct azimuth angle[1]. Subsequently, we use the cross-entropy loss function to guide model training. Thanks to the design of the output layer, there is less need to focus on the discrepancies between the network's predictions and the actual outcomes.\n\nHowever, during model performance evaluation, the similarity between results at $355\\degree$ and $5\\degree$ can pose issues in accurately assessing model performance. We utilize the evaluation metrics [2] to valiate the performance of model. Specifically, toaddress the aforementioned issue, **MAE** is further defined as:\n\n$ MAE = \\frac{1}{N}\\sum_{i=1}^{N} 180\\degree -||\\hat{\\theta}-\\theta|-180\\degree|$\n\nReference: \n\n[1] Multi-tone phase coding of interaural time difference for sound source localization with spiking neural networks. In: TASLP (2021).\n\n[2] A hybrid neural coding approach for pattern recognition with spiking neural networks. In TPAMI (2024).\n\n# W2: Robustness Validation of Localization\n\n**A**: Considering the potential presence of noise in SSL tasks, it is crucial to assess model performance in noisy environments. As described in Appendix C, we introduce the same noise information, $noise(n)$, into each microphone channel $mic_i(n)$ to simulate noise coming from various directions. The dataset can be represented as:\n\n$ComplexVideo_i (n) = mic_i (n) + λnoise(n), i=1,2,3,4$\n\nAdditionally, we can utilize the Signal-to-Noise Ratio (SNR) to represent the complexity of the dataset after adding noise.\n\n$SNR(dB) = 10 log_{10}(\\frac{P_{signal}}{P_{noise}}) $\n\n# W3: Validity of Experimental Methods\n\n**A**: The SLoClas dataset stands as one of the most **challenging** benchmarks for SSL tasks. In contrast, other datasets, such as those employed in studies [1] and [2], suffer from limited localization precision, making them less representative of real-world scenarios. Moreover, we have also tested our model on datasets [1] and [2], where it achieved nearly 100% accuracy. Therefore, we are particularly focused on the performance of the SLoClas dataset, as it is the only one collected in a real-world environment.\n\n|Dataset|Previous SOTA Acc (%)|Ours Acc (%)|Azimuth Range|Resolution|\n|:-:|:-:|:-:|:-:|:-:|\n|HRTF [1]|97.38%|**99.84%**|$-60\\degree\\sim60\\degree$|10$\\degree$|\n|Single Words [2]|96.30%|**99.63%**|$-45\\degree\\sim45\\degree$|10$\\degree$|\n|**SLoClas [3]**|95.61%|**96.90%**|$-180\\degree\\sim180\\degree$|5$\\degree$|\n\nReference:\n\n[1] Spiking neural network model of sound localization using the interaural intensity difference. In: TNNLS (2012).\n\n[2] A biologically inspired spiking neural network model of the auditory midbrain for sound source localisation. In: Neurocomputing (2010).\n\n[3] A hybrid neural coding approach for pattern recognition with spiking neural networks. In: TPAMI (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lR65ln7F3C", "reviewer": "Reviewer_ym4t", "summary": "This paper constructs a sound source localization model by leveraging efficient spiking neural networks and biologically-inspired auditory localization mechanisms. Although numerous studies have previously approached this subject from a biomimetic perspective, this paper commendably considers the balance between biomimicry and practicality, while also addressing the robustness of the SSL model. It offers a new perspective on the effective integration of artificial neural networks and biological intelligence.", "questions": "1.The authors are asked to explain how to determine the azimuth of a sound source, including an example of encoding an audio segment and predicting the azimuth.\n2.The effectiveness of the MAA Block needs verification. While Table 2 shows improved localization performance without added computational demand, further experiments should confirm these results are not due to other factors like increased parameter counts.\n3.Although this work focuses on the algorithm of SSL and I agree that the workload is sufficient, I wonder whether the RF neuron has the potential to be implemented on current neuromorphic hardware or FPGA.", "limitations": "None", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The paper proposes substituting the energy-intensive Fourier Transform (FT) operations with frequency band preferences of RF neurons, which is highly suitable for the edge computing scenarios mentioned by the authors. This consideration is notably absent in most related works.\n2.In the appendix, the authors provide a detailed and reliable mathematical proof explaining why RF neurons can effectively replace FT operations. The results are robust and convincingly presented.\n3.Inspired by the mechanisms of biological auditory localization, the paper designs an efficient SSL encoding and model. This innovative approach is commendable and encourages further exploration in the field.", "weaknesses": "1.It remains insufficient for readers without a background in SSL tasks or RF technology. Although the supplementary materials are helpful, they seem not to adequately bridge the knowledge gap for those outside the specialty.\n2.I understand that the outputs of the RF-PLC method will serve as inputs for the classification model. It would be beneficial if the authors could provide a step-by-step example that shows how a given audio input is encoded into a spike pattern, and how these encoded outputs are subsequently utilized to determine the azimuth of a sound source relative to the listener.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your recognition of our work. In response to the questions you raised, we provide the following replies:\n\n# W1: Enhance the Background Introduction\nWe strongly agree with your perspective. In the background section, we will incorporate additional research on SSL tasks and compare methodologies. Additionally, we will further augment the appendix with theoretical proofs of the formulas.\n# W2: Intuitive Demonstration of Model\nWe greatly appreciate your suggestions. We will revise Fig.1 to more clearly demonstrate the function of each part of the model.\n\n# Q1: How to Determine the Azimuth of the Sound Source\n\n**A**: We will detail the function of each part of the model and its data format, as well as how to ultimately determine the azimuth.\n- **RF-PLC Method**: An audio is initially segmented into a sequence of frames, denoted as $x\\in R^{T \\times C_{mic} \\times L}$, where $T$ represents the number of frame sequences, $C_{mic}$ indicates the number of microphone channels, and $L$ denotes the length of each frame sequence. Subsequently, audio from different channels is paired and input into the RF-PLC encoding section, resulting in the encoding $E\\in Z^{T \\times C \\times F \\times \\tau}$, where $C$ represents the number of channel combinations, $F$ denotes the number of frequency band decompositions, and $\\tau$ indicates the number of detection neurons.\n- **Back-end Model**: Encoding result $E\\in Z^{T \\times C \\times F \\times \\tau}$ will be input into a network featuring the MAA module. It primarily consists of **frequency band preference** and **short-term memory components**. The frequency band preference ensures the network focuses more on crucial frequency band information, while the short-term memory enhances the temporal capabilities of neurons. Combined, these features significantly boost the environmental robustness of the backend network. Finally，the output layer of the network consists of 360 units, representing the predicted outcomes.\n\n# Q2: Effectiveness of the MAA Module\n\n**A**: To ensure that the effectiveness of the MAA module is **NOT** merely due to an increase in parameter count, we conducted additional ablation experiments detailed in the supplementary materials. As depicted in Fig.9 and below Table, 'Local-XXX' represents the scenario where the module is replaced with a convolutional layer of equivalent parameter size. The results clearly demonstrate that even with an equal number of parameters, our module exhibits superior performance, underscoring its unique benefits.\n\n\n| model | Type | Param (M) | MAE ($\\degree$) | ACC. (\\%) |\n|:-----:|:---------:|:----------:|:---------:|:---------:|\n|  LocalST-M | SNN | 1.62M | $0.48\\degree$ |  $95.40$\\% | \n|  ST-M | SNN | 1.62M | $0.45\\degree$ |  $95.67$\\% | \n|  LocalFSJA | SNN | 1.63M | $0.62\\degree$ |  $95.40$\\% | \n|  FSJA | SNN | 1.63M | $0.49\\degree$  |  $95.95$\\% | \n|  LocalMAA ( LocalST-M + LocalFSJA) | SNN | 1.64M | $0.62\\degree$ |  $95.40$\\% | \n|  **MAA ( ST-M + FSJA)** | SNN | 1.64M | **$0.33\\degree$** |  **$96.40$\\%** | \n\n\n# Q3: Hardware-friendly of RF Neurons\n\n**A**: We agree with your view. RF neurons and our SSL model are indeed compatible with current neuromorphic platforms like Loihi 2 [1] and Tianjic [2]. Yet, substantial challenges hinder their broader adoption. Primarily, the high costs and varied interfaces of open-source edge devices like robots limit practical deployments. Furthermore, the scarcity and procurement difficulties of neuromorphic chips or specialized hardware impede the effective utilization of SNNs.\n\nReference: \n\n[1] Efficient Video and Audio Processing with Loihi 2. In: ICASSP (2024).\n\n[2] Towards artificial general intelligence with hybrid Tianjic chip architecture. In: Nature (2019).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8gq57ba3ZP", "reviewer": "Reviewer_uJwn", "summary": "This study draws inspiration from the intrinsic mechanisms of sound source localization in biological auditory systems to design an efficient and robust SSL model. The core contributions include two primary aspects: firstly, replacing the energy-intensive Fourier Transform(FT) operations with RF neurons in the auditory encoding process, thereby encoding Interaural Time Differences (ITD) as spatiotemporal spike trains. Secondly, the backend SSL model effectively implements frequency band preferences and memory-based decision-making processes observed in biological hearing.", "questions": "1. The short-term memory structure proposed seems to only consider relationships between adjacent frames. Could methods involving multi-frame integrated decision-making be explored?\n2. The authors point that RF neurons can effectively perform the basilar membrane's frequency band decomposition in the cochlea. Is there a biological precedent for similar RF neurons in the cochlea?", "limitations": "While the proposed neuromorphic SSL model demonstrates promising results, several limitations should be acknowledged:\n\n1. **Lack of Edge Device Deployment:** The current study does not include the deployment of the proposed SSL model on edge devices. Practical implementation and testing on such devices are crucial to validate the model's real-world applicability and performance.\n\n2. **Experimental Validation of RF Neurons:** The claim that RF neurons can replace energy-intensive Fourier Transform operations lacks sufficient experimental validation. Further empirical evidence or references to existing research are needed to substantiate this claim convincingly.\n\n\nBy addressing these limitations, future research can provide a more comprehensive evaluation of the neuromorphic SSL model and its potential impact on sound source localization and broader applications in neuromorphic computing.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall, this is an interesting work. The strengths are outlined as follows:\n\n1. The combination of RF neurons and Jeffress models creates a robust auditory localization encoding method. RF neurons effectively replace high-energy Fourier Transform operations. Additionally, the multi-band cooperative decision-making for ITD encoding significantly enhances robustness.\n2. The authors have thoroughly considered the full-spike information transmission mechanism of SNNs, providing a composite spike-driven attention paradigm for similar future tasks. Figure.5 clearly illustrates the advantages of the proposed spiking attention.\n3. The article provides a clear description of the biological plausibility of the encoding and backend SSL model, making it easy to understand.\n4. The visualizations within the paper are outstanding. The diagrams in Figures 1 and 4 are densely packed with information, yet the text and captions effectively guide the reader to comprehend them well.\n5. The mathematical proofs in the appendix are comprehensive and credible.", "weaknesses": "While the paper is generally convincing, I believe the following areas require attention:\n1. The claim that RF neurons can effectively replace high-energy Fourier Transform operations needs further experimental validation or existing research evidence. I suggest the authors bolster this section with reliable proof.\n2. While there are currently no datasets to test localization robustness, the experiments on robustness lack detailed descriptions of how noise was introduced in the SLoClas dataset. \n3. Despite the high cost of deploying the proposed sound localization model on hardware, I recommend that the authors attempt actual deployment on neuromorphic agents in future work to empirically verify model performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your appreciation of our paper. In response to the issues you raised, we provide the following responses:\n\n# Q1: Mathematical Equivalence of RF Neurons and FT Operations\n\n**A**: The mathematical proof demonstrating the **equivalence** of RF neurons as an efficient substitute for FT is provided in Appendix A, with the equivalence relationship expressed as follows:\n\n$ FT[k] = \\sum_{n=0}^{N-1} x[n] e^{-\\mathrm{i}\\frac{2\\pi}{N}nk} = \\sum_{n=0}^{N-1} x[n] \\left( \\cos{\\left(\\frac{2\\pi}{N}nk\\right)} - \\mathrm{i} \\sin{\\left(\\frac{2\\pi}{N}nk\\right)} \\right) $\n\n$ RF[k] = \\sum_{n=1}^{T} \\lambda_n \\cos(n\\omega_k \\Delta t) - \\mathrm{i} \\sin(n\\omega_k \\Delta t) x[t-n] $\n\nMoreover, In Figure 5(a), we validated the error between RF-ITD and FT-ITD across a substantial audio dataset, and the **error ratio** between these two methods was only **1%**.\n\n# Q2: Why no Datasets for Robustness\n\n**A**: Currently, there are no existing SSL datasets specifically designed for **noisy environments**. To validate robustness in all current SSL tasks, noise is artificially added, as demonstrated in the study [1,2,3]. Furthermore, we have detailed the process of adding noise to the SLoClas dataset and the preparation of robustness experiments in Appendix C.\n\n$ComplexVideo_i (n) = mic_i (n) + \\lambda noise(n), i=1,2,3,4$\n\nReference:\n\n[1] Multi-tone phase coding of interaural time difference for sound source localization with spiking neural networks. In: TASLP (2021).\n\n[2] A hybrid neural coding approach for pattern recognition with spiking neural networks. In: TPAMI (2023).\n\n[3] SLoclas: A database for joint sound localization and classification. In: O-COCOSDA (2023).\n\n# Q3: Deploying the Proposed SSL Model on Hardware\n\n**A**: We fully agree with you. However, we faced significant challenges in this area. Firstly, open-source edge devices, such as robots and robotic dogs, are costly and have inconsistent interfaces. Secondly, the optimal deployment of SNNs is contingent on neuromorphic chips or specialized hardware, which are both scarce and difficult to procure.\n\n# Q4: Further Analysis of Short-term Memory Structures\n\n**A**: We fully agree with your perspective and have conducted extensive experimental studies in ablation study. Our research explored the performance of **adjacent frames (our)**, **multi-frame**, and **global frames (TA [1])**. The ablation study results, shown in the below table, indicate that adjacent frames yield the best performance. This is because, while global and multi-step frames introduce additional useful information, they also introduce **more noise**, hindering the model's ability to quickly capture key ITD features. Moreover, the temporal interactions in global and multi-step frames require multiple audio frames for decision-making, undermining the model's real-time localization.\n\n| model | Type | Param (M) | MAE ($\\degree$) | ACC. (\\%) |\n|:-----:|:---------:|:----------:|:---------:|:---------:|\n| baseline | SNN | 1.61M | $1.23\\degree$ |  $93.95$\\% | \n| global frames (TA [1]) | SNN | 1.62M | $0.65\\degree \\pm 0.05\\degree$ |  $93.37$\\% $\\pm$ $1.2$\\% | \n| multi-frames | SNN | 1.62M | $0.62\\degree \\pm 0.04\\degree$  |  $93.47$\\% $\\pm$ $1.1 $\\% | \n| **adjacent frames (ours)** | SNN | 1.62M | **$0.45\\degree \\pm 0.03\\degree$** |  **$95.67$\\% $\\pm $$0.5$\\%** | \n\nReference:\n\n[1] Temporal-wise Attention Spiking Neural Networks for Event Streams Classification. In: ICCV (2021).\n\n# Q5: A Biological Precedent for Similar RF Neurons in the Cochlea\n\n**A**: Yes, RF neurons do have biological precedents in the cochlea. However, it's important to note that RF neurons are not a simplification of any specific biological cell. Instead, they are a mathematical abstraction of the combined functions of the cochlear basilar membrane and inner hair cells [1,2].\n\nReference:\n\n[1] Dendritic channelopathies contribute to neocortical and sensory hyperexcitability in Fmr1−/y mice. In: NAT NEUROSCI (2014).\n\n[2] An oscillator model better predicts cortical entrainment to music. In PANS (2019).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a novel Sounds Source Localization (SSL) model that is based on Spiking Neural Networks (SNNs). Specifically, the paper use RF-PLC neurons to encode interaural time differences that are critical to SSL. These neurons avoid the significant computational overhead of calculating Fourier transforms as used in state-of-the-art (SOTA) SSL baselines. Additionally, the authors propose a SNN based multi-auditory attention module which implements biologically inspired properties of frequency selectivity and short-term auditory memory, which is known to improve SSL accuracy and robustness in noisy environments. The paper presents experiments demonstrating the superior accuracy and robustness of the proposed SNN-SSL on SloClas dataset. Analytical energy estimates show that the proposed SSL model has comparable energy to the SOTA SNN baseline. 

The reviewers primarily suggested adding more datasets and more SNN/ANN SSL baselines to compare the proposed model against. The authors have done a great job by adding SSL accuracy data for more datasets and other baselines and all experiments continue to show strong accuracy results for the proposed SSL. Spike-Driven V2 is one baseline that achieves a better accuracy than this paper, but as the authors highlight, it also has ~10x more parameters compared to the proposed model. Consequently, the proposed SSL offers a better tradeoff when it comes to accuracy vs model size. 

The authors showed great flexibility by adding ablation experiments that further strengthened the justification for their model design choices. To conclude, the paper presents a novel SSL model which is biologically motivated and shows high accuracy and robustness for the given model size. I recommend that this paper should be accepted for NeurIPS.

Minor note, the authors should proof-read their submission for small grammatical errors and try to organize the paper to improve its readability. For e.g., Table 2 shows useful ablation study data, but it could use a more detailed description describing all the experiment configs. Also, the authors show comparable energy consumption to the existing SNN-SSL baseline in Table 2, adding a comment on that in the results section of the paper will be helpful.

Edit: Updated my recommendation to poster (prev spotlight) post discussion with SAC.
Decision: Accept (poster)