Spike-based Neuromorphic Model for Sound Source Localization

Authors: dehao  zhang
shuai  wang
ammar  belatreche
wenjie  wei
yichen  xiao
haorui  zheng
zijian  zhou
malu  zhang
yang  yang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 11:14:48

Abstract: Biological systems possess remarkable sound source localization (SSL) capabilities that are critical for survival in complex environments. This ability arises from the collaboration between the auditory periphery, which encodes sound as precisely timed spikes, and the auditory cortex, which performs spike-based computations. Inspired by these biological mechanisms, we propose a novel neuromorphic SSL framework that integrates spike-based neural encoding and computation. The framework employs Resonate-and-Fire (RF) neurons with a phase-locking coding (RF-PLC) method to achieve energy-efficient audio processing. The RF-PLC method leverages the resonance properties of RF neurons to efficiently convert audio signals to time-frequency representation and encode interaural time difference (ITD) cues into discriminative spike patterns. In addition, biological adaptations like frequency band selectivity and short-term memory effectively filter out many environmental noises, enhancing SSL capabilities in real-world settings. Inspired by these adaptations, we propose a spike-driven multi-auditory attention (MAA) module that significantly improves both the accuracy and robustness of the proposed SSL framework. Extensive experimentation demonstrates that our SSL framework achieves state-of-the-art accuracy in SSL tasks. Furthermore, it shows exceptional noise robustness and maintains high accuracy even at very low signal-to-noise ratios. By mimicking biological hearing, this neuromorphic approach contributes to the development of high-performance and explainable artificial intelligence systems capable of superior performance in real-world environments.

Main paper: 
 introduction: 
 Sound source localization (SSL) [11, 43] is a critical skill for mammals that enables them to identify and locate external auditory stimuli. This skill plays a vital role in survival behaviors like prey detection and predator evasion. Over decades of scientific exploration [39, 48] , SSL has evolved from a purely biological concept to a sophisticated technology with a wide range of applications across various fields [10, 39] . Today, SSL methods are finding increasing use in areas like security monitoring [12] , robotic navigation [37] , and autonomous driving [13, 36] .Early SSL approaches rely on hand-crafted analysis of speech signals from multiple receivers. While offering a basic ability to localize sound sources, these methods suffer from limitations in accuracy and robustness. The emergence of Deep Neural Networks (DNNs) and their success in various domains lead researchers to explore their application in SSL tasks, achieving significant performance improvements [64, 67] . However, DNN-based approaches face two key challenges. Firstly, DNNs achieving high SSL accuracy often require substantial computational resources, leading to increased energy consumption. Secondly, DNNs struggle to learn the intricate relationships between localization behaviors and noisy environmental constraints. These limitations hinder the development of portable, edge-based SSL models [16] for real-world environments.Recently, Spiking Neural Networks (SNNs) [14, 20, 34] , inspired by brain neural architectures, have gained significant attention for their energy-efficient simulation of neural systems. Spiking neurons [33] simulate the information transmission mechanism of biological neurons, computing only upon the arrival of input spikes and remaining silent otherwise [53] . Such an event-driven mechanism results in sparser information transmission [60, 61] , hence reducing computational costs [6, 69] . Therefore, the SNN-based SSL models enable a more energy-efficient emulation of biological SSL processes. Pan et al. [40] propose a SNN-based SSL model that achieves localization in real audio signals. Chen et al. [7] improve the model's performance through a hybrid encoding method, achieving competitive results with less energy consumption. Although these examples achieve edge-friendly SSL ability, limitations still exist in neural encoding efficiency and robustness under noisy conditions.In terms of neural encoding, most methods still rely on Fourier Transform (FT) [27, 55] to encode ITD [11] present in the received audio signals into spike trains for processing by back-end SSL model. However, FT operations involve many multiply-accumulate (MAC) computations and require significant computational resources [58] which hinders our goal of developing energy-efficient SSL models. In terms of robustness, the superior localization ability in biology not only relies on ITD cues but also on various auditory mechanisms [22, 54] , such as frequency preferences, short-term memory, etc. Frequency preference significantly mitigates the impact of complex environments on localization accuracy [31, 62] , and auditory short-term memory effectively filters out irrelevant noise [49, 50] , focusing on important auditory signals. However, most SNN-based solutions [3, 32] primarily focus on ITD cues, with little attention to multiple auditory mechanisms. Therefore, investigating more energy-efficient and robust SNN-based SSL models remains a pressing challenge to address. In this paper, we propose a novel SNN-based SSL framework, which primarily comprises an ITD encoding front-end method and a biomimetic localization back-end model. As illustrated in Fig. 1 , we introduce a phase-locking coding (RF-PLC) method using Resonate-and-Fire (RF) neurons [8, 38] and detection neurons [40] . It simulates the frequency band decomposition function of the basilar membrane and captures ITD cues, respectively. Furthermore, we introduce a novel back-end SSL model based on multi-auditory attention (MAA) that integrates frequency preferences and short-term memory characteristics. This approach significantly improves both the accuracy and robustness of localization. Extensive experiment on the HRTF [57] , Single Words [30] , and SLoClas [42] datasets demonstrates that our SNN-based model achieves state-of-the-art performance. Moreover, evaluation in noisy environments reveals the model's enhanced adaptability to real-world conditions. The work introduces the following key contributions:• Spike-based ITD encoding: The RF-PLC method leverages the resonance properties of RF neurons to perform energy-efficient auditory time-frequency transformations, avoiding the high resource costs of FT operations. Additionally, it utilizes a phase-locking loop and ITD detection neurons to encode auditory ITD cues into spike patterns directly, ensuring the fully spike-driven nature of the entire SSL framework. • Biologically inspired attention: The MAA module incorporates knowledge of biological auditory frequency preferences and short-term memory characteristics. Frequency preferences effectively mask the ITD information of irrelevant frequency bands and spatial regions, while short-term memory focuses on the interaction of information across adjacent time steps. This combination enhances the robustness of the SNN model in noisy environments. • State-of-the-art performance with reduced complexity: By integrating these methods, we present a SNN-based SSL framework that achieves state-of-the-art performance while reducing energy consumption to existing works. Additionally, extensive experimentation demonstrates that our system exhibits superior robustness in noisy environments.2 Related Work 
itd cues for ssl tasks: 
 To develop biologically inspired models for SSL tasks, researchers have drawn upon the auditory localization mechanisms observed in mammals [23, 25] . The cues of ITD are recognized as critical for these models [29, 44, 46, 47] . ITD refers to the temporal disparity in sound arrival between the ears. Specifically, when a sound source is closer to the listener's right side, audio reaches the right ear sooner than the left. The Jeffress model [3] and BiSoLaNN [56] encode ITD cues into spike trains and corroborate their biological credibility through experiments on barn owls [5] . But these approaches primarily focus on the localization of pure tones, which significantly differs from the time-varying audio signal in daily life. Substantially, some researchers [7, 40] have utilized complex FT operations to obtain the phase information of audio signals. However, FT operations require substantial computational resources and pose significant challenges when implementing systems on edge devices with limited computational capabilities. Therefore, the exploration of low-power ITD encoding methods become a pressing direction to pursue.In the field of auditory science, frequency preference and short-term memory characteristics are essential for understanding auditory processing. Numerous studies [21, 52, 63] have demonstrated that biological auditory systems exhibit heightened sensitivity to specific frequency ranges, such as 20-20 000 Hz in humans, with other species like bats and blue whales adapted to different ranges. Further research [51] has revealed tonotopic maps and variations in frequency tuning across regions, underscoring the importance of frequency selectivity in hearing. Electrophysiological experiments [31] confirmed that inner hair cells on the basilar membrane of the cochlea exhibit significant differences in response to various frequency bands.These studies underscore the irreplaceable role of frequency band preference in auditory decisionmaking. Additionally, compared to visual short-term memory, auditory short-term memory [2, 28] has a shorter retention span. Nonetheless, it is essential for real-time integration and coherent environmental perception. Simultaneously, some researchers [50] propose that neurofeedback training targeting auditory short-term memory can significantly enhance selective attention to auditory signals in noisy environments. Moreover, Zhong et al. [70] suggested that auditory short-term memory can highlight sound source characteristics under reverberant conditions, reducing interference from other sources. However, current SSL methods mainly focus on ITD cues, neglecting these well-established biological mechanisms. Therefore, the effective integration of diverse auditory attention mechanisms within SSL tasks to enhance robustness remains a significant ongoing challenge. 
method: 
 In this section, we introduce our spike-based SSL framework, which includes a front-end ITD encoding method and a back-end localization model. For the front-end ITD encoding, we propose the energy-efficient RF-PLC method, which uses RF neurons to capture ITD cues and detection neurons to convert these encoded cues into spike patterns. For the back-end localization model, we take inspiration from biological adaptation and propose the MAA mechanism to enhance the model's localization performance and robustness.3.1 RF Phase-locking Coding: A Direct Font-end ITD Encoding Method Due to the physical separation of the ears, sound waves arrive at each ear with slightly different timing. It leads to differences in the initial phase information between the two audio channels. Pan et al. [40] propose a Multi-Tones Phase Coding (MTPC) method that utilizes this information to exploit ITD cues. However, this approach relies on computationally expensive FT operations and introduces an additional phase transformation step during processing. To overcome these limitations, we propose the RF-PLC method, leveraging RF neurons' resonance filtering and periodic decay properties. This approach effectively eliminates the need for energy-intensive FT and phase transformation processes. Subsequently, a set of detection neurons with varying delays is employed to efficiently encode the ITD cues from different microphones into spike patterns. The first step of our model processes the raw audio to capture ITD cues. We segment the audio into y l based on the smallest durations by the human ear. These segments are then encoded by specialized RF neurons [38] tuned to different frequency bands. The dynamics of these RF neurons can be described as follows:EQUATION) where ω represents the resonant frequency of the neuron, indicating the number of radians it progresses per second. λ serves as the dampening factor, which causes the oscillation to decay exponentially. ∆t represents the sampling rate, which is set to 1. I[t] denotes the audio input. Z RF [t] can be reformulated as x + iy ∈ C. A detailed process can be found in Appendix. A. The real component x of Z RF reflects the current-like behavior of the neuron, capturing the dynamics of voltage-gated and synaptic currents. The imaginary component y of Z RF [t] serves as a voltage-like variable.Based on Eq. 1, we depict the spiking behavior of the RF neuron in Fig. 2 (a) and summarize its characteristics in two aspects. Firstly, the complex form of the RF neuron's dynamics enables it to capture the phase information in a specific frequency band ω, termed resonance filtering. Secondly, the dampening factor λ allows it to exhibit periodic decay characteristics when there is no input.By leveraging RF neurons' resonance filtering and periodic decay properties, we encode input signals into ITD cues efficiently and effectively. To better describe the RF-PLC process, we decompose the dynamics of the RF neurons into two stages: a silent stage and a spike stage. The silent stage is utilized to decompose audio information into distinct frequency components and store this data in the state of the RF. The spike stage then oscillates phase information, effectively converting it into ITD cues through phase-locking mechanisms. These stages can be described as follows:EQUATION)In the silent stage, RF neurons with distinct ω values selectively respond to specific resonant frequencies. As illustrated in Fig. 2 (b), when the frequency ω 1 of the audio input I[t] closely matches the RF neuron's resonant frequency ω, a significant increase in its membrane potential occurs. Conversely, misalignment between these frequencies leads to a slower accumulation of membrane potential. This characteristic offers an energy-efficient alternative to the computationally expensive FT operations. The result of the silent stage can be interpreted as analogous to the initial phase information of each pure sinusoidal component within the audio signal. In the spike stage, we introduce a PLC method that ensures the RF neuron fires a spike only at a specific phase. Specifically, the spike firing time t lock is defined as the special state when the real part of the RF neuron state reaches zero and the imaginary part reaches its maximum (Z RF [t lock ] = 0 + iy max ). This precise spike timing can be directly utilized as an ITD cue, with details validated in Appendix B. Notably, the periodic decay characteristic of RF neurons guarantees that only one spike is generated using our PLC method, ensuring the efficiency of ITD encoding.As illustrated in Fig. 3 (a), we provide a schematic representation for obtaining ITD cues from input audio. During the silent stage, RF neurons receive audio signals and convert them into phase information of pure tones at different frequencies. During the spike stage, the PLC method leverages this phase information to determine whether the RF neuron fires a spike. The precise spike timing of the RF neuron serves as the ITD cue. Compared to traditional FT-based methods that rely on multiple network layers, our RF-PLC significantly reduces computational costs and offers a more biologically plausible representation of ITD cues.The final step of the RF-PLC method involves detection neurons that convert spike times (also, ITD cues) into spike patterns. As illustrated in Fig. 3 (b), a series of detection neurons are used in each band, with each neuron tuned to a specific delay preference (from τ 1 to τ n ). These detection neurons then encode the overall ITD cues for the audio signal. Interestingly, similar symmetrical detection structures have been observed in mammalian auditory pathways [15] , which support the biological plausibility of our approach.After encoding audio signals into spike patterns, we construct a back-end SNN-based model to process this encoded information for SSL tasks. The SNN-based SSL model is built based on the Leaky Integrate-and-Fire (LIF) neuron due to its computation efficiency. The LIF model receives the resultant current and accumulates membrane potential which is used to compare with the threshold to determine whether to generate the spike. Its dynamic can be described as follows:EQUATIONEQUATIONH[t + 1] = V reset S[t + 1] + τ U [t + 1](1 -S[t + 1]).(5) At each time step t + 1, the spatial input current X[t + 1] is obtained through convolution and linear layers. This current integrates with the previous temporal input H [t] to update the membrane potential U [t + 1]. The Heaviside step function Θ(•) determines whether the binary spike S[t + 1] is generated by comparing the membrane voltage with the threshold V th .If there is spike emission, H[t] is reset to the resting potential V reset ; otherwise, U [t + 1] decays with a time constant τ and directly feeds into H[t + 1]. We denote the LIF spiking neuron layer as SN (•), which takes X[t + 1] as input and produces the spike tensor S[t + 1] as output. Existing back-end SNN-based SSL models only rely on simple convolutional and fully connected layers for localization, without considering biological adaptation mechanisms such as frequency band selectivity and short-term memory. This leaves substantial room for improvement in localization performance. Therefore, we draw on these biological mechanisms to propose the computationally efficient MAA.In the field of SNNs, there have been some studies on attention mechanisms [19, 65, 71] . However, these methods almost rely heavily on squeeze-and-excitation operations, which introduce additional MAC operations. Therefore, we propose a novel spike-driven MAA mechanism that comprises a frequency-spatial joint attention module and a short-term memory structure. The former enhances networks' focus on critical ITD cues within key frequency bands. The latter strengthens the model's memory for wise decisions across timeframes. Notably, our MAA module is tailored for SSL tasks and achieves the best trade-off between performance and efficiency. To enhance adaptive learning and selection of preferred frequency in our SNN-based SSL model, we propose a spike-driven FSJA module. For each time step, the output of the RF-PLC method is defined as X[t] ∈ R C×F ×S , where F and S respectively represent the number of RF and detection neurons, and C denotes the microphone array. The FSJA module can be expressed as follows:Z = SN ( 1 F × S F i=1 S j=1 X i,j [t]), Att F S (Z) = SN (ConvBN(Z)) , FSJA = Att F S (Z)•X[t], (6)where ConvBN is convolution operations with a 3 × 3 kernel and batch normalization. The matrix Z is defined as the average of X[t] across the F dimension and S dimension and the spike results after passing through the LIF neuron. Due to the attention map of FSJA module is in binary spike form, it effectively concentrates on spike information at specific frequency and spatial dimensions.To further demonstrate the difference between our FSJA module and previous SNN-based attention methods across frequency and spatial dimensions, we show the difference between them. As shown in Fig. 4 (a), the existing SNN-based attention module relies on full-precision values. Although the broadcast operation is a spike-driven computational paradigm, it introduces additional MAC operations for the next layer. As shown in Fig. 4 (c), it effectively avoids MAC-based broadcasting operations which substantially improves energy efficiency. Additionally, our method effectively masks the ITD information of irrelevant frequency bands and spatial regions, substantially boosting the SSL model's robustness in noisy environments.The auditory short-term memory characteristic enables sustained perception of SSL processing, yet few studies have focused on this aspect. Although the membrane potential accumulation of spiking neurons partially reflects this mechanism, its simplified mathematical expression is insufficient for describing short-term memory adequately. Therefore, we develop an innovative ST-M structure that emphasizes the interaction of information across adjacent time steps to enhance the neuronal memory capacity. The structure can be represented as:EQUATIONwhere In[t] represents the preliminary feature extraction of the input X[t], and ST [t] denotes the enhanced memory unit, with α serving as the hyperparameter that balances adjacent time steps. Our ST-M architecture is asynchronous, processing information frame-by-frame rather than employing time-dimension attention, thereby significantly reducing the computational resources required by the network and facilitating the direct processing of streaming audio information. Additionally, by balancing the memory residual at time t -1 with the input information at time t, our approach facilitates memory interaction between adjacent time steps. This processing paradigm that relies on input from adjacent time steps pays more attention to short-term memory, thereby granting our model improved localization robustness in dynamic environments.To avoid the MAC operations present in α ConvBN (ST [t -1]), we integrate α into the firing threshold of SN , which can be expressed by the following formula:EQUATIONHere, SN ′ denotes a layer of spiking neurons with a threshold of V th /α. Due to ST [t] and X[t] being binary spikes and ConvBN can be fused during inference, Eq. 8 contains no MAC operations which ensure low power consumption in inference. Compared with attention in the temporal dimension, the ST-M structure demonstrates asynchronous inference and low-power computational characteristics. As shown in Fig. 4 (b) and Fig. 4 (d), TA methods require the processing of all temporal information and rely on full-precision attention representation, whereas our ST-M structure utilizes only the spike information from adjacent time steps and features spike-driven computation. This ensures the model can perform inference in a low-power manner.Combining the FSJA and ST-M modules, we propose a spike-driven MAA mechanism, with its insertion location detailed in the Appendix. D. The proposed MAA mechanism leverages the FSJA module to effectively filter noise, and the ST-M module to strengthen the model's memory for wise decisions across timeframes. As a result, our biologically inspired MAA significantly improves the localization accuracy and robustness of our SNN-based back-end network model. 
experiments: 
 In this section, we evaluate our proposed spike-based SSL framework performance on three datasets: the HRTF [57] , Single Words [30] , and SLoClas dataset [42] . Moreover, we examine its energy efficiency and robustness through extensive ablation studies and noise addition experiments on the SLoClas dataset.The HRTF dataset and Single Word are examined utilizing 2-channel audio at a singular frequency, with a minimum angular resolution of 10 • . In contrast, the SLoClas dataset comprises 4-channel audio in real-world scenarios, with a higher resolution of 5 • . Consequently, the SLoClas dataset presents a higher level of challenge and more closely resembles real-world scenarios. We report in detail the Mean Absolute Error (MAE) and the classification accuracy (Acc.) [7] , defined as shows:EQUATIONwhere θi represents the estimated azimuth angle, and θ i denotes the ground truth azimuth angle of the sample i. MAE quantifies the deviation between predicted and true angles, where a lower value means superior performance. Moreover, Acc quantifies the similarity between predicted angles and actual output angles. The η is set differently across datasets to align with their specific characteristics: For the HRTF and Single Word datasets, η is set at 5 • , while for the SLoClas dataset, it is set to 2.5 • . In this manner, η is rounded to the nearest increment corresponding to the minimum localization resolution when calculating classification accuracy. To assess the efficiency of the proposed RF-PLC method and MAA module, we conduct a series of ablation studies. Specifically, we compare our ITD extraction approach in the RF-PLC method with the established FT-based method used in previous work [40] . As depicted in Fig. 5 (a), our method achieves an accuracy nearly identical to that of the conventional approaches, with an error rate of only 1%. Furthermore, prior research has demonstrated that RF neurons exhibit significantly lower energy consumption compared to FT operations, particularly when implemented on neuromorphic hardware [17, 18, 45] . These findings validate the effectiveness of our RF-PLC method in achieving a highly efficient and accurate ITD encoding scheme. where P signal represents the power of the signal and P noise denotes the power of the noise. A lower SNR indicates a higher proportion of noise. Specifically, we incorporate noise from the NOISEX-92 database into audio recordings from different microphone channels. Detailed information about the noise addition process and the experimental setup is described in the Appendix.C.As shown in Fig. 6 (a), we visualize the encoding results of the RF-PLC method under various SNR conditions to better understand the input form of our back-end model. In addition, we also present the distribution of recognition over 360 • . As shown in Fig. 6 (b) and Fig. 6 (c), the MTPC method is more likely to predict certain angles, especially in terms of information in the noise direction; however, our method is not significantly affected by noise information. It indicates that our model exhibits higher stability. As SNR increases, our method demonstrates higher recognition accuracy. This indicates that our model effectively suppresses noise in specific frequency bands, thereby preventing significant variations in recognition results due to increased noise. The results provide strong evidence of the model's superior generalization and robustness when applied to complex real-world scenarios.  
conclusion: 
 Inspired by the efficiency of biological auditory systems, this work proposes a novel spike-based SSL framework. The core components are the RF-PLC method and the MAA module. The RF-PLC method leverages the resonance properties of RF neurons to bypass computationally expensive FT operations. It utilizes a phase-locking loop and ITD detection neurons to efficiently encode ITD cues from the audio signal into spike trains. Furthermore, the study incorporates insights from auditory biology, including frequency preferences and short-term memory characteristics. By designing a fully spike-driven MAA module, our SNN-based SSL model effectively filters irrelevant environmental noise in the frequency domain while temporally focusing on specific auditory content. This approach achieves superior performance, robustness, and interpretability, significantly advancing the field of neuromorphic SSL research. It establishes a new benchmark for the development of SSL techniques. Future work will investigate the deployment of this model on neuromorphic hardware platforms.  
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: Yes, the paper properly credits the creators or original owners of assets (e.g.,code, data, models) and explicitly mentions and respects the relevant licenses and terms of use.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: EQUATION)where N is the number of discrete samples used in the FT. The complex vector F[k] quantifies the spectral components at varying frequencies. Utilizing Lemma 1, these components represent sinusoidal signals decomposed at frequencies indexed by k, scaled by fs N . For each component F[k] = a k + ib k , the corresponding time-domain signal can be described by:EQUATION)Proof: Assume a series of RF neurons, each with a resonant frequency of ω = [-0* 2π N , -1 * 2π N , -2 * 2π N , . . . , -(N -1) * 2π N ],initially set to zero. When exposed to a real-time input audio x, the response of the k th neuron at time t is given by the recursive update:EQUATION)This recursive filtering mimics a discrete Fourier transform when λ = 1. Moreover, RF neurons can be efficiently implemented on neuromorphic hardware like the Loihi2 chip, facilitating low-power and high-speed computations.Inspired by observations of specific mechanisms within the auditory and visual cortices [9, 26] , biological evidence are expressed through phase coding found. It can encode the pure tone audio into precise-timing spike. Regarding the results of the decomposition:EQUATIONy L and y R represent the single-tone sinusoidal signals arriving at the left and right ears. Subsequently, the first peak time is encoded into spike time as the arrival time of the sound.EQUATION)where f represents the frequency of the sinusoidal signal. Thus, ITD can be expressed as t L -t R .Proof: When RF neurons (from both ears) enter the spike stage, their initial states are represented as a k L + ib k L for the left ear and a k R + ib k R for the right ear. Here, L indicates the left ear, R denotes the right ear, and K refers to the RF neuron associated with the intrinsic frequency f k . Subsequently, the state will decay oscillations over time. To facilitate understanding, we calculate the real and imaginary parts in a discrete manner:EQUATIONWith the RF-PLC method we propose, we can directly spike timing acquisition. Specifically, RF neurons will fire spike ata k L [t n L ] = 0, b k L [t n L ] = max(b k L ).In the phase space, this state represented the first peak of time aligns with the neural phase coder:       ϕ locked = tan -1 b k L [t n L ] a k L [t n L ] = π 2 ,Phase-locking LoopEQUATION)wheret n L satisfies ϕ locked = Z RF [t n L ] and t n R satisfies ϕ locked = Z RF [t n R ].Due to the discrete form, our ITD encodings are not identical to those in Lemma 1. The error is in the difference between the audio's sampling rate and the actual frequency. It results in our inability to accurately obtain the first peak time.Specifically, as Lemma 1 demonstrates, to obtain the spike timing for pure tone audio of different frequencies, the phase coding model requires leveraging Eq.15 to compute the audio's ITD cues. However, with the RF-PLC method we propose, it can only iterate according to the audio's sampling rate. The difference between them is dependent on the sampling rate f s , which can be represented as 1/f s . For the dataset SLoClas that we tested, its errors is only approximately 1%.We primarily validated the accuracy and robustness of our proposed method on the SLoClas dataset. It utilizes a 4-channel microphone array to collect data on RWCP sound scenes to ensure SNR of 40 to 50dB. It is comprised of ten distinct categories of ambient sounds: bells, bottles, buzzers, cymbals, horns, metal, particles, phones, rings, and whistles. Each category includes approximately 100 instances, providing a diverse and comprehensive set of audio samples.Additional, to evaluate the robustness of our method, we need to construct source localization information under various complex scenarios. Specifically, we can introduce different types of noise into the audio. The noise sounds are sourced from the NOISEX-92 database, which contains recordings of various types of real-world noise. We add noise audio to each microphone channel to simulate noise coming from different directions.EQUATIONIn this setup, mic i (n), where i = 1, 2, 3, 4, represents the 4-channel signals recorded by the microphone array. ComplexVideo i denotes the multi-channel data with noise. The noise data consists of randomly selected audio clips from a noise database. λ is the scaling factor used to adjust the audio to a specific SNR ratio. A smaller SNR indicates a stronger noise presence in the audio which is more similar to real environment. The overall network architecture of our SNN-based SSL model is illustrated in Fig. 8 , featuring a comprehensive system design tailored for sound localization. The architecture consists of two main components: a front-end RF-PLC method and a back-end MAA-based localization decision network.In the RF-PLC method, we utilize 512 RF neurons with widths ω that increase incrementally from 0 to 8000Hz. These neurons are strategically deployed as an alternative to the traditional FT operation, optimizing the model for energy efficiency and computational speed. Additionally, we utilize a cochlear filter bank following the Constant Q Transform (CQT) which is the most commonly used and easily implemented cochlear filter bank, to extract auditory features of appropriate dimensions. This encoding approach not only mimics the cochlear filtering process but also enhances the temporal dynamics of sound processing. Additionally, N detection neurons are engaged to characterize the ITD with delays ranging from -25τ to 25τ . Based on the audio sample ratio, the value of τ is set at 0.0625 ms. Furthermore, our model utilizes data from four microphones to compute ITDs between each pair, resulting in six distinct sets of ITD cues. As a result, each 170 ms speech frame is encoded into X ∈ Z C×N ×Nτ , capturing a rich array of spatial and temporal information. Details of these parameters can be found in Table .3. This setup ensures a detailed and dynamic spatial representation of auditory scenes.In the back-end decision network, we propose a fully spike-based model, which is illustrated in Fig. 8 . We will validate our module within the SSL models [7, 40] consisting of convolutional and MLP layers. To enhance the performance of the SSL model, we augment this basic structure with our MAA module. This module is designed to emulate biological auditory processing by focusing on frequency band preference and short-term memory capabilities. Compared with traditional spatio-temporal spike attention techniques [65, 71] , our module markedly improves the system's computational efficiency, as shown in Table . 2. Additionally, to better illustrate the performance of our module, we present a comparison with various modules. Details of these networks are provided in Table . 4. This enhancement allows for faster and more accurate sound localization, demonstrating the potential of our model in real-world applications. To describe the energy consumption calculations in the ablation experiments, we introduce a theoretical energy estimation method for the proposed attention mechanism. Compared to the ANN model, the energy consumption calculation of the spiking version requires information on the timesteps (T) and spike firing rates (R). The spike firing rate is defined as the proportion of non-zero elements in the spike tensor. Since our proposed MAA method is spike-driven, we only need to evaluate the model's FLOPs, along with T and R, to estimate the theoretical energy consumption of our methods.In ANN [35] , the FLOPs for the n-th Conv layer are expressed as:EQUATIONwhere k n denotes the kernel size, (h n , w n ) specifies the dimensions of the output feature map, and c n-1 and c n represent the numbers of input and output channels, respectively. The FLOPs of the m-th MLP layer in ANNs are:EQUATIONwhere i m and o m represent the input and output dimensions of the m-th MLP layer.Refer on previous research [24, 68] , we assume that all computational data are implemented using 45nm technology for 32-bit floating-point calculations, with E M AC = 4.6pJ and E AC = 0.9 pJ.In Table 3 , we further present the details of different models. Referring to [66, 71] , the attention matrix is derived using the sigmoid function, which results in the network input for the subsequent layer being non-spiking. Consequently, during the energy consumption calculation, this component is computed using the energy consumption of MAC operations, leading to a significant increase in energy consumption within the network. Due to the unique properties of the MAA attention matrix, our method does not introduce additional floating-point operations.The limitations of this study include the lack of deployment on edge devices. Furthermore, the limited availability of datasets for SSL tasks restricts the validation of our model across a broader array of datasets. Future research will seek to overcome these challenges by amassing a more extensive collection of datasets and implementing our model on edge devices to more effectively ascertain the efficacy of our approach. The experimental results reported herein are reproducible, with detailed descriptions of the model architectures and hyperparameter configurations available in Appendix. Additionally, our code will be made available on subsequent after review.To further validate that the enhancement in our model's performance is indeed due to the effective implementation of band selection and short-term attention mechanisms, rather than an increased parameter count. We designed a Conv2d layer module with the same amount of parameters, ensuring all other parameters remained consistent. Specially, As illustrated in Fig. 9 , LocalMAA refers to a network with the same number of parameters as our proposed MAA. It can be observed that our MAA method achieves a lower MAE( • ) and higher Acc(%). Furthermore, we compared the MAE of different modules, finding that our frequency band preference and short-term memory structure significantly enhance the network's localization performance.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: We mentioned our data in the Appendix. D and code in supplemental material, respectively. Guidelines:• The answer NA means that paper does not include experiments requiring code. Justification: This work is foundational research and not tied to particular applications. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 


Decision: This paper presents a novel Sounds Source Localization (SSL) model that is based on Spiking Neural Networks (SNNs). Specifically, the paper use RF-PLC neurons to encode interaural time differences that are critical to SSL. These neurons avoid the significant computational overhead of calculating Fourier transforms as used in state-of-the-art (SOTA) SSL baselines. Additionally, the authors propose a SNN based multi-auditory attention module which implements biologically inspired properties of frequency selectivity and short-term auditory memory, which is known to improve SSL accuracy and robustness in noisy environments. The paper presents experiments demonstrating the superior accuracy and robustness of the proposed SNN-SSL on SloClas dataset. Analytical energy estimates show that the proposed SSL model has comparable energy to the SOTA SNN baseline. 

The reviewers primarily suggested adding more datasets and more SNN/ANN SSL baselines to compare the proposed model against. The authors have done a great job by adding SSL accuracy data for more datasets and other baselines and all experiments continue to show strong accuracy results for the proposed SSL. Spike-Driven V2 is one baseline that achieves a better accuracy than this paper, but as the authors highlight, it also has ~10x more parameters compared to the proposed model. Consequently, the proposed SSL offers a better tradeoff when it comes to accuracy vs model size. 

The authors showed great flexibility by adding ablation experiments that further strengthened the justification for their model design choices. To conclude, the paper presents a novel SSL model which is biologically motivated and shows high accuracy and robustness for the given model size. I recommend that this paper should be accepted for NeurIPS.

Minor note, the authors should proof-read their submission for small grammatical errors and try to organize the paper to improve its readability. For e.g., Table 2 shows useful ablation study data, but it could use a more detailed description describing all the experiment configs. Also, the authors show comparable energy consumption to the existing SNN-SSL baseline in Table 2, adding a comment on that in the results section of the paper will be helpful.

Edit: Updated my recommendation to poster (prev spotlight) post discussion with SAC.

Reviews: review: paper_summary: paper_summary: The paper presents a novel neuromorphic model for sound source localization (SSL) inspired by biological auditory systems. The model integrates spike-based neural encoding and computation, employing Resonate-and-Fire (RF) neurons with a phase-locking coding (RF-PLC) method. The RF-PLC method leverages the resonance properties of RF neurons to efficiently convert audio signals to time-frequency representations and encode interaural time difference (ITD) cues into discriminative spike patterns. Additionally, the model incorporates a spike-driven multi-auditory attention (MAA) module inspired by biological adaptations, which enhances SSL capability in noisy environments. The authors demonstrate that their model achieves state-of-the-art accuracy and robustness in real-world conditions.
strength_weakness: strength_weakness: Comparative Analysis: While the paper claims state-of-the-art performance, the comparative analysis with existing models is not comprehensive. The authors should provide more detailed comparisons and discuss why their model outperforms others. Complexity: The model's complexity and computational requirements are not adequately addressed. It's important to understand the trade-offs between performance gains and computational costs. Generalization: The experiments are promising but limited in scope. Additional tests across various datasets and real-world scenarios are needed to validate the generalizability of the model. Biological Plausibility: The biological plausibility of some components, such as the MAA module, is not thoroughly discussed. More insights into how closely these components mimic biological processes would strengthen the paper.
questions: questions: The introduction provides a clear motivation and context for the research. However, it could benefit from a more detailed discussion of existing SSL models and their limitations. The methodology is well-described, but some parts are overly technical and could be simplified for clarity. Diagrams illustrating the RF-PLC method and MAA module would be helpful.The results section presents compelling evidence of the model's performance. Including statistical significance tests and error bars in the figures would add rigor to the findings. The discussion touches on the implications of the results but lacks depth in exploring future work and potential applications. More emphasis on the practical applications of the model would be beneficial.
limitations: limitations: See questions.
paper_summary: paper_summary: This work proposes a SNN-based model for SSL. To achieve efficient processing of raw
 speech signals, they introduce a phase-locking coding (RF-PLC) method using Resonate-and-Fire (RF)
 neurons and detection neurons.
strength_weakness: strength_weakness: 1. The improvements of performance compared with current SOTA methods are quite limited. And the paper did not compred with Spike-driven V2 and other SOTA methods. Hence the contribution on model performance improvements  is unclear.

2. The novelty of proposed attention mechanisms  is limited compared to existing  attention methods for SNN.

3. The experiments results should be conducted multiple trials. The source code could be provided to show the reproducability.
questions: questions: 1. The improvements of performance compared with current SOTA methods are quite limited. And the paper did not compred with Spike-driven V2 and other SOTA methods. Hence the contribution on model performance improvements  is unclear.

2. The novelty of proposed attention mechanisms  is limited compared to existing  attention methods for SNN.

3. The experiments results should be conducted multiple trials. All source code could be provided to show the reproducability.
limitations: limitations: See mentioned above.
paper_summary: paper_summary: This paper introduces a spike-based neuromorphic model designed for sound source localization (SSL), capitalizing on the inherent properties of Resonate-and-Fire (RF) neurons. By encoding sound via phase-locking to leverage the resonance characteristics of these neurons, the model efficiently represents interaural time differences (ITD), facilitating precise sound localization. The inclusion of a multi-auditory attention (MAA) module, inspired by biological auditory systems, enhances the model's performance in noisy conditions by focusing on relevant sound frequencies and temporal sequences. Extensive experimental results demonstrate the model's state-of-the-art accuracy and robustness against noise, presenting an advance in neuromorphic SSL applications.
strength_weakness: strength_weakness: __Limited Dataset__: The experiments are conducted on a single dataset, which does not validate the universality of the proposed solution.

__Incremental Costs in Module Stacking__: The design seems incremental by stacking modules that introduce additional costs to improve performance. Specifically, the introduction of additional Parameters and Power in the ablation studies (Table 2) appears to contradict the paper's motivation to reduce computing complexity and energy consumption. The trade-off between these aspects may need clarification.

__Novelty__: The application of SNNs for SSL seems to have been first introduced in reference [38]. Thus, the contributions of this paper are optimizations based on [38], and the novelty needs further clarification.
questions: questions: __Q1: Training Time Cost__. I am curious about the time cost during the training phase. SNNs are known for their training overhead, even though they theoretically have lower power consumption during inference. It might be beneficial for the paper to discuss the additional overhead introduced by the three modules during training, perhaps including this as a metric in the ablation experiments.

__Q2: Suitability of Neuromorphic Systems for SSL Tasks__. Why are neuromorphic systems more suitable for SSL tasks? From Table 1, it appears that the proposed SNN-based solution significantly outperforms ANN-based implementations, which is contrary to the common understanding that ANNs generally lead in performance compared to SNNs. I would appreciate insights from the authors on this discrepancy.
limitations: limitations: See Weaknesses and Questions.
paper_summary: paper_summary: This paper introduces a spike-based neuromorphic model for sound source localization. It utilizes the RF-PLC methods for auditory-like spectral analysis and encoding. Additionally, it is supported by the MAA module, which simulates attention mechanisms in specific biological frequency bands. These technologies are integrated to enhance the efficiency and accuracy of SSL Tasks.
strength_weakness: strength_weakness: 1. Regarding the design of the loss function: In practical settings, there should be no significant difference between the model outputs at 355° and 5°. It is worth designing the loss function to avoid this issue effectively. I hope the authors can provide a detailed discussion of the loss function.
2. Concerning robustness validation of localization: To verify the model's accuracy under noisy and complex conditions, the paper involves adding noise to pure audio before classification. I hope the authors can provide a detailed discussion on how the noise dataset was constructed and why it was structured in this way.
3. Validity of experimental methods: The paper appears to be tested only on the SLoClas dataset, which may not sufficiently demonstrate the effectiveness of the proposed method. It is hoped that the authors will test the method on more datasets. If no other public datasets are available, did the authors use any other methods to validate the effectiveness of the proposed approach?
questions: questions: Please refer to the weaknesses section.
limitations: limitations: Please refer to the weaknesses section.
paper_summary: paper_summary: This paper constructs a sound source localization model by leveraging efficient spiking neural networks and biologically-inspired auditory localization mechanisms. Although numerous studies have previously approached this subject from a biomimetic perspective, this paper commendably considers the balance between biomimicry and practicality, while also addressing the robustness of the SSL model. It offers a new perspective on the effective integration of artificial neural networks and biological intelligence.
strength_weakness: strength_weakness: 1.It remains insufficient for readers without a background in SSL tasks or RF technology. Although the supplementary materials are helpful, they seem not to adequately bridge the knowledge gap for those outside the specialty.
2.I understand that the outputs of the RF-PLC method will serve as inputs for the classification model. It would be beneficial if the authors could provide a step-by-step example that shows how a given audio input is encoded into a spike pattern, and how these encoded outputs are subsequently utilized to determine the azimuth of a sound source relative to the listener.
questions: questions: 1.The authors are asked to explain how to determine the azimuth of a sound source, including an example of encoding an audio segment and predicting the azimuth.
2.The effectiveness of the MAA Block needs verification. While Table 2 shows improved localization performance without added computational demand, further experiments should confirm these results are not due to other factors like increased parameter counts.
3.Although this work focuses on the algorithm of SSL and I agree that the workload is sufficient, I wonder whether the RF neuron has the potential to be implemented on current neuromorphic hardware or FPGA.
limitations: limitations: None
paper_summary: paper_summary: This study draws inspiration from the intrinsic mechanisms of sound source localization in biological auditory systems to design an efficient and robust SSL model. The core contributions include two primary aspects: firstly, replacing the energy-intensive Fourier Transform(FT) operations with RF neurons in the auditory encoding process, thereby encoding Interaural Time Differences (ITD) as spatiotemporal spike trains. Secondly, the backend SSL model effectively implements frequency band preferences and memory-based decision-making processes observed in biological hearing.
strength_weakness: strength_weakness: While the paper is generally convincing, I believe the following areas require attention:
1. The claim that RF neurons can effectively replace high-energy Fourier Transform operations needs further experimental validation or existing research evidence. I suggest the authors bolster this section with reliable proof.
2. While there are currently no datasets to test localization robustness, the experiments on robustness lack detailed descriptions of how noise was introduced in the SLoClas dataset. 
3. Despite the high cost of deploying the proposed sound localization model on hardware, I recommend that the authors attempt actual deployment on neuromorphic agents in future work to empirically verify model performance.
questions: questions: 1. The short-term memory structure proposed seems to only consider relationships between adjacent frames. Could methods involving multi-frame integrated decision-making be explored?
2. The authors point that RF neurons can effectively perform the basilar membrane's frequency band decomposition in the cochlea. Is there a biological precedent for similar RF neurons in the cochlea?
limitations: limitations: While the proposed neuromorphic SSL model demonstrates promising results, several limitations should be acknowledged:

1. **Lack of Edge Device Deployment:** The current study does not include the deployment of the proposed SSL model on edge devices. Practical implementation and testing on such devices are crucial to validate the model's real-world applicability and performance.

2. **Experimental Validation of RF Neurons:** The claim that RF neurons can replace energy-intensive Fourier Transform operations lacks sufficient experimental validation. Further empirical evidence or references to existing research are needed to substantiate this claim convincingly.


By addressing these limitations, future research can provide a more comprehensive evaluation of the neuromorphic SSL model and its potential impact on sound source localization and broader applications in neuromorphic computing.

Comments: title: Seeking Comments on Updated Review Scores
Comments
Thank you for your valuable suggestions and positive feedback.
Thank you for your  valuable suggestions and positve feedback.
Thank you for your positive feedback and insightful suggestions.
Good rebuttal
Request for Feedback on Response Clarifications
comment: Dear Reviewer,

We would like to confirm whether our recent response has adequately addressed your concerns. Regarding the issue of convincing results, all the relevant codes and data are provided in the supplementary materials. May we kindly request that you review these materials at your earliest convenience? **We appreciate your reconsideration and hope for a fair evaluation**. 

Thank you for your time and attention.
Dear Reviewer, 

We sincerely appreciate your response. In terms of the convincing results problem pointed by the Reviewer 2syp in Q3,  we have included all relevant codes and data in the supplementary materials alongside our original manuscript. In our work, all the experiments in our work are conducted at least five times. We apologize for any misunderstanding this may have caused and kindly request that you review the supplementary materials to verify the performance of our methods. 

Moreover, extensive comparative experiments have demonstrated that our model not only achieves SOTA accuracy among similarly sized models but also reduces MAE by approximately 50%. Especially, compared to current SOTA models, such as Spike-driven V2, our model is significantly smaller, with its size being only 1/10 of theirs.

We hope our responses have addressed your concerns. We look forward to your further comments and evaluation.
Thank you for your thoughtful review and valuable suggestions regarding our manuscript. We have carefully incorporated your recommendations and conducted extensive experiments to address your concerns.

We observed that the score for our manuscript decreased from 4 to 3. We are puzzled by this change and would like to understand the reasons. Could this reduction be a misunderstanding, or are there new concerns arising from our rebuttal? We would greatly appreciate any additional feedback that could help us improve our manuscript further.

Thank you very much for considering our request.
Thanks for rebuttal, but the results seem not convinced just as reviewer 2syp mentioned, so I will lower my score.
We are pleased to know that we have addressed your concerns regarding this manuscript. Thank you again for your valuable suggestions to improve the quality of our work.
Thank you for your response and recognition. We are pleased to know that most of your concerns have been addressed. Your comments are crucial in enhancing the quality of our work.
Thank you for your response and recognition. We're pleased that your concerns have been addressed. Your feedback is invaluable in helping us improve the quality of our work.
Thank you for acknowledging our work. As you per, traditional ITD encoding often relies on FT operations [1], which are difficult to deploy on neuromorphic hardware, resulting in a divided implementation. In contrast, our ITD encoding scheme employs RF neurons, which have been successfully deployed on Loihi 2 [2] with excellent performance. This demonstrates that our encoding method does not face technical barriers when deployed on neuromorphic chips. Additionally, our backend localization model is strictly spike-driven, with extensive research confirming that such models can be effectively implemented on neuromorphic hardware [3, 4]. Thus, our SSL model can be deployed as a complete system on neuromorphic hardware.

[1] A hybrid neural coding approach for pattern recognition with spiking neural networks. In: TPAMI (2024).

[2] Efficient Video and Audio Processing with Loihi 2. In: ICASSP (2024).

[3] 22.6 anp-i: A 28nm 1.5 pj/sop asynchronous spiking neural network processor enabling sub-o. 1 μj/sample on-chip learning for edge-ai applications. In: ISSCC (2023).

[4] Towards artificial general intelligence with hybrid Tianjic chip architecture. In: Nature (2019).
The experimental examples provided have effectively addressed my questions, and the ablation studies on the MAA block also confirm the model's effectiveness. However, I am particularly interested in how the model performs when deployed on neuromorphic chips or FPGAs. From the authors' responses, it appears that the model's inference relies mainly on two parts, which seems to contradict the original intent of designing a system. From a hardware deployment perspective, it is crucial to determine whether the model can function as a complete system for inference, without considering resource scarcity and procurement difficulties.

Overall, most of the concerns have been addressed. Thank you for your response. I would like to increase my score to 8.
The author answered my confusion and therefore raised the score.
Thank you for the response, which has addressed my concerns very well, so I will increase the score.
Dear Reviewers,

Could you please provide feedback on our responses to your questions? We are eager to know if our answers have addressed your concerns. Additionally, are there any further comments or insights you might have regarding our work?

Your input is crucial for us to enhance the quality of our submission, and we greatly appreciate your guidance and time.