Sound Source Detection from Raw Waveforms with Multi-Scale Synperiodic Filterbanks

1 INTRODUCTION . The fundamental task for an agent to perceive and interact with the 3D environment is to know the location and semantic identity of its nearby objects . The location includes spatial location that is either stationary or moving , temporal location like start time and end time . Vision-based such environment perception has received large attention in the past decade and we have witnessed huge progress in tasks such as object detection ( Liu et al. , 2016 ; Lin et al. , 2014 ; Yang et al. , 2019 ) , classification ( He et al. , 2016a ) and tracking ( Wang et al. , 2019 ) . Nevertheless , the sound-based counterpart research has far lagged behind , despite all the fascinating properties sound signal exhibits . For example , sound is ubiquitous and insensitive to ambient illumination change , it has no field-of-view ( FoV ) constraints and is capable of circumventing physical barriers to perceive scene beyond line-of-sight . As a complementary sensing approach to vision , sound-based environment perception is of vital importance for acoustic scene understanding . A typical example is the sound source detection ( answer where is it , when does it happen and what is it ) , given the recorded multi-channel sound waveforms . To detect sound sources , we often deploy a spatially-configured microphone array to record an acoustic environment . Unlike camera or LiDAR scanner that directly captures RGB image or measures the range distance , sound waveform itself is a highly compressed one-dimensional points with high sampling rate , all sound source signals are compressed and mixed into the one-dimensional data format . Since different sound sources have different frequency property , it is essential convert waveform into time-frequency representation so that frequencies hidden in the waveform are explicitly split out . This is often achieved by projecting the raw waveform into various orthogonal frequency basis . At the same time , a sound source ’ s spatial location clue lies in inter-channel difference of recorded waveforms ( i.e . phase difference in this work ) . It is essential to design a neural network that jointly encodes mono-channel time-frequency representation and inter-channel phase difference from the raw waveforms in a unified , parameter-frugal and computation-efficient manner . The learned representation should have elegant resolution in both time , frequency and space domain so that sound sources can be precisely detected . However , learning such representation is a tough task . Challenges derive from both theoretical side and practical side . According to Uncertainty Principle , we can not achieve the optimal resolution in time and frequency domain at the same time , but instead keep a trade-off between them . Traditional hand-engineered sound feature ( Davis & Mermelstein , 1980 ; Cao et al. , 2021 ; Tho Nguyen et al. , 2020 ; Brandstein & Silverman , 1997 ) and some recently proposed learnable filter bank ( Ravanelli & Bengio , 2018 ; Zeghidour et al. , 2018 ) empirically set the same length for all filters , resulting in human-biased , unadjustable time-frequency resolution map . Some other work ( Zeghidour et al. , 2021 ) correlates filter frequency response and filter length by initializing in mel-scale , but it is not scalable nor stable because the final initialization depends on the filter number . Moreover , all existing methods process raw waveform with one-scale filter bank , we argue that the one-scale sound perception easily leads to incomprehensive sound sources sensing , especially when sound sources have different frequency property or undergo various spatial motion . In this paper , we first give comprehensive theoretical analysis on the filter bank impact on its extracted feature ’ s time and frequency resolution . Based on the analysis , we propose a simple yet effective synperiodic filter bank construction strategy in which each filter ’ s frequency response and length are correlated by rotating periodicity such that each filter ’ s length is inversely proportional to its frequency resolution . The synperiodic filter bank thus internally maintains a better time-frequency resolution trade-off than traditional fixed-length filter bank . Coupling the filter length with its frequency response helps us to reduce human intervention in filter bank design . By simply alternating the periodicity term , we further construct a group of synperiodic filter banks , with which we achieve multi-scale perception in time domain . At the same time , by applying a synperiodic filter bank to process one raw waveform as well as its consecutively-downsampled versions , we achieve multi-scale perception in frequency domain . The multi-scale perception in both time and frequency domain of synperiodic filter bank enables the neural network to dynamically learn better representation for sound source detection in a data-driven way . It is worth noting that synperiodic filter bank parameter number is just linear to filter number ( adds up to less than 1 % of the whole parameters ) and it can be efficiently implemented as a 1D convolution operator . Following the aforementioned learnable front-end , we add a Transformer-like backbone network with two paralleling branches with intermediate soft-parameter sharing to learn sound source ’ s semantic and spatial location related representation both jointly and separately . Experiment on both direction-of-arrival ( DoA ) task and physical location estimation task shows that our proposed framework outperforms comparing methods by a large margin . Replacing existing method ’ s head with our proposed synperiodic filter bank also improvdes the performance . 2 RELATED WORK . Sound signal processing has been thoroughly studied in traditional digital signal processing area . The preliminary step of sound signal processing is usually to convert raw waveform into 2D timefrequency representation . There are two main realms : Fourier transform based and Wavelet based transform ( Sturm , 2007 ) . Traditional sound feature design are motivated influenced by human-auditory system . For example , they often convert frequency bins into mel-scale to imitate human hearing system , like MFCC ( Davis & Mermelstein , 1980 ) , LogMel ( Cao et al. , 2021 ; Grondin et al. , 2019 ) , the filter length is empirically chosen and often a windowing is added to avoid spectrum leakage . For inter-channel phase difference encoding , it is often recommended to encode in frequency domain due to the less-computation advantage . Typical phase difference features include GCC-Phat ( Brandstein & Silverman , 1997 ) and intensity vector ( Cao et al. , 2021 ) . Sound source detection has been previously treated as sound structure ( Thrun , 2006 ) estimation , sound object detection ( He et al. , 2021 ) and sound event detection and localization ( SELD ) problem ( Adavanne et al. , 2018 ; Thi Ngoc et al. , 2021 ) . It involves jointly identifying a sound source ’ s semantic label and predicting its spatial location , the two sub-tasks have been thoroughly studied separately in acoustics ( Nandwana & Hasan , 2016 ; Mohan et al. , 2008 ; Sundar et al. , 2020 ; Vera-Diaz et al. , 2018 ) and computer vision community ( He et al. , 2016a ) . T. Kim et al . ( Kim et al. , 2019 ) provided a review and discussion for raw-audio based event classification . Benefiting from the success of the traditional hand-engineered sound feature and the large availability of mature image-based deep neural networks ( He et al. , 2016a ) , most work ( Grondin et al. , 2019 ; Cao et al. , 2021 ; Grondin et al. , 2019 ; Adavanne et al. , 2018 ; Thi Ngoc et al. , 2021 ) tackle the task by first extracting hand-engineered sound feature and then feeding them to mature image-based neural networks . This workflow is straightforward and often guarantees reasonably good results but it is not end-to-end trainable and heavily depend on image side which may not be optimal for sound processing . At the same time , some work ( He et al. , 2021 ; Adavanne et al. , 2018 ; Tho Nguyen et al. , 2020 ) have simplified the problem by assuming no two sound sources of the same semantic label but different spatial location happen at the same time . This assumption avoids semantic label and spatial location association issue but may not reflect real scenarios . In recent years , a bunch of work tried to design neural work to directly learn from raw sound waveform , ranging from the earlier methods that directly apply stacked layers to process raw waveform ( Schneider et al. , 2019 ; Palaz et al. , 2013 ; Jaitly & Hinton , 2011 ; Sainath et al. , 2013 ) to the recent frequencysensitive filter bank learning methods ( Zeghidour et al. , 2021 ; Ravanelli & Bengio , 2018 ; Zeghidour et al. , 2018 ; He et al. , 2021 ; Hoshen et al. , 2015 ; Sainath et al. , 2015 ; Luo & Mesgarani , 2019 ) . The filter bank parameter is initialized in either mel-scale ( Ravanelli & Bengio , 2018 ; Zeghidour et al. , 2021 ; He et al. , 2021 ; Zeghidour et al. , 2018 ) or as Gammatone filter ( Hoshen et al. , 2015 ; Sainath et al. , 2015 ) . SoundDet ( He et al. , 2021 ) is the first work to directly learn from multi-channel raw waveforms to detect sound sources . It designs MaxCorr filter bank to directly convolve with multi-channel raw waveforms to learn phase difference aware features . Multi-scale representation has a rich history in computer vision community ( ( Liu et al. , 2016 ; 2021 ; Lin et al. , 2016 ; He et al. , 2016b ) ) , in which multi-scale representation strategy has been proposed to accommodate large object scale variation . For example , SSD ( Liu et al. , 2016 ) combines feature maps of various scale to cover potential objects of various sizes . In addition to spatial multi-scale representation , feature space multi-scale representation has been explored as well . For example , He et . al . ( He et al. , 2016b ) incorporate features arising from intermediate layers to boost attribute recognition . In sound signal processing , Stéphane Mallat ( Bruna & Mallat , 2013 ; Mallat , 2012 ) proposed wavelet scattering to obtain multi-scale sound representation by iteratively treating the proceeding processed sound waveform as new virtual waveform for further process . 3 SOUND SOURCE DETECTION FROM MULTI-CHANNEL RAW WAVEFORMS . 3.1 SOUND SOURCE DETECTION PROBLEM DEFINITION . We use a spatially-deployed microphone array , like four closely-bounded microphones in a circular coplanar configuration , to record an acoustic environment where various sound sources undergo independent and unconstrained spatial motion . Recorded multi-channel raw waveforms are represented as W “ txiptqu4i “ 1 , each waveform is 1-d vector of size T sampled at a fixed sampling rate . We denote K sound sources by S “ tsk “ ptks , tke , spk , slkquKk “ 1 , one single sound source corresponds to a start time ts , end time te , stationary or moving spatial motion sp and a semantic identity label sl . Our goal is to learn a representation P that is representative enough to detect sound sources both spatially and semantically . P is a temporal framewise representation of shape T1 ˆN . We propose to learn P directly from the raw waveforms via a deep neural network F parameterized by θ , PT1ˆN “ FpWTˆ4|θq , T1 ! T ( 1 ) A learned elegant representation should have large resolution in both time , frequency and space domain and exhibit strong capability at addressing three intrinsic challenges : Polyphonicity , sound sources are mutually independent , they can freely overlap temporally ; Time-scale variance , the temporal duration of different sound sources varies enormously , from seconds to minutes long ; Datasize , sound waveform is sampled with large sampling rate , resulting in large data size . Sound source presence clue lies in mono-channel time-frequency representation and its spatial motion clue lies in inter-channel interaction , like phase difference . Our proposed synperiodic filter bank group jointly encodes time-frequency representation and phase difference from raw waveforms in a unified way . More detail about phase difference encoding is given in the Appendix A.1 .

Summary: This submission addresses the sound source detection problem. Compared to previous method, the authors propose a novel synperiodic filerbank compared to syndistance bank, which learns a data-dependent time-frequency resolution map. Experiments show that the proposed method achieves state-of-the-art performance on several datasets.