Sound Source Detection from Raw Waveforms with Multi-Scale Synperiodic Filterbanks

1 INTRODUCTION . The fundamental task for an agent to perceive and interact with the 3D environment is to know the location and semantic identity of its nearby objects . The location includes spatial location that is either stationary or moving , temporal location like start time and end time . Vision-based such environment perception has received large attention in the past decade and we have witnessed huge progress in tasks such as object detection ( Liu et al. , 2016 ; Lin et al. , 2014 ; Yang et al. , 2019 ) , classification ( He et al. , 2016a ) and tracking ( Wang et al. , 2019 ) . Nevertheless , the sound-based counterpart research has far lagged behind , despite all the fascinating properties sound signal exhibits . For example , sound is ubiquitous and insensitive to ambient illumination change , it has no field-of-view ( FoV ) constraints and is capable of circumventing physical barriers to perceive scene beyond line-of-sight . As a complementary sensing approach to vision , sound-based environment perception is of vital importance for acoustic scene understanding . A typical example is the sound source detection ( answer where is it , when does it happen and what is it ) , given the recorded multi-channel sound waveforms . To detect sound sources , we often deploy a spatially-configured microphone array to record an acoustic environment . Unlike camera or LiDAR scanner that directly captures RGB image or measures the range distance , sound waveform itself is a highly compressed one-dimensional points with high sampling rate , all sound source signals are compressed and mixed into the one-dimensional data format . Since different sound sources have different frequency property , it is essential convert waveform into time-frequency representation so that frequencies hidden in the waveform are explicitly split out . This is often achieved by projecting the raw waveform into various orthogonal frequency basis . At the same time , a sound source ’ s spatial location clue lies in inter-channel difference of recorded waveforms ( i.e . phase difference in this work ) . It is essential to design a neural network that jointly encodes mono-channel time-frequency representation and inter-channel phase difference from the raw waveforms in a unified , parameter-frugal and computation-efficient manner . The learned representation should have elegant resolution in both time , frequency and space domain so that sound sources can be precisely detected . However , learning such representation is a tough task . Challenges derive from both theoretical side and practical side . According to Uncertainty Principle , we can not achieve the optimal resolution in time and frequency domain at the same time , but instead keep a trade-off between them . Traditional hand-engineered sound feature ( Davis & Mermelstein , 1980 ; Cao et al. , 2021 ; Tho Nguyen et al. , 2020 ; Brandstein & Silverman , 1997 ) and some recently proposed learnable filter bank ( Ravanelli & Bengio , 2018 ; Zeghidour et al. , 2018 ) empirically set the same length for all filters , resulting in human-biased , unadjustable time-frequency resolution map . Some other work ( Zeghidour et al. , 2021 ) correlates filter frequency response and filter length by initializing in mel-scale , but it is not scalable nor stable because the final initialization depends on the filter number . Moreover , all existing methods process raw waveform with one-scale filter bank , we argue that the one-scale sound perception easily leads to incomprehensive sound sources sensing , especially when sound sources have different frequency property or undergo various spatial motion . In this paper , we first give comprehensive theoretical analysis on the filter bank impact on its extracted feature ’ s time and frequency resolution . Based on the analysis , we propose a simple yet effective synperiodic filter bank construction strategy in which each filter ’ s frequency response and length are correlated by rotating periodicity such that each filter ’ s length is inversely proportional to its frequency resolution . The synperiodic filter bank thus internally maintains a better time-frequency resolution trade-off than traditional fixed-length filter bank . Coupling the filter length with its frequency response helps us to reduce human intervention in filter bank design . By simply alternating the periodicity term , we further construct a group of synperiodic filter banks , with which we achieve multi-scale perception in time domain . At the same time , by applying a synperiodic filter bank to process one raw waveform as well as its consecutively-downsampled versions , we achieve multi-scale perception in frequency domain . The multi-scale perception in both time and frequency domain of synperiodic filter bank enables the neural network to dynamically learn better representation for sound source detection in a data-driven way . It is worth noting that synperiodic filter bank parameter number is just linear to filter number ( adds up to less than 1 % of the whole parameters ) and it can be efficiently implemented as a 1D convolution operator . Following the aforementioned learnable front-end , we add a Transformer-like backbone network with two paralleling branches with intermediate soft-parameter sharing to learn sound source ’ s semantic and spatial location related representation both jointly and separately . Experiment on both direction-of-arrival ( DoA ) task and physical location estimation task shows that our proposed framework outperforms comparing methods by a large margin . Replacing existing method ’ s head with our proposed synperiodic filter bank also improvdes the performance . 2 RELATED WORK . Sound signal processing has been thoroughly studied in traditional digital signal processing area . The preliminary step of sound signal processing is usually to convert raw waveform into 2D timefrequency representation . There are two main realms : Fourier transform based and Wavelet based transform ( Sturm , 2007 ) . Traditional sound feature design are motivated influenced by human-auditory system . For example , they often convert frequency bins into mel-scale to imitate human hearing system , like MFCC ( Davis & Mermelstein , 1980 ) , LogMel ( Cao et al. , 2021 ; Grondin et al. , 2019 ) , the filter length is empirically chosen and often a windowing is added to avoid spectrum leakage . For inter-channel phase difference encoding , it is often recommended to encode in frequency domain due to the less-computation advantage . Typical phase difference features include GCC-Phat ( Brandstein & Silverman , 1997 ) and intensity vector ( Cao et al. , 2021 ) . Sound source detection has been previously treated as sound structure ( Thrun , 2006 ) estimation , sound object detection ( He et al. , 2021 ) and sound event detection and localization ( SELD ) problem ( Adavanne et al. , 2018 ; Thi Ngoc et al. , 2021 ) . It involves jointly identifying a sound source ’ s semantic label and predicting its spatial location , the two sub-tasks have been thoroughly studied separately in acoustics ( Nandwana & Hasan , 2016 ; Mohan et al. , 2008 ; Sundar et al. , 2020 ; Vera-Diaz et al. , 2018 ) and computer vision community ( He et al. , 2016a ) . T. Kim et al . ( Kim et al. , 2019 ) provided a review and discussion for raw-audio based event classification . Benefiting from the success of the traditional hand-engineered sound feature and the large availability of mature image-based deep neural networks ( He et al. , 2016a ) , most work ( Grondin et al. , 2019 ; Cao et al. , 2021 ; Grondin et al. , 2019 ; Adavanne et al. , 2018 ; Thi Ngoc et al. , 2021 ) tackle the task by first extracting hand-engineered sound feature and then feeding them to mature image-based neural networks . This workflow is straightforward and often guarantees reasonably good results but it is not end-to-end trainable and heavily depend on image side which may not be optimal for sound processing . At the same time , some work ( He et al. , 2021 ; Adavanne et al. , 2018 ; Tho Nguyen et al. , 2020 ) have simplified the problem by assuming no two sound sources of the same semantic label but different spatial location happen at the same time . This assumption avoids semantic label and spatial location association issue but may not reflect real scenarios . In recent years , a bunch of work tried to design neural work to directly learn from raw sound waveform , ranging from the earlier methods that directly apply stacked layers to process raw waveform ( Schneider et al. , 2019 ; Palaz et al. , 2013 ; Jaitly & Hinton , 2011 ; Sainath et al. , 2013 ) to the recent frequencysensitive filter bank learning methods ( Zeghidour et al. , 2021 ; Ravanelli & Bengio , 2018 ; Zeghidour et al. , 2018 ; He et al. , 2021 ; Hoshen et al. , 2015 ; Sainath et al. , 2015 ; Luo & Mesgarani , 2019 ) . The filter bank parameter is initialized in either mel-scale ( Ravanelli & Bengio , 2018 ; Zeghidour et al. , 2021 ; He et al. , 2021 ; Zeghidour et al. , 2018 ) or as Gammatone filter ( Hoshen et al. , 2015 ; Sainath et al. , 2015 ) . SoundDet ( He et al. , 2021 ) is the first work to directly learn from multi-channel raw waveforms to detect sound sources . It designs MaxCorr filter bank to directly convolve with multi-channel raw waveforms to learn phase difference aware features . Multi-scale representation has a rich history in computer vision community ( ( Liu et al. , 2016 ; 2021 ; Lin et al. , 2016 ; He et al. , 2016b ) ) , in which multi-scale representation strategy has been proposed to accommodate large object scale variation . For example , SSD ( Liu et al. , 2016 ) combines feature maps of various scale to cover potential objects of various sizes . In addition to spatial multi-scale representation , feature space multi-scale representation has been explored as well . For example , He et . al . ( He et al. , 2016b ) incorporate features arising from intermediate layers to boost attribute recognition . In sound signal processing , Stéphane Mallat ( Bruna & Mallat , 2013 ; Mallat , 2012 ) proposed wavelet scattering to obtain multi-scale sound representation by iteratively treating the proceeding processed sound waveform as new virtual waveform for further process . 3 SOUND SOURCE DETECTION FROM MULTI-CHANNEL RAW WAVEFORMS . 3.1 SOUND SOURCE DETECTION PROBLEM DEFINITION . We use a spatially-deployed microphone array , like four closely-bounded microphones in a circular coplanar configuration , to record an acoustic environment where various sound sources undergo independent and unconstrained spatial motion . Recorded multi-channel raw waveforms are represented as W “ txiptqu4i “ 1 , each waveform is 1-d vector of size T sampled at a fixed sampling rate . We denote K sound sources by S “ tsk “ ptks , tke , spk , slkquKk “ 1 , one single sound source corresponds to a start time ts , end time te , stationary or moving spatial motion sp and a semantic identity label sl . Our goal is to learn a representation P that is representative enough to detect sound sources both spatially and semantically . P is a temporal framewise representation of shape T1 ˆN . We propose to learn P directly from the raw waveforms via a deep neural network F parameterized by θ , PT1ˆN “ FpWTˆ4|θq , T1 ! T ( 1 ) A learned elegant representation should have large resolution in both time , frequency and space domain and exhibit strong capability at addressing three intrinsic challenges : Polyphonicity , sound sources are mutually independent , they can freely overlap temporally ; Time-scale variance , the temporal duration of different sound sources varies enormously , from seconds to minutes long ; Datasize , sound waveform is sampled with large sampling rate , resulting in large data size . Sound source presence clue lies in mono-channel time-frequency representation and its spatial motion clue lies in inter-channel interaction , like phase difference . Our proposed synperiodic filter bank group jointly encodes time-frequency representation and phase difference from raw waveforms in a unified way . More detail about phase difference encoding is given in the Appendix A.1 .

Summary: This paper addresses the problem of sound event detection and localization from multi-channel raw audio waveforms. Essentially, a different audio front end feature extraction scheme - "Synperiodic Filterbank" if proposed which can be parameterized and jointly learnt along with a backbone classifier in an end-to end manner. The feature extraction scheme proposed is grounded in principle of uncertainty governing time-frequency resolution. Through ablation studies the authors show that this feature extraction scheme performs better (albeit marginally) compared to some of the traditionally used feature extraction schemes like MFCC, and LFBEs. Subsequently, the authors show that combining the proposed "Synperiodic Filterbank" with a significantly large model (~3X times the closest baseline model in terms of No. of trainable parameters) based on the Transformer Architecture offers a dramatic improvement in the results compared to the baseline.