Sound Source Detection from Raw Waveforms with Multi-Scale Synperiodic Filterbanks
Authors: Yuhang He
Venue: ICLR 2022 Submitted (2022)
Keywords: speech processing
object detection
deep neural network
sound object
detection and localization
filter bank design

Abstract: Accurately estimating sound sources' temporal location, spatial location and semantic identity label from multi-channel sound raw waveforms is crucial for an agent to understand the 3D environment acoustically. Multiple sounds form a complex waveform mixture in time, frequency and space, so accurately detecting them requires a representation that can achieve high resolutions across all these dimensions. Existing methods fail to do so because they either extract hand-engineered features\,(i.e. STFT, LogMel) that require a great deal of parameter tuning work (i.e. filter length, window size), or propose to learn a single filter bank to process sound waveforms in a single-scale that often leads to a limited time-frequency resolution capability. In this paper, we tackle this issue by proposing to learn a group of parameterized synperiodic filter banks. Each synperiodic filter's length and frequency response are inversely related, hence is capable of maintaining a better time-frequency resolution trade-off. By alternating the periodicity term, we can easily obtain a group of synperiodic filter banks, where each bank differs in its temporal length. Convolution of the proposed filterbanks with the raw waveform helps to achieve multi-scale perception in the time domain. Moreover, applying synperiodic filter bank to recursively process a downsampled waveform enables us to also achieve multi-scale perception in the frequency domain. Benefiting from the advantage of the multi-scale perception in both time and frequency domain, our proposed synperiodic filter bank groups learn a data-dependent time-frequency resolution map. Following the learnable synperiodic filter bank group front-end, we add a Transformer-like backbone with two parallel soft-stitched branches to learn semantic identity label and spatial location representation semi-independently. Experiments on both direction of arrival estimation task and the physical location estimation task shows our framework outperforms existing methods by a large margin. Replacing existing methods' front-end with synperiodic filter bank also helps to improve the performance.

Under review as a conference paper at ICLR 2022
SOUND SOURCE DETECTION FROM RAW WAVEFORMS WITH MULTI-SCALE SYNPERIODIC FILTERBANKS
SOUNDSYNP:
Anonymous authors Paper under double-blind review
ABSTRACT
Accurately estimating sound sources' temporal location, spatial location and se- mantic identity label from multi-channel sound raw waveforms is crucial for an agent understand the 3D environment acoustically. Multiple sounds form complex waveform mixture in time, frequency and space, so accurately detecting them requires representation that can achieve high resolutions across all these dimensions. Existing methods often cannot do it well because they either extract hand-engineered features (i.e. STFT, Log that require great deal of param- eter tuning work, or propose to learn a single-scale filter bank to process sound waveforms that has limited tume-frequency resolution capability. In this paper, we tackle this issue by proposing to learn group of parameterized synperiodic filter banks. Each synperiodic filter's length and frequency resolution are inversely related, hence is capable of maintaining a better time-freque frequency resolution trade-off. By alternating the periodicity term, we can easily obtain a group of synperiodic filter banks, where each bank differs in its temporal length. Convolution of the proposed filterbanks with the raw waveform helps to achieve multi-scale perception in the time domain. Moreover, applying synperiodic filter bank to recursively process a downsampled waveform enables to achieve multi-scale perception in the frequency domain. Benefiting from the multi-scale perception in both time and frequency domain advantage, our proposed synperiodic filter bank group learns a dynamic multi-scale frequeeyy representation in data-driven way. Following synperiodic filter bank group front-end, we add a Transformer-like backbone with two parallel soft-stitched branches learn semantic identity label and spatial loca- tion representation semi-indee comiiindeeeedentlyy Experiments on both direction of arrival estimation task and the physical location estimation task shows our framework outperforms existing methods by a large margin. Replacing existing methods' front-end with synperiodic filter bank also helps to improve the performance.
INTRODUCTION
The fundamental task for an agent to perceive and interact with the 3D environment is to know the location and semantic identity it nearby objects. The location includes spatial location that either stationary or moving, temporal location like start time and end time. Vision-based such environment perception has received large attention in the past decade and have witnessed huge progress in tasks such as object detection(Liu et al., 2016; Lin et al., 2014; Yang et al., 2019), classification(He al., 2016a) and tracking( (Wang et al., 2019). Nevertheless, the sound-based counterpart research has far lagged behind, despite all the fascinating properties sound signal exhibits. For example, sound ubiquitous and insensitive to ambient illumination change, has no field-of-view (FoV) constraints and is capable of circumventing physical barriers to perceive scene beyond line-of-sight. As a complementary sensing approach to vision, sound-base environment perception is of vital importance for acoustic scene understanding. A typical example is the sound source detection (answer where is it, when does it happen and what is it), given the recorded multi-channel sound waveforms.
To detect sound sources, often deploy apatially-configuees microphone array to record an acoustic environment. Unlike camera or LiDAR scanner that directly captures RGB image or measures the range distance, sound waveform itself is highly compressed dinedinnnll points

---

Under review as a conference paper at ICLR 2022
with high sampling rate, all sound source signals are compressed and mixed into the one-dimensional data format. Since different sound sources have different frequency property, it is essential convert waveform into time- -eeeueeeeyncyy representation so that frequencies hidden in the waveform are explicitly split out. This is often achieved by projecting the raw waveform into various orthogonal frequency basis. At the same time, sound source's spatial location clue lies in inter-channel difference of recorded waveforms (i.e. phase difference in this work). It is essential to design a neural network that jointly encodes mono-channel time-freug free-freyeeccy representation and inter-channel phase difference from the raw waveforms in unified, parameter-fruge frugal and computation-efficie manner. The learned representation should have elegant resolution in both time, frequency and space domain so that sound sources can be precisely detected.
However, learning such representation is tough task. Challenges derive from both theoretical side and practical side. According to Uncertainty Principle, we cannot achieve the optimal resolution in time and frequency domain at the same time, but instead keep a trade-off between them. Traditional hand-engineered sound feature(Davis & Mermelstein, 1980; Cao et al., 2021; Tho Nguyen et al., 2020; Brandstein & Silverman, 1997) and some recently proposed learnable filter bank Ravanelllll & Bengio, 2018; Zeghidour et al., 2018) empirically set the same length for all filters, resulting in human-biased unadjustable time-freet freeufryy resolution map. Some other (eek et al., 2021) correlates filter frequency response and filter length by initializing in mel-scale, but it is not scalable nor stable because the final initialization depends on the filter number. Moreover, all existing methods process raw waveform with one-scale filter bank, we argue that the one-scale sound perception casily leads to incomprehensive sound sources sensing, especially when sound sources have different frequency property or undergo various spatial motion.
In this paper, we first give comprehensive theoretical analysis on the filter bank impact on its extracted feature's 's time and frequency resolution. Based on the analysis, we propose a simple yet effective synperiodic filter bank construction strategy in which each filter's 's frequency response and length are correlated by rotating periodicity such that each filter's length is inversely proportional to its frequency resolution. The synperiodic filter bank thus internally maintains a better time- tine-freueency resolution trade-off than traditional fixed-length filter bank. Coupling the filter length with its frequency response helps us to reduce human intervention in filter bank design. By simply alternating the periodicity term, we further construct a group of synperiodic filter banks, with which we achieve multi-scale perception in time domain. At the same time, by applying a synperiodic filter bank to process one raw waveform as well as its in frequency domain. The multi-scale perception in both time and frequency domain of synperiodic filter bank enables the neural network to dynamically learn better representation for sound source detection data-driven way. It worth noting that synperiodic filter bank parameter number is just linear to filter number (adds up to less than 1% of the whole parameters) and it can be efficiently implemented as a 1D convolution operator.
versions, we achieve multi-scale perception
Following the aforementioned learnable front-end, we add Transformer-like backbone network with two paralleling branches with intermediate soft-parameter sharing learn sound source's 's semantic and spatial location related representation both jointly and separately. Experiment on both crrretion-of-arrivaa (DoA) task and physical location estimation task shows that our proposed framework outperforms comparing methods by a large margin. Replacing existing method's 's head with our proposed synperiodic filter bank also improvdes the performance.
RELATED WORK
Sound signal processing has been thoroughly studied in traditional digital signal processing area. The preliminary step of sound signal processing is usually to convert raw waveform into 2D time- frequency representation. There are two main realms: Fourier transform based and Wavelet based transform(Stum (Sturm, 2007). Traditional sound feature design are motivated influenced by human-auditory system. For example, they often convert frequency bins into mel-scale to imitate human hearing system, like MFCC(Davis & Mermelstein, 1980), LogMel(Cao et al., 2021; Grondin et al., 2019), the filter length is empirically chosen and often a windowing is added to avoid spectrum leakage. For inter-channel phase difference encoding, it is often recommended to encode in frequency domain due to the less cessputtuttaiion advantage. Typical phase difference features include GCC-Phatt Brandstein & Silverman, 1997) and intensity vector(Cao et al., 2021).
2

---

Under review as a conference paper at ICLR 2022
Sound source detection has been previously treated sound structure(Thrun, 2006) estimation, sound object detection(He et al., 2021) and sound event detection and localization (SELD) prob- (em(Adavanne et al., 2018; Thi Ngoc et al., 2021). It involves jointly identifying sound source's semantic label and predicting its spatial location, the two sub-tasks have been thoroughly studied separately in acousticx(Nandwant & Hasan, 2016; Mohan et al., 2008; Sundar et al., 2020; Vera-Diaz et al., 2018) and computer vision community(He et al., 2016a). T. Kim et al. (Kim et al., 2019) pro- vided a review and discussion for raw-audio based event classification. Benefiting from the success of the traditional hand-engineered sound feature and the large availability of mature image-based deep neural networks(He et al., 201 6a), most work(Grondin et al., 2019; Cao et al., 2021; Grondin et al., 2019; Adavanne et al., 2018; Thi Ngoc et al., 2021) tackle the task by first extracting hand-engineered sound feature and then feeding them to mature image-based neural networks. This workflow is straightforward and often guarantees reasonably good results but it is not end-to-end trainable and heavily depend on image side which may not be optimal for sound processing. At the same time, some work(He et al., 2021; Adavanne et al., 2018; Tho Nguyen al., 2020) have simplified the problem by assuming no two sound sources of the same semantic label but different spatial location happen the same time. This assumption avoids semantic label and spatial location association issue but may not reflect real scemarios.
In recent years, a bunch of work tried design neural work to directly learn from raw sound waveform, ranging from the earlier methods that directly apply stacked layers to process raw waveform al., 2019; Palaz et al., 2013; Jaitly & Hinton, 2011: Sainath et al., 2013) to the recent frequeney- sensitive filter bank learning methods(Zeehiidour et al., 2021; Ravanelli & Bengio, 2018; Zeghidour et al.. 2018; He al., 2021: Hoshen al., 2015; Sainath et al., 2015; Luo & Mesgarani, 2019). The filter bank parameter is initialized in either mel-seale(Ravanellli & Bengio, 2018; Zeghidour et al., 2021: He et al., 2021; Zeghidour al., 2018) or as Gammatone filter(Hoshen et al., 2015; Sainath et al., 2015). SoundDet(He et al., 2021) is the first work to directly learn from multi-channel raw waveforms to detect sound sources. designs MaxCorr filter bank to directly convolve with multi-channel raw waveforms to learn phase difference aware features.
Multi-scale representation has a rich history in computer vision community ((Liu et al., 2016; 2021; Lin et al., 2016; He et al., 2016b)), in which multi-scale representation strategy has been proposed to accommodate large object scale variation. For example, SSD(Liu et al., 2016) combines feature maps of various scale to cover potential objects of various sizes. In addition to spatial multi-scale representation, feature space multi-scale representation has been explored as well. For example, He (He al., 2016b) incorporate features arising from intermediate layers boost attribute recognition. In sound signal processing, Stéphane Mallat & Mallat, 2013; Mallat, 2012) proposed wavelet scattering obtain multi-scale sound representation by iteratively treating the proceeding processed sound waveform as new virtual waveform for further process.
SOUND SOURCE DETECTION FROM MULTI-CHANNEL RAW WAVEFORMS
3.1 SOUND SOURCE DETECTION PROBLEM DEFINITION
We use a opatially-deploeer microphone array, like four closely-bounded microphones in a circular coplanar configuration, to record an acoustic environment where various sound sources undergo inde- pendent and unconstrained spatial motion. Recorded multi-channel raw waveforms are represented as W {.,(t)){= each waveform is 1-d vector of size T sampled at a fixed sampling rate. We denote K sound sources by S = {s = (t& tk spk, slg) k=
s one single sound source corresponds to start time f., end time fe, stationary or moving spatial motion sp and a semantic identity label sl, Our goal is to learn a representation P that is representative enough to detect sound sources both spatially and semantically. P is temporal framewise representation of shape Ti x N. We propose to learn P directly from the raw waveforms via a deep neural network IF parameterized by 0,
PTIXN = F(WTxa(0), Ti & T
A learned elegant representation should have large resolution in both time, frequency and space domain and exhibit strong capability at addressing three intrinsic challenges: Polyphonicity, sound sources mutually independent, they can freely overlap temporally; Time-scale variance, the
3

---

Under review as a conference paper at ICLR 2022
Fixed Length fn
Variable Length
Syndistance
Syndistance
Synperiodic
Frequency
fn-1
Frequency
f1
Synperiodic
Time
Group
Group
Time
Time
Visual. in Complex Plane
Syndistance Filter
Synperiodic Filter
Dynamic Resolution Map
Syndistance filter bank (green color) rotates the same distance in complex plane and thus has the same kernel length, regardless of frequency it carries. Its time-frequency dynamic resolution map thus rectangular. our proposed synperiodic filter bank (blue color) are generated by rotating the same periodicity number. So filter carrying lower frequency has larger kernel size than those with higher frequency response. As result, synperiodic filter bank's 's time-frequence dynamic resolution map that is able to achieve better trade-off than traditional syndistance filter bank.
Figure 1: Synperiodic filter bank illustration:
temporal duration of different sound sources varies enormously, from seconds to minutes long; Datasize, sound waveform is sampled with large sampling rate, resulting in large data size.
Sound source presence clue lies in mono-channel time-frrquuee frequency representation and its spatial motion clue lies in inter-channel interaction, like phase difference. Our proposed synperiodic filter bank group jointly encodes time- frequeryyyncy representation and phase difference from raw waveforms in a unified way. More detail about phase difference encoding is given in the Appendix A.l.
3.2 MULTI-SCALEE SYNPERIODIC FILTER BANK
We denote the general filter bank of M filters by JF (F, D., 4. ((w;) w(o))) Each single filter F; is created by multiplying the frequency-selective filter ((w;) of frequency response wi (i.e. sinusoidal basis or band-pass basis(Ravanelli & Bengio, 2018)) with windowing function w(o)). The windowing function has locality property which means it is just active within a local region controlled by o; (i.e. Gaussian kernel window). Then I, points are cropped around the active region and treated as a frequency-selectton their way of parameter setting, including filters' frequency response distribution, filter length length and windowing function. In classic hand-engineered features such as short-time Fourier transform (STFT) and LogMel feature, all these parameters are empirically chosen. Contrary to STFT, wavelet transform (Sturm, 2007) inversely correlates window length with frequency response that the active region of the filter is internally decided by its frequency response.
filter (usually I, oz)). Existing filter bank differs
Time-Frequent Dynamic Resolution Frequency resolution indicates the ability of discerning two adjacent frequency bins, time resolution corresponds to the capability of precisely localizing sound source in time domain. The resolution in time and frequency domain jointly influence spatial localization precision because we encode spatial location on top of on pre-extracted fime-freuenney representation. According to Uncertainty Principle. however, the frequency resolution A and time resolution A, satisfies A A, 3 C' (C is a constant). It means we cannot get optimal resolution in both time and frequency domain at the same time, but rather keep a trade-off between them: increasing the resolution in one domain inevitably sacrifices the resolution in the other domain.
Traditional hand-engineered sound feature's fime-freyuency resolution map fixed because all filter bank construction relevant parameters are empirically chosen. Therefore, its time-frequency resolution map is evenly divided across both the time and frequency domain (see Fig. second last figure). Relaxing the frequency response and windowing length trainable(Zeghidour et al., 102 helps to achieve better time-free free-ffry resolution trade-off. The improvement is, however, limited because it uses one filter bank of the same filter length to process all sound signals (or perceive at one-scale). We categorize these traditional filter bank as syndistance filter bank to emphasize their equal temporal length property across all frequency responses.
In this paper, we rethink filter bank design and propose a synperiodic filter bank construction strategy. Our motivation is based the fact that fighffrryeency sound signal can be adequately detected
4

---

Under review as a conference paper at ICLR 2022
with short filter because its temporal length of one period is small. Conversely, lower frequency sound signals require wider filters. It thus shows using the same window size for all filters across the frequency range is not a good choice, more desirable way is to narrower window for filters with higher frequency responses. Under this guidance, we propose a synperiodic filter bank construction strategy. To better understand the main difference between syndistance filter bank and our proposed synperiodic filter bank, we can visualize them in complex-valued plane (see Fig. left-most), in which filter is complex exponential rotating in the complex plane counter-llockwisely, the rotating speed corresponds to the frequency it carries. In the complex-value rotate to the same distance. Synperiodic filters, however, rotate a predefined periodicity p. naturally resulting in narrow window for high-frequency filters and wide window for low-frequency filters.
plane, all syndistance filters
Synperiodie filter bank lends us three advantages: first avoids us setting window length for each filter separately, which is quite empirical and random; second, the constructed filter bank by design maintains good time-frequency resolution trade-off; third, by simply varying the periodicity term p. we can easily obtain group of synperiodic filter bank to process the raw waveform in multi-scale manner (as we will present in below). Our synperiodic filter bank construction strategy shares similar idea with Wavelet transform(Stune (2007)) where it adopts a time shift and "squeezing ratio" to create achieve multi-scale perception, the difference is that we omit the time shift and instantiate the squeezing ratio with our proposed synperiodicity strategy. Moreover, synperiodic filter bank is multi-scale in time and frequency domain and seamlessly works well with convolution operation, and it is end-to-end trainable. Specifically, our proposed synperiodic filter bank can be represented as,
Feymp symp (fe(ws. p. 1,) = o(w;) w(w.,p)) where w(ax) = www., p)
(2)
where we can see that the windowing function just depends on the center frequency w; and the periodicity term p. By varying the periodicity term p. we can obtain group of synperiodic filter bank Fsymp = (FP: _FP.... .... FP²), ), each group differs in its window size. The comparison between syndistance and synperiodic filter bank shown in Fig. 1.
There are many ways to instantiate w(ws, p). as long as we guarantee the window length gradually reduces as the the frequency response increases. The simplest choice is to treat w(w; p) as a constant, but we find it either results in too wide window for low-frequency filters or too narrow window for feeeffrryeency filters. To mitigate this dilemma, we use logarithmic windowing function.
wWi, Wi, p) 27 log - Ps = (-6, - 11, -66]
(3)
We set P as 1-6,-11,-11 16] respectively to construct three synperiodic filter banks. The design of this window function motivated by mel-scale frequency initialization strategy. By roughly setting a filter's 's bank width to be equal to the distance between its preceding and next firequency location in frequency domain, converting to time domain we can roughly get logarithmic scale frequency- periodicity relationship (see Fig.4 4 in Appendix). In our implementation, synperiodic filter is created by multiplying a sinusoidal basis with learnable frequency response initialized in mel-scale by a Gaussian window with learnable width initialized through the windowing function by Eqn.3. Please note that each synperiodic filter bank group is initialized with independent learnable frequencies and window length, they are independently updated during training stage.
3.3 MULTI- LEARNING IN TIME AND FREQUENCY DOMAIN
We use the previously constructed synperiodic filter bank group to convolve with mono-channel sound waveform with the same step size and padding strategy, resulting in the same size output for each single synperiodic filter bank. Since different filter group has different window size, we naturally achieve multi-scale learning in time domain. It maximally avoids us empirically selecting one window scale p which might not be optimal, but instead uses group of filter bank to enforce the neural network to strike better time- feneefreyunccy resolution trade-off in a data-driven way.
We further propose a strategy to enable multi-scale learning in frequency domain. The strategy is hierarchical: given a raw sound waveform with sampling frequency Fs. synperiodic filter bank's frequency is initialized within the range [o, f under Nyquist sampling theorem. If we downsample the sound waveform by factor of 2, the resulting waveform can be processed by the lower-half
5

---

Under review as a conference paper at ICLR 2022
Group
Group
Frequency
Cony
Maxpool
Multi-scale Conv in Freq.
Domain
Multi-scalc Time-Freq. Rep.
Pooled Fime-Freq. Rep.
Figure 2: Multi-scale learning in frequency domain. Given the raw one channel sound waveform and pre-constructed synperiodic filter bank, consecutively downsample the waveform by factor 2x, the newly downsampled waveform is processed by low-half filter bank from the proceeding filter bank. We can obtain fime-frequency representation for on each frequency scale. These time- frequency representations share the same time length by adjusting step size. The final fineefryyynncy representation obtained by sax-pooling them together.
filters in each group whose frequency response lies in [0, ] This process that 2x 2x-downsampling the waveform further process the downsampled waveform with filters with lower-half frequency response can potentially iterate couple of times (in our case three times), resulting in multi-scale perception in frequency domain. The figurative illustration is shown by Fig. 2. Multi-scale learning in frequency domain brings us two extra benefits: 1) from data augmentation perspective, hierarchical 2x downsampling inevitably creates more waveforms, which matters a lot when labelling audio data is technically hard and time-consuming. 2) from the causality perspective, the adjacent 2x- downsampling strategy leads to dilated convolution for lower- weer-ffequency filters, because applying filter to convolve with a downsampled waveform equals to convolve on the original waveform with dilated convolution (skip-2 connection). The resulting wider field of view (FoV) for lower frequency filters enables to learn better sound causality along the time axis (see Appendix A.4). In sum, by using learnable synperiodic filter bank group process the raw waveform in multi-scale manner, achieve a dynamic time-frequency resolution map that naturally maintains a better time-frequency resolution map fitting for sound source detection in data-driven way.
Computational Analysis Synperiodic filter bank group introduces very few parameters (less than 1%) because they are parameterized filters. The trainable parameter number increases linearly w.r.t. synperiodic filter bank number. Their convolution with raw waveforms can also be efficiently implemented with ID convolution.
3.4 TANSFORMER-LIKE BACKBONE WITH TWo SOFT-STITCHING BRANCIIE
Synperiodic filter bank group generates feature representation of shape [To, No. C] (To is much smaller than the T but still larger than Ti. No equals the filter number and we use 256. C is channel number). Jointly learning framewise sound source semantic label and spatial location representation multi-task problem(Kendal et al., 2018: 8; Misra et al., 2016). We propose a Transformer-I like backbone with two paralleling and identical branches to learn each sub-task sepa- rately. To enforce information communication, we add a layerwise information exchange module: for the intermediate semantic label feature f² and spatial location feature fg learned by the i-th block, learnable weight W, is introduced to linearly combine them together to get an updated ft and f = W. ff.. practice, the Transformer-like
before feeding them to the next layer, backbone gradually reduces the learned feature's temporal length but increases the channel dimension accordingly. Finally we can get the final sound source representation of shape [T..N. On top of the representation, add trackwise trains the whole neural network in an end-to-end manner. The permutation invariant training strategy helps to address the polyphonicity challenge because it encapsulates potential sound source's semantic label and spatial location within a track, so it won't 't be influenced by other sound sources.
ionnutaaioo-invarinn training (PIT) strategy et al., 2021) to
EXPERIMENTS
We conduct experiments on two tasks: direction of arrival (DoA) and physical location estimation. DoA estimation requires to decide arrival angle of sound source microphone array. Physical location

---

Under review as a conference paper at ICLR 2022
Table 1: Evaluation result for DoA task. For Segment-based evaluation, we report detection error E E2 F measure F20- under DoA threshold 20", and classification dependent localization error LECD and localization recall LRCD. For wvent-based evaluation, we report mAP/mAR, The "Input" column labels are: O. Raw waveform, 1. Log- Mel. 2. GCC-Phat, 3. Intensity Vector
Methols
Pamms
Inpui
Segment Based Evalustion
(1)
LE(T)*>
LR(1)
Event based Evaluation
mAP(1)
mA
SELDDNeufoo Foox Adavanme al., 2018)
0.5
0.3
046
25.1
0.69
0.087
0152
MELLDNtmmie Adavaanee et al.. 2018)
0.5M
1.2
0.66
0.43
24.2
0.66
0,079
0.140
EINifoxaCann 2023
26.0
1.2
0.30
0.77
3.9
0.84
0.134
0.256
SoundDeeffuyffffe 2021)
M
8.3
1isC:lllvve/foo-mmiiill et al., 20200
Ensemble
1,23
0.20
0.85
6.0
0.89
SoundSyappmii)
60,0 M
0.19
0.56
5.5
0.91
0.210
0.313
Time GT
Time SELDNet
Time BIN
Time SoundDet
Time SoundSynp
Adimath GT
Azimuth SELDNet
Azimuth EIN
Kamuth SoungDet
Azimuth SoundSynp
100
-100
Figure 3: Qualitative comparison on DoA task. We show detected sound source temporal location (top row) and azimuth (bottom row). The horizontal axis is time, the vertical axis is semantic label (top) and azimuth in degrec(bottom). Different color indicates different sound source category.
directly localizes a sound source's physical location (i.e. (z, y. 2) coordinates). For DoA estimation task, we use DCASE2020 sound event detection and localization dataset(Politis et al., 2020). It contains 14 sound sources with azimuth range [- 180 and elevation range [-45", 45 Two recording formats are available: FOA and MIC-array (refer to Appendix for detailed discussion). For more details about this dataset, please refer to (Politis et al., 2020). For physical location estimation, we use pyrommacousiicsScchiibk.. et al., 2018) simulator to simulate the dataset.
4.1 DIRECTION OF ARRIVAL ESTIMATION
We adopt two evaluation metrics: segment-based and event-based metric. Segment- based metric is widely adopted evaluation metric(AAavanni et al., 2018; Cao et al., 2021), couples semantic label and spatial location together: semantic-correctt detected sound source needs to be spatially close enough to its ground truth location in order to be regarded as true positive detection. Event- based metric is newly proposed by(He et al., 2021) to comprehensively evaluate under different confidence scores. Like object detection from images(Lin et al., 2014), it computes mean average precision (mAP) and mean average recall (mAR) score.
We call our framework SoundSynp and compare it with four most recent methods: SELD- Net((Aavanne et al., 2018), EIN(Cao et al., 2021), SoundDet(He et al., 2021) and Utsc-Ifltte(Wang et al., 2020). SELDNet is the baseline model and it jointly trains sound source's semantic label and spatial location with convolutional recurrent neural network (CRNN)(Chung et al., 2014). EIN(Cao et al., 2021) is very recent work. It adopts multi-heads self-attention( Vaswani et al., 2017) to model temporal dependency and trackwise SoundDer(He et al., 2021) directly learns from raw waveform with MaxCorr kernels, followed by an encoder-decoder neural network. Utsc-Iflytek( (Wang et al., 2020) is ranked first in DECASE2020 challenge leaderboard it combines MIC and FOA features and ensembles different models like ResNet(He et al., 2016a) and Xception(Chooll Chollet, 2017) to give final prediction.
ietiuuattioniivvaiiaa training to train the model.
Implementation Detail The network architecture is shown in table 4 in Appendix material. To train the neural network, we evenly divide the one minute long four-channel raw waveform into non-overlapping 4s short snippets. The raw waveform is first normalized to [-1, 1] before feeding the neural network. We adopt Adam optimizer( (Kingma & Ba, 2015) with an initial learning rate 0.0002 in the first 100 epochs and 0.00007 in the following 50 epochs. Batchsize is 16. The loss combination weight between classification head and regression head is 1 2. During training, data
this link for leaderboard report

---

Under review as a conference paper at ICLR 2022
augmentation method SpecAugment(Park al., 2019) is applied. For DoA task, we regress direction of arrival angle in Cartesian coordinates [a, y, 2|. In synperiodie filter bank groups, the filter length is 1025, each group's 's filter number is 256 and the step size is 600. Particularly, we have observed the initialized learnable synperiodic filter banks update its parameters intensively during the very early several epochs, and then gradually becomes stable. The final learned parameters are close to their initialization, one group's s learned parameters (like frequency response and filter window length) is different from those learned by other groups, although they are initialized as the same in the start. We train each model five times independently and report the average score. The standard deviation is within 0.04 (for recall) and 0.2 for angle, 0.003 for mAP and mAR.
The result is given in Ta from which we see that SoundSynp achieves the best performance over all comparing methods by large margin, under both segment-han and event-based metrics. Both SELDNet, EIN and UTSC-Iffytek use pre-extracted hand-engineered sound features, such as Logmel, GCC-Phat and Intensity Vector. SoundDet(He et al., 2021) and SoundSynp are the only two methods that directly learn from raw waveforms. At the same time, SoundSynp obtains better performance on FOA than MIC format, the same phenomena has been observed by all other methods. thus shows et al.,
FOA better fits for sound souree detection than MIC. It is worth noting that Ifae--lllyekkeWWee 2020) ensembles different powerful image-based 2D models to detect sound sources. However, our proposed SoundSynp still outperforms Utsc-Iflytek by large margin. We don't report of mAP/mAR value for Utsc-Iflytek because it is a complex system and no detail about their system is available.
Ablation Study To disentangle the individual contribution of each part of our SoundSynp framework to the whole performance improvement, we conduct three ablation studies. First, the individual contribution of Synperiodic filter bank. We replace hand-engineered sound feature head of SELDNet, EIN and learnable MaxCorr filter bank with our proposed Synperiodic filter bank to test their corre- sponding performance. It helps to remove the influence of the backbone neural network of different models and thus helps to get direct comparison of synperiodic filter bank with other front-end. Second, we replace SoundSynp's synperiodic filter bank group with widely-used MFCC(Davis Mermelstein, 1980) and Log-Mel that is used by SELDNet( Adavanne et al., 2018) and HIN(Cao SEL.DNe(AAdaaaee
et al., 2021), I), respectively. It disentangles synperiodic filter bank group with Transformer-like back- bone, thus helps to figure out if the performance gain is simply brought by backbone network. Third, internally, we test five synperiodic variants: (1) synperiodic filter bank with just multi- scale perception in frequency domain domain (SoundSynp (SoundSynp.MSTTimm MSTime), (3) Synperiodic filter bank with frequency responses linearly ini- tialized in Nyquist frequency range (SoundSynp Linear, compare with our mel-scale initialization), (4) just one synperiodic filter bank without multi-scale perception neither in time nor frequency domain SeaunddynpSSingleeeeee
(2) just multi-scale perception in time
(5) synperiodic filter bank with rectangular band-pass frequency response initialization (SoundSynp Sinc), like SinvNee(Ravanelli & Bengio, 2018) does. The internal comparison helps us to figure out the necessity of each part of synperiodic filter bank design.
The ablation study result is shown in Table 2. We can observe that: First, using synperiodic filter bank as a replacement of existing filter bank can help to improve the corresponding performance. Second, replacing SoundSynp's synperiodic filter bank with classic hand-engineered features inevitably reduces the performance under all evaluation metrics, its thus shows learning from pre-extracted fixed and single-scale sound representation leads to inferior performance than our proposed multi- scale synperiodic filter bank. Third, the absence of multi-scale perception in either frequency domain time domain inevitably reduces the performance. We find sound source semantic label detection suffers more in single-scale perception in time domain than in frequency domain (see better performance E Rg0,, and F20 score), which shows frequency domain multi-scale perception is vital for semantic label estimation. Similarly, we can observe that multi-scale perception in time domain is vital for sound source spatial location estimation (see better performance on LE and LR score). Linearly initialized filter bank frequency response reduces the performance, which shows assigning more filters to the lower frequency range important. But this conclusion might be data-dependent because we find DCASE dataset contains many low- www-freqeeccy sounds like burning fire and footsteps. Moreover, reducing the synperiodic filter bank group to one groups with just single- scale perception leads to comparably the worst performance, it thus shows multi-scale perception in both time and frequency domain is essential for DoA- sound source detection. Lastly, SoundSynp_SSin Sinc leads to slightly inferior performance than our used mel-scale initialization strategy, it shows our proposed synperiodic filter bank is a general filter bank that can be adopted to other frequeney sensitive filter bank.
8

---

Under review as a conference paper at ICLR 2022
Table 2: Ablation study report. Internally, replace SELDNet, EIN and SoundDet head with
synperiodic filter bank (denoted as SSyp)). Externally, we test different modified SoundSynp versions.
Sogmen: Rased Evalution
Event- based Evaliation
Methods
Blocks
LE(D)'
LR(())
mAP(1)
mAR(C)
SRI DNer Synptfoui Adhramee et ol. 2018)
M.LLDNetfoooAAAavaaa ai., 2018)
CanvlD, biGRU
0.6
0.57
0.99
22.1
0.75
0.003
0.165
SELDDeeimmj (Adzvanne
2018)
Com22D, HCTRR:
0.66
0.13
21.2
0.66
0.09
0.140
SELD Maw.eeSSyyymmmiAAAaaaaaaa
2018)
23.0
EIN SympplyppuiiCaa et 202
EIN(fouX Cao et al., 2021)
Coev2D, MHSA
0.27
0.30
0.77
0.79
8.9
8.3
0.87
0.84
0.138
0.134
0.258
0.256
SoundDet Synptlou) (Ile et al., 2021)
SoundDeittoaatte
2021)
CoevID, LSTM
0.22
0.83
8.3
7.9
0.85
0.204
0.299
SoundSyng (oMMollaa)))))
JounlSynnMMPCCC(ffu MFCC(((u))
Comv21, MHSA
0.21
083
6.9
0.85
0.201
0.301
3.306
SoundSyup-MSFreqtta MSPreqioos
0.20
035
7.3
0.86
0.210
0.284
SoundSynp.Wo MMM)mmm
0.22
0.82
7.0
0.58
0.172
0.282
SoundSyap Lincur (foa)
Conv2D, MHSA
0.83
8.8
0.168
0.259
SouadSyup-Singleeolll Songleeealll)o))
9.0
SoundSynp Sinctfoa) (foa)
0.19
0.86
5.2
0.91
0.226
0.316
SoundSynpiloa
Co MISSA
0.16
0.38
4,3
0.93
0.23
0,327
Table 3: Physical location estimation result report. We set distance threshold 1.0m, which means a detected sound sources has to lie within this threshold in order to be treated true positive.
Exalustion
Evalosionn
F (m. (4)
"1.mll
T.((T)
mAP(T)
mAR(1)
ElNoCao at, 30213
0.45
0.71
1.71
0073
0.114
0.340
SynpSyn 2021)
0.72
0.244
SoundDatHee et 2021)
0.67
0.68
0.74
0.118
CoundDee.Symppii al., 202))
0.44
0.69
0.64
0.79
0.123
0.271
SoundSyup
0.79
0,53
0.83
0.154
0.379
One qualitative comparison is shown in Fig. 3. We can clearly see that SELDNet generates mixed prediction at different time steps and DoA locations. SoundDet and EIN give non-existing sound sources (orange color). When multiple sound sources happen at the same time (polyphonicity). SoundDet and EIN are easily failed to predict the right spatial location (discretized blue and red color). Our method predicts more spatially and temporally consistent sound sources by maximally keeping sound source's 's continuity and consistency.
4.2 PHYSICAL LOCATION ESTIMATION
We use Pyroomacoustics simulator(Scheible al., 2018) to simulate a shoebox like room with [7m, 5m, 3m] size. Four microphones (24kHz sampling rate) are configured in a plane with 5cm mutual distance and put in the room center ([3.5m, 2.5m, 1.5m)). We choose three commonly heard sounds from BBC sound effect site?: cat meowing, baby talking and dog barking. All these seed sounds last less than 10s. During simulation, we put all seed sound in the plane 2 1.5m (we constrain to a plane because we find relaxing z casily leads EIN model to get stuck in local minima like [-1, 1, -1]), but with æ randomly in [2m,, 5m] and y randomly in [1m, 4m]. up to two sounds can happen at the same time. In sum, we have simulated 500 one-minute recordings (400 for train, 100 for test), with an average of four sounds recorded per recording. For physical location prediction, regress the angle and range. The result is in Table 3. We do not report SELDNet result as find it doesn't 't converge during train. We can see from the table that SoundSynp produces the best performance than over EIN and SoundDet network. Replacing their front-end with synperiodic filter bank also improves the performance. thus shows our proposed SoundSynp is capable of learning representation for ahysical-location based sound source defection.
4.3 DISCUSSION AND CONCLUSION
We have proposed a novel learnable sound feature extractor front-end: synperiodic filter bank. It requires very few parameters and computational efficient. It extracts feature in multi-scale manner in both time and frequency domain. Although it shows excellence in detecting sound sources, we believe it can be used as general front-end to tackle other sound tasks, like speech processing and keyword spotting.
see https://soondd bborewi nd.co.uk

---

Under review as a conference paper at ICLR 2022
REFERENCES
Sharath Adavanne, Archontis Politis, Joonas Nikunen, and Tuomas Virtanen. Sound event localization and detection of overlapping sources using convolutional recurrent neural networks. In IEEE Journal of Selected Topics in Signal Processing, 2018.
M. S. Brandstein and H. F. Silverman. A robust method for speech signal time-delay estimation in reverberant rooms. In IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 1997.
J. Bruna and S. Mallat. Invariant scattering convolution networks. IEEE Transactions Pattern Analysis Machine Intelligence (TPAMI)) 2013.
Yin Cao, Turab lqbal, Qiuqiang Kong, Fengyan An, Wenwu Wang, and Mark D Plumbley. An improved event-independent network for polyphonic sound event localization and detection. In IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP). 2021.
François Chollet. Xception: Deep learning with depthwise separable convolutions. In IEEE Confer- ence on Computer Vision and Pattern Recognition (CVPR), 2017.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural ntworks sequence modelling. In Adv. Neural Inform. Process. Syst., 2014.
Steven Davis and Paul Mermelstein. Comparison of parametric representations for monosyllabic word recognition in continuously spoken sentences. In IEEE Transactions on Acoustics, Speech, and Signal Processing, 1980.
Francois Grondin, James Glass, Iwona Sobieraj, and Plumbley Mark D. A study of the complexity and accuracy of direction of arrival estimation methods based on goe-phat for a pair of close microphones. In Proceedings of the Detection and Classification of Acoustic Scenes and Events Workshop, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Sun Jian. Identity mappings in deep residual networks. In European Conference on Computer Vision (ECCV), 2016a.
Yuhang He, Long Chen, and Jianda Chen. Multi-task relative attribute prediction by incorporating local context and global style information. In Proceedings of the British Machine Vision Conference (BMVC), 201
Yuhang He, Andrew Markham, and Niki Trigoni. Sounddet: Polyphonic sound event detection and localization from waveform. In International Conference on Machine Learning (ICML), 2021.
Yedid Hoshen, Ron J. Weiss, and Kevin W. Wilson. Speech acoustic modeling from raw multichannel waveforms. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2015.
Navdeep Jaitly and Geoffrey Hinton. Learning a better representation of speech soundwaves using restricted boltzmann machines. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2011.
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Computer Vision and Pattern Recognition (CVPR), 2018.
Taejun Kim, Jongpil Lee, and Juhan Nam. Comparison and analysis of sampleenn architectures for audio classification. IEEE Journal of Selected Topics in Signal Processing, 2019.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representation (ICLR), 2015.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In David Fleet. Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), European Conference on Computer Vision (ECCV), 2014.
10

---

Under review as a conference paper at ICLR 2022
Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yare Fu, and Alexander Berg. Ssd: Single shot multibox detector. In European Conference Computer Vision (ECCV), 2016.
Ze Liu, Yutong Lin, Cao Yue, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Guo Bain- ing. Swin transformer: Hierarchical vision transformer using shifted windows. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Yi Luo and Nima Mesgarani. A sequence matching network for polyphonic sound event localization and detection. In EEE/ACM Transactions on Audio, Speech and Language Processing, 2019.
Stéphane Mallat. Group invariant scattering. In Communications in Pure and Applied Mathematics, 2012.
Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. Cross-stitch networks for multi-task learning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2016.
Satish Mohan, Michael Lockwood, Michael L. Kramer, and Douglas Jones. Localization of multiple acoustic sources with small arrays using a coherence test. In The Journal of Acoustical Society of America, 2008.
Mahesh Kumar Nandwana and Hasan. Towards smart-cars that can listen: Abnormal acoustic event detection the road. In Interspeech, 2016.
Dimitri Palaz, Ronan Collobert, and Mathew Magimai-Doss Estimating phoneme class conditional probabilities from raw speech signal using convolutional neural networks. In Interspeech, 2013.
Daniel Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin Cubuk, and Quoc Le. Specaugment: A simple data augmentation method for automatic speech recognition. 2019. URL https::/ /aapiii.aaiivvaa/aa org/ abs 1904. 08779. 79, arxiv:190.00877.
Archontis Politis, Sharath Adavanne, and Tuomas Virtanen. dataset of reverberant spatial sound scenes with moving sources for sound event localization and detection. In Detection and Classifi- cation of Acoustic Scenes and Events Workshop (DCASE), 2020.
Mirco Ravanelli and Yoshua Bengio. Speaker recognition from raw waveform with sincnet. In In IEEE Workshop on Spoken Language Technology (SLT), 2018.
Tara N. Sainath, Brian Kingsbury, Abdel-rahman Mohamed, and Bhuvana Ramabhadran. Learning filter banks within a deep neural network framework. IEEE Workshop on Automatic Speech Recognition and Understanding, 2013.
Tara N. Sainath, Ron J. Weiss, Andrew Senior, Kevin W. Wilson, and Oriol Vinyals. Learning the speech front-end with raw waveform cldnns. In Interspeech, 2015.
Robin Scheibler, Eric Bezzam, and Dokmanić Ivan. Pyroomacousties: A python package for audio room simulations and array processing algorithms. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2018.
Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. wav2vee: Unsupervised pre-training for speech recognition. In Interspeech, 2019.
Bob L. Sturm. Stéphane mallat: wavelet tour of signal processing, 2nd edition. Computer music journal, 2007.
Harshavardhan Sundar, Weiran Wang, Ming Sun, and Chao Wang. Raw waveform based end-to- end deep convolutional network for spatial localization of multiple acoustic sources. In IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP). 2020.
11

---

Under review as a conference paper at ICLR 2022
Tho Nguyen Thi Ngoc, Ngoc Khanh Nguyen, Huy Phan, Lam Pham, Kenneth Ooi, Douglas Jones, and Woon-Seng Gan. A general network architecture for sound event localization and detection using transfer learning and recurrent neural network. In IEEE International Conference Acoustics, Speech, and Signal Processing (ICASSP), 2021.
T. N. Tho Nguyen, D. L. Jones, and W. Gan. A sequence matching network for polyphonic sound event localization and detection. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP),
Sebastian Thrun, Affine structure from sound. In Advances in Neural Information Processing Systems (NeurIPS), volume 18, pp. 1353-1360. MIT Press, 2006.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jacob Uszkoreit, Llion Jones Jone, Aidan N. Gomez, and Lukasz Kaiser. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Juan Manuel Vera-Diaz, Daniel Pizarro, and Javier Macias-Guarase, Towards end-to-end acoustic localization using deep learning: From audio signals to source position coordinates. Sensors, 2018.
Qiang Wang, Zhang, Luca Bertinetto, Weiming Hu, and Philip H.S. Torr. Fast online object tracking and segmentation: A unifying approach. In JEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR), 2019.
Qing Wang, Huaxin Wu, Zijun Jing, Feng Ma, Yi Fang, Yuxuan Wang, Tairan Chen, Jia Pan, Jun Du, and Chin-Hui Lee. The ustc-iflytek system for sound event localization and detection of dcase2020 challenge. In Tech. Report of SELD Decase2020 Challenge, 2020.
Bo Yang, Jianan Wang, Ronald Clark, Qingyong Hu, Sen Wang, Andrew Markham, and Niki Trigoni. Learning object bounding boxes for 3d instance segmentation on point clouds. In Advances in Neural Information Processing Systems, pp. 6737-6746, 2019.
Neil Zeghidour, Nicolas Usumier, Iasonas Kokkinos, Thomas Schatz, Gabriel Synnaeve, and Em- manuel Dupoux. Learning filterbanks from raw speech for phone recognition. In /EEE International Conference Acoustics, Speech and Signal Processing (ICASSP). 2018.
Neil Zeghidour, Olivier Teboul, Félix de Chaumont Quitry, and Marco Tagliasacchi. Leaf: A learnable frontend for audio classification. International Conference on Learning Representations (ICLR), 2021.
12

---

Under review as a conference paper at ICLR 2022
A APPENDIX
A.1 SPATIAL LOCATION ENCODING IN FREQUENCY DOMAIN AND RECORDING FORMAT DISCUSSION
We discuss the detailed spatial location encoding for FOA and MIC sound waveform recording format. The spatial location encoding is based on 2D feature learned by the Gabor filter bank for each sound waveform channel, which can be represented as {F; = (R, 4:)ii_ R² and I; are real part and imaginary part feature of the i-th channel sound waveform, respectively.
FOA format is well-known as first-order Ambisonics (B-format). It contains four channels: omni- directional, x-directional, y-directional and z-directional components, respectively. The instantaneous sound intensity vector is often used as spatial location (or phase difference) feature, which can be computed through the cross-spectrum between the uuni-directional channel to the remaining x, y, 2- directional. As result, we have obtained channel spatial location encoding feature.
(4) where F indicates the conjugate of the omni-directional feature. The three cross-spectrum feature TV,, IV, and IV2 are stacked together and further normalized before serving as the spatial encoding feature.
= -
IV, F F3
MIC format well-known as tetrahedral microphone array. The four microphones are mounted in spherical coordinates with four distinct orientations. We treat the four microphones equally and compute the phase difference between any two microphones. Thus a total of six channels spatial location feature can be constructed. Specifically, we choose to compute Silverman, 1997) like cross-spectrum feature. For any two channel m and n. we compute the angle between the real part and imaginary part of the cross-spectrum.
&
SL = 1:nggleeFFF F* Fn), + n2, m 1,00-,,2,,22 1,2,3,4; = 1,2,3,4
(5)
SL indicates the spatial location feature computed by the sound waveform channel m and n. The angle(-) equals frequency amplitude normalization operation, like the GCC-PHAAT(rrnndsein & Silverman, 1997) does. Please note that all the spatial location feature computation operations are differentiable so the whole neural network becomes end-to-end trainable.
A.2 NETWORK ARCHITECTURE AND TRAINING DETAILS
SoundSynp neural network architecture is given in Table To train the neural network, we evenly divide the one minute long four-channel raw waveform into non-overlapping raw waveform is first normalized to [-1, 1J before feeding the neural network. We adopt Adam optimizer(Kingnn & Ba, 2015) with an initial learning rate 0.0002 in the first 100 epochs and 0. 0.00007 in the following 50 epochs. Batchsize is 16. The loss combination weight between classification head and regression head is 1 2. During training, data augmentation method SpecAugment(/ (Park et al., 2019) is applied. For DoA task, we regress direction of arrival angle in Cartesian coordinates [xc. y, 2]. In synperiodic filter bank groups, the filter length is 1025, each group's filter number is 256 and the step size is 600. Particularly, have observed the initialized learnable synperiodic filter banks update its parameters intensively during the very early several epochs, and then gradually becomes stable. The final learned parameters are close to their initialization, one group's 's learned parameters (like frequency response and filter window length) is different from those learned by other groups, although they are initialized as the same in the start.
short snippets. The
A.3 SYNPERIODIC FILTER BANK PREQUUENCY-PERIDDICCT PERIODICITY RELATIONSHIP DETERMINATION
Mel-scale time-frequency representation has been widely used in both traditional sound feature like MFCC(Davis & Mermelstein, 1980), LogMel and learnable filter bank(Zeghidour et al., 2021). It initializes the filter bank in frequency domain, in which gighfrregeenyy filter has wider window length. We transform the filter bank into time domain and can naturally get a roughly logarithmic-see relationship, in which narrowr window width is associated with freeufrryeyncy
13

---

Under review as a conference paper at ICLR 2022
Table 4: SoundSynp neural network architecture. The layer follow name@kernelsize, stride format, and synperiodic filter bank follow name@kernelsize stride, groups format. FC is fully connection layer, AvgPool the average pooling layer, MaxPool is max- max--oolligg layer. the batch- size, T is input waveform time-length. All convolution layers are followed by batch normalization layer and Relu activation layer. We represent the neural network architecture on FOA recording format with sampling rate 24k Hz and label resolution 100 ms. It can be easily adjusted to fit other cases. Please note that since the backbone neural network has two identical branches, we just show one branch here.
layer
filter num
output size
Input: [B,4,T]
Synperiodic Filter Bank Groups
SynperiodiicFilceraankk @ 1024,600,3
256
[B. 256, T/600, 21]
Backbone Conv blockl
Conv2d @3,1
128
[B, 256, T/600, 128]
Conv2d @ 3,1
128
[B, 256, T/600. 128]
AvgPool @2,1
None
[B, 128, T7600, 128)
Backbone Conv block2
Conv2d@13 @3,1
256
BB 128, T/600, 256]
Conv2d@13 @3,1
256
B, 128, T/600, 256]
AvgPool @2.1
None
[B, 64, T/600, 256]
Backbone Conv block3
Conv2d 2d @ 3,1
256
[B, 64, T/600, 256]
Conv2d @3,1
256
[B, 64, T/600, 256]
AvgPool @2.1
None
[B. 32, T/600, 256]
Backbone Conv block4
Conv2d@3,1 Conv2d @3,1
512
B. 32, T7600, 512]
512
[B. 32, T/600, 512]
AvgPool @221
None
[B, 16, T/600, 512]
Backbone Cony block5
Conv2d @3,1
512
[B. 16, T/600, 512]
Conv2d@23 @3,1
512
[B, 16, T/600, 512]
AvgPool @ @1,1
None
[B, T7600, 512]
Backbone MHSA block1
MHSA @8, 1024
512
[B, T/600, 512]
AvgPool
None
[B, T/1200, 512]
Backbone MHSA block2
UIHSA@8,102 AvgPool @21
512
[B T/1200, 512]
None
[B, T/2400, 512]
Backbone MHSA block3
MHSA @8, 1024
512
[B. T/2400, 512]
FC
class num
[B, T72400, class num]
FC
class num x
[B, T/2400, class num x 3]
Trackwise Permutation Invariant Head
Multi-label Classification
None None
scalar
Location Regression
scalar
filters. We thus set w(ws; p) = 27 (w:) - p. We plot our synperiodic filter bank window
function and the mel-scale initialized windowing function in Fig. 4, it shows our proposed windowing function naturally approximates the mel-scale windowing function.
A.4 CAUSALITY IN MULTI-SCALE FREQUENCY DOMAIN PERCEPTION
Fig.5 shows how causality is achieved in multi-scale frequency domain perception.
14

---

Under review as a conference paper at ICLR 2022
Periodicity
40
30
20
10-
0-
-10
0
200
400
600
800
1000
Frequency Response
Figure 4: The relationship between filter frequency response and the periodicity. Green curve: our proposed windowing function. Light orange dots: mel-scale initialized relationship.
0-
-2
0.2
1.2
Figure 5: Consecutively downsampled waveform has longer time range in the original waveform (top row). So processing downsampled waveform increases the Causality.
A.5 COMPUTATION TIME REPORT
The computation time of all comparing methods is given in Table 5, from which we can see that our proposed SoundSynp framework requires between 2x-3x more computation time than SELDNet, EIN and SoundDet due to its large Transformer-like backbone. However, SoundSynp computation time still within controllable range.
15

---

Under review as a conference paper at ICLR 2022
Table 5: Inference time on Intel(R) Cor)(TM) i9-7920X CPU. The waveform pre-processing time is contained for SELDNet and EIN. Each individual computation time computed by averaging 100 independent inference of 4s sound waveform.
SELDNett
EIN
SoundDet
SoundSynp
s.20
2.20
1.25 s
3.1 s
16


Reviews:
[{"review_id": "izdSNk36KZ3", "reviewer": "Reviewer_TyRd", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The premise of the paper that using a more appropriate Filterbank which learns to choose different time-frequency resolution tradeoffs based on data is indeed interesting. The authors have made a nice effort in figuring out the details and their experimental evaluation. \n2. The ablation study of using the proposed \"Synperiodic Filterbank\" features with existing methods was definitely needed and insightful to show that features do carry more information than the traditional features. \n\nWeakness:\n1. Personally I think using a significantly larger backbone of transformers on the proposed features to show improvement was not a fair comparison. I would recommend the authors to also experiment with the same or similar Transformer architecture using traditional MFCC or LFBE features to clearly understand what brings more value to the table - the model architecture or the feature extractor. From the existing ablation study it feels like the feature extractor offers a modest improvement at best. \n2. It appears that the authors have reported single / best model accuracy numbers and it is not clear how reliable these numbers are - perhaps reporting average metrics  + standard deviation across multiple model runs (with random initialization) would be better ?\n3. Several related works concerning raw waveform based sound detection and separately raw waveform based source localization are missing and hence the related works section does not offer a balanced review of the problem space:\n[1] T. Kim, J. Lee, and J. Nam, “Comparison and analysis of sample CNN architectures for audio classification,” IEEE Journal of Selected Topics in Signal Processing, vol. 13, no. 2, pp.285–297, May 2019. - Review of Raw-audio based architectures using CNN for event classification.\n[2] J. M. Vera-Diaz, D.Pizarro, and J. M. Guarasa, “Towards end-to-end acoustic localization using deep learning: from audio signal to source position coordinates,” CoRR, vol. abs/1807.11094, 2018. Raw audio based single source localization\n[3] H. Sundar, W. Wang, M. Sun and C. Wang, \"Raw Waveform Based End-to-end Deep Convolutional Network for Spatial Localization of Multiple Acoustic Sources,\" IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020, pp. 4642 - 4646. \n4. The paper can benefit greatly from re-writing the motivations and editing. A few examples of unnecessary claims not justified in the experiments and sentences which can benefit from revision are :\na) In the Abstract: \"...Existing methods fail to do so because they either extract hand-engineered features (i.e. STFT, LogMel) that require a great deal of parameter tuning work (i.e. filter length, window size), or propose to learn a single filter bank to process sound waveforms in a single-scale that often leads to a limited time-frequency resolution capability\" - I don't think the existing methods are \"failing\". They may not be performing as well as the model the authors of this paper have. \nb) In multiple places the phrase - \"filter’s length and frequency response are inversely related\" is used. This is not making sense to me. Should this be \"filter’s length and frequency resolution are inversely related\" ?\nc) Sentence needs revision:\n    i) \"Traditional sound features are human-auditory system biased.\"\n   ii) \"Sound source detection has been previously treated as sound structure (Thrun, 2006), sound object(He et al., 2021) and sound event detection and localization (SELD task)\"\n  iii) \"Our motivation is based on the fact that high frequency sound signal can be adequately detected with a shorter filter, enlarging the filter length doesn’t necessarily increase the frequency resolution but inevitably reduces the filter’s time resolution.\" Amenable to wrong interpretation - Are the authors referring to bio-inspired processing where humans are less sensitive resolving between \n  iv) \"The design of this window function is motivated by the mel-scale frequency initialization strategy, in which two neighboring frequencies are more closely spaced in lower frequency part and higher frequency part\" - not clear what this means. \n  v) \"The parameter number is just linear to filter number.\" - needs revision. \nd) In Eq. (5) what is l (ell) ?\n", "summary_of_the_paper": "This paper addresses the problem of sound event detection and localization from multi-channel raw audio waveforms. Essentially, a different audio front end feature extraction scheme - \"Synperiodic Filterbank\" if proposed which can be parameterized and jointly learnt along with a backbone classifier in an end-to end manner. The feature extraction scheme proposed is grounded in principle of uncertainty governing time-frequency resolution. Through ablation studies the authors show that this feature extraction scheme performs better (albeit marginally) compared to some of the traditionally used feature extraction schemes like MFCC, and LFBEs. Subsequently, the authors show that combining the proposed \"Synperiodic Filterbank\" with a significantly large model (~3X times the closest baseline model in terms of No. of trainable parameters) based on the Transformer Architecture offers a dramatic improvement in the results compared to the baseline. ", "summary_of_the_review": "Overall I think the authors address a curious idea. The presentation is not entirely widely accessible and can benefit greatly from revision. The experiments are interesting but not conclusive enough to understand the root cause of improvements. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "RxVAILVb_kN", "reviewer": "Reviewer_sfZq", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Accurately estimating sound sources' temporal location can be used in a wide range of machine cognition tasks, such as inference on the type of environment, self-localization, navigation without visual input, smart-home applications, scene visualization systems, and etc. \nThis manuscript covers a very interesting topic of sound sources' temporal localization. \nThe authors suggest using Synperiodic Filterbank with the exiting methods to outperform the traditional methods. NOTE, they have proposed a new combination of feature sets and new model structures. \nHowever, I have the following few concerns. \n1- The writing of the manuscript requires improvements, there are many ambiguous sections/ paragraphs that require more elaboration. There are many typos and grammatical errors in the manuscript. \n2- It is not clear to me whether the improvement in the performance is mainly due to the Synperiodic filterbank or the Synperiodic filterbank+ new structure, which brings the total param size to 60M! \n3- the experimental section requires more supporting materials. It is not clear how the training/testing is conducted. It would be more helpful if the authors could elaborate more on some implementation details. \n4- The authors in section 3.1 claim their mathematical formulation covers the \"moving spatial motion sp\" as well. However, they have not discussed/supported this claim in the experimental section. \n", "summary_of_the_paper": "I am sorry, by mistake I posted the incomplete review version! \n\nThis manuscript proposes a sound source localization method utilizing \"synperiodic filter\" bank representation as to the input of a deep convolutional neural network. \nAuthors claim that the convolution of the proposed filterbanks with the raw waveform helps to achieve multi-scale perception in the time domain which results in performance improvement. \nThe authors demonstrated that their method could outperform some of the state-of-the-art similar work.  \n", "summary_of_the_review": "This manuscript addresses an ongoing and open problem of sound source localization. I like the main idea in the paper, however, I think this work requires some minor improvements to be in proper shape for this proceedings.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "47NGToJhMvs", "reviewer": "Reviewer_DURm", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Note**: I am not an experienced researcher in the sound source detection. My reviews may miss some published results or technology. \n\n## Strengths\n\n+ The paper is clearly written, with sufficient technical details.\n+ The novelty and motivation is valid and the proposed method achieves state-of-the-art performance.\n\n## Weakness\n\n- Does the synperiodic filterbank consider the physical characteristics of the sound receiver system ($e.g.$, spatially-deployed microphone array)? If the receiver system structure is changed, does the synperiodic filterbank still generalize well?", "summary_of_the_paper": "This submission addresses the sound source detection problem. Compared to previous method, the authors propose a novel synperiodic filerbank compared to syndistance bank, which learns a data-dependent time-frequency resolution map. Experiments show that the proposed method achieves state-of-the-art performance on several datasets. ", "summary_of_the_review": "**Note**: I am not an experienced researcher in the sound source detection. My reviews may miss some published results or technology. \n\nOverall, I think this submission has solid novelty and state-of-the-art performance on the sound source detection task. Thus, I am inclined to accept the submission. However, my comments may not be professional and I would wait for authors response before making the final decision.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2rHTAoPSpb2", "reviewer": "Reviewer_dYuW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Despite interesting presented results with the proposed approach, this paper contains several major flaws.\nFor all the reasons reported below, I cannot recommend to accept this paper in the current form, however I encourage the authors to submit again an improved version of their work.\n\nI) the overall organization of the paper should be improved. First,  it contains a lot of references to previous work for which the link with the present paper is not very clear (cf. Section 2). I recommend to formulate the problem before discussing the existing approaches to explain in what the new proposed method is new and significant. I also suggest to add a formal and clear definition of \"synperiodicity\" that is missing since it is not a standard concept in signal processing theory. This point is important  since it is a possible new contribution claimed by the authors. For the moment, I understand that \"synperiodic\" filter-bank is a reformulation of well-known concepts already rigorously presented in time-frequency analysis books such as \"P. FLANDRIN, Time-frequency/time-scale analysis. Academic press, 1998\". or \"S. MALLAT, A wavelet tour of signal processing. Elsevier, 1999.\"  The authors should explain rigorously what is \"synperiodicity\" and \"syndistance\" with consideration with previously established theoretical results. The handled concepts such as time-frequency resolution are currently too superficially presented in the actual form of the paper.\n\nII)The chosen mathematical notations are not always adequate. For example from Eq. (1), the goal of sound source detection is to estimate the number of sources K with their parameters of the sources S. Here, the link between the representation $\\mathcal P$ and $S$ is not clear. Moreover, source detection and tracking is a problem not directly related to deep neural network that is only a possible approach to address this problem. I suggest to the author the use a standard and general introduction to the problem as presented in previous works to make the link.  Looking in detail, the formal definition of what is $sp_k$, $sl_k$ is not very clear, this is also the case for several parameters implicitly introduced in the paper such as $\\sigma_i$, $\\rho$, etc. The other equations and the following notations are not very rigorous too (eg. Eqs. (2)- (5) ) and don't help to clarify the understanding of the method and to make the link with standards tools such as the Fourier transform. For example, one should guess to what refers the frequency response $\\phi(\\omega_i)$ multiplied by the window function  $w$ that should be a function of time and should be parametrized by a width $\\sigma_i$... In Eq.(5) the window now becomes a function of $\\omega_i$ and $\\rho$ that is no more consistent.\nTo fix that, I suggest to the authors to  only describe in detail their own contribution mainly summarized in the neural network architecture presented in Table 4 and to discuss the changes in comparison to similar deep neural-based approaches with dedicated simulations or theoretical results to motivate their choices. For example, I would be interested in a accurate comparison of the resolution of the proposed method with several existing TF representation such as STFT or CWT using different analysis windows.\n\nIII) Despite good claimed results, the experiments are not sufficiently detailed to be reproduced. What is exactly the evaluation protocol here ? What are the set of training/validation and test ? The computation time ?  Do the authors have reimplemented the compared state-of-the-art methods or did they only reported results from the literature ? Can please provide more details about the implementation of your method, used language and framework, providing implementation considerations and/or with source code ? Can you please explain how Fig. 4 and 5 are computed ? Can you please detail what are the axis with the proper units in all the figures ? Moreover, the authors should explain the evaluation metrics with a fair discussion with the other methods. What is the motivation of the ablation study? This  problem of interpreting the results is related to my Section II) where the authors should explain why their method perform better, discuss several relevant points with a consideration of the time-frequency resolution of the proposed \"synperiodic\" filterbank claimed as the main new contribution.\n\nIV) The paper could also benefit of a major rewriting and English spell-check to fix typos and grammatical errors.", "summary_of_the_paper": "This paper proposes a method for improving the detection and the localization of several sound sources using a 4-channel signal (The addressed configuration, determined or under-determined case is not detailed in this paper).\nThe proposed approach is based on a so-called \"synperiodic\" filter bank representation that is used as the input of a deep convolutional neural network. Finally method proposed by the authors is comparatively evaluated on the DCASE2020 dataset and pretends to provide the best results.\n\n", "summary_of_the_review": "The authors address a very interesting topic considering the computation of the suitable signal representation as the input of a deep neural network to address the problem of estimating the signal parameters of interest.\nDespite the presented \"good\" results, the experiments are not reproducible since relevant information related to the protocol and the implementation are missing. Moreover, the theoretical concepts introduced in this study to describe the main contributions from the authors  are too superficially presented. \nHence, I recommend to reject this paper in present state but I strongly encourage the authors to resubmit a more clear and more transparent study (for example to DCASE 22) to introduce their possibly novel approach.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work studies the task of sound source localization from multi-channel audio. An approach to design a wavelet-like filter bank for audio feature extraction is proposed.

After discussion, all reviewers have given this work borderline ratings. Concerns were raised about the quality of the writing, missing related work, experimental methodology, and especially with regards to confounding factors in the experimental results, which make it difficult to assess the individual merit of the different components in the proposed system (i.e. features vs. transformer model). The authors have addressed this to some extent with additional experiments in the updated version of the manuscript, but this revealed that the gains obtained by the proposed feature extraction method in isolation are actually quite modest. This is in contrast with how the paper is written, with much more emphasis on this particular contribution than seems to be warranted by the empirical results.

Additionally, I believe the manuscript would benefit from a more careful and thorough revision to improve clarity and accessibility, beyond what is possible within a single review cycle. Therefore I am recommending rejection.
Decision: Reject