Question: Does the synperiodic filterbank consider the physical characteristics of sound receiver system?
Evidence: The fundamental task for an agent to perceive and interact with the 3D environment is to know the location and semantic identity of its nearby objects. The location includes spatial location that is either stationary or moving, temporal location like start time and end time. Vision-based such environment perception has received large attention in the past decade and we have witnessed huge progress in tasks such as object detection, classification and tracking. Nevertheless, the sound-based counterpart research has far lagged behind, despite all the fascinating properties sound signal exhibits. For example, sound is ubiquitous and insensitive to ambient illumination change, it has no field-of-view (FoV) constraints and is capable of circumventing physical barriers to perceive scene beyond line-of-sight. As a complementary sensing approach to vision, sound-based environment perception is of vital importance for acoustic scene understanding. A typical example is the sound source detection (answer where is it, when does it happen and what is it), given the recorded multi-channel sound waveforms.
To detect sound sources, we often deploy a spatially-configured microphone array to record an acoustic environment. Unlike camera or LiDAR scanner that directly captures RGB image or measures the range distance, sound waveform itself is a highly compressed one-dimensional points with high sampling rate, all sound source signals are compressed and mixed into the one-dimensional data format. Since different sound sources have different frequency property, it is essential convert waveform into time-frequency representation so that frequencies hidden in the waveform are explicitly split out. This is often achieved by projecting the raw waveform into various orthogonal frequency basis. At the same time, a sound source's spatial location clue lies in inter-channel difference of recorded waveforms (i.e. phase difference in this work). It is essential to design a neural network that jointly encodes mono-channel time-frequency representation and inter-channel phase difference from the raw waveforms in a unified, parameter-frugal and computation-efficient manner. The learned representation should have elegant resolution in both time, frequency and space domain so that sound sources can be precisely detected.
However, learning such representation is a tough task. Challenges derive from both theoretical side and practical side. According to Uncertainty Principle, we cannot achieve the optimal resolution in time and frequency domain at the same time, but instead keep a trade-off between them. Traditional hand-engineered sound feature and some recently proposed learnable filter bank empirically set the same length for all filters, resulting in human-biased, unadjustable time-frequency resolution map. Some other work correlates filter frequency response and filter length by initializing in mel-scale, but it is not scalable nor stable because the final initialization depends on the filter number. Moreover, all existing methods process raw waveform with one-scale filter bank, we argue that the one-scale sound perception easily leads to incomprehensive sound sources sensing, especially when sound sources have different frequency property or undergo various spatial motion.
In this paper, we first give comprehensive theoretical analysis on the filter bank impact on its extracted feature's time and frequency resolution. Based on the analysis, we propose a simple yet effective synperiodic filter bank construction strategy in which each filter's frequency response and length are correlated by rotating periodicity such that each filter's length is inversely proportional to its frequency resolution. The synperiodic filter bank thus internally maintains a better time-frequency resolution trade-off than traditional fixed-length filter bank. Coupling the filter length with its frequency response helps us to reduce human intervention in filter bank design. By simply alternating the periodicity term, we further construct a group of synperiodic filter banks, with which we achieve multi-scale perception in time domain. At the same time, by applying a synperiodic filter bank to process one raw waveform as well as its consecutively-downsampled versions, we achieve multi-scale perception in frequency domain. The multi-scale perception in both time and frequency domain of synperiodic filter bank enables the neural network to dynamically learn better representation for sound source detection in a data-driven way. It is worth noting that synperiodic filter bank parameter number is just linear to filter number (adds up to less than 1% of the whole parameters) and it can be efficiently implemented as a 1D convolution operator.
Following the aforementioned learnable front-end, we add a Transformer-like backbone network with two paralleling branches with intermediate soft-parameter sharing to learn sound source's semantic and spatial location related representation both jointly and separately. Experiment on both direction-of-arrival (DoA) task and physical location estimation task shows that our proposed framework outperforms comparing methods by a large margin. Replacing existing method's head with our proposed synperiodic filter bank also improvdes the performance.
We use a spatially-deployed microphone array, like four closely-bounded microphones in a circular coplanar configuration, to record an acoustic environment where various sound sources undergo independent and unconstrained spatial motion. Recorded multi-channel raw waveforms are represented as W " tx i ptqu 4
i"1 , each waveform is 1-d vector of size T sampled at a fixed sampling rate. We denote K sound sources by S " ts k " pt k s , t k e , sp k , sl k qu K k"1 , one single sound source corresponds to a start time t s , end time t e , stationary or moving spatial motion sp and a semantic identity label sl. Our goal is to learn a representation P that is representative enough to detect sound sources both spatially and semantically. P is a temporal framewise representation of shape T 1 ˆN . We propose to learn P directly from the raw waveforms via a deep neural network F parameterized by θ,
A learned elegant representation should have large resolution in both time, frequency and space domain and exhibit strong capability at addressing three intrinsic challenges: Polyphonicity, sound sources are mutually independent, they can freely overlap temporally; Time-scale variance, the Figure: Synperiodic filter bank illustration: Syndistance filter bank (green color) rotates the same distance in complex plane and thus has the same kernel length, regardless of frequency it carries. Its time-frequency dynamic resolution map is thus rectangular. our proposed synperiodic filter bank (blue color) are generated by rotating the same periodicity number. So filter carrying lower frequency has larger kernel size than those with higher frequency response. As a result, synperiodic filter bank's time-frequency dynamic resolution map that is able to achieve a better trade-off than traditional syndistance filter bank.
temporal duration of different sound sources varies enormously, from seconds to minutes long; Datasize, sound waveform is sampled with large sampling rate, resulting in large data size.
Sound source presence clue lies in mono-channel time-frequency representation and its spatial motion clue lies in inter-channel interaction, like phase difference. Our proposed synperiodic filter bank group jointly encodes time-frequency representation and phase difference from raw waveforms in a unified way. More detail about phase difference encoding is given in the Appendix A.1.
We use the previously constructed synperiodic filter bank group to convolve with mono-channel sound waveform with the same step size and padding strategy, resulting in the same size output for each single synperiodic filter bank. Since different filter group has different window size, we naturally achieve multi-scale learning in time domain. It maximally avoids us empirically selecting one window scale ρ which might not be optimal, but instead uses a group of filter bank to enforce the neural network to strike a better time-frequency resolution trade-off in a data-driven way.
We further propose a strategy to enable multi-scale learning in frequency domain. The strategy is hierarchical: given a raw sound waveform with sampling frequency F S , synperiodic filter bank's frequency is initialized within the range r0, F S 2 s under Nyquist sampling theorem. If we downsample the sound waveform by a factor of 2, the resulting waveform can be processed by the lower-half Figure: Multi-scale learning in frequency domain. Given the raw one channel sound waveform and pre-constructed synperiodic filter bank, we consecutively downsample the waveform by factor 2x, the newly downsampled waveform is processed by low-half filter bank from the proceeding filter bank. We can obtain time-frequency representation for on each frequency scale. These timefrequency representations share the same time length by adjusting step size. The final time-frequency representation is obtained by max-pooling them together. filters in each group whose frequency response lies in r0, F S 4 s. This process that 2x-downsampling the waveform further process the downsampled waveform with filters with lower-half frequency response can potentially iterate a couple of times (in our case three times), resulting in multi-scale perception in frequency domain. The figurative illustration is shown by Fig.. Multi-scale learning in frequency domain brings us two extra benefits: 1) from data augmentation perspective, hierarchical 2x downsampling inevitably creates more waveforms, which matters a lot when labelling audio data is technically hard and time-consuming. 2) from the causality perspective, the adjacent 2xdownsampling strategy leads to dilated convolution for lower-frequency filters, because applying a filter to convolve with a downsampled waveform equals to convolve on the original waveform with dilated convolution (skip-2 connection). The resulting wider field of view (FoV) for lower frequency filters enables to learn better sound causality along the time axis (see Appendix A.4). In sum, by using learnable synperiodic filter bank group to process the raw waveform in multi-scale manner, we achieve a dynamic time-frequency resolution map that naturally maintains a better time-frequency resolution map fitting for sound source detection in a data-driven way.
Computational Analysis Synperiodic filter bank group introduces very few parameters (less than 1%) because they are parameterized filters. The trainable parameter number increases linearly w.r.t. synperiodic filter bank number. Their convolution with raw waveforms can also be efficiently implemented with 1D convolution.
Since different sound sources have different frequency property, it is essential convert waveform into time-frequency representation so that frequencies hidden in the waveform are explicitly split out. This is often achieved by projecting the raw waveform into various orthogonal frequency basis. At the same time, a sound source’s spatial location clue lies in inter-channel difference of recorded waveforms (i.e. phase difference in this work). It is essential to design a neural network that jointly encodes mono-channel time-frequency representation and inter-channel phase difference from the raw waveforms in a unified, parameter-frugal and computation-efficient manner.****Sound source presence clue lies in mono-channel time-frequency representation and its spatial motion clue lies in inter-channel interaction, like phase difference. Our proposed synperiodic filter bank group jointly encodes time-frequency representation and phase difference from raw waveforms in a unified way. More detail about phase difference encoding is given in the Appendix A.1.****We use the previously constructed synperiodic filter bank group to convolve with mono-channel sound waveform with the same step size and padding strategy, resulting in the same size output for each single synperiodic filter bank.
Answer: Our proposed synperiodic filterbank is designed to process mono-channel waveform. For inter-channel interaction, we encode the phase difference on the time-frequency representation extracted by synperiodic filterbank. The detailed discussion is presented in Appendix A.1. Since we use inter-channel phase difference as sound source spatial location clue, we require the four microphones are configured close to each other (e.g. 4 cm distant with each other) because there are other inter-channel information to be considered like amplitude difference and energy decay if microphone array is distantly-configured (e.g. put microphones at a large house's different corners).