Question: Whether the improvement is mainly due to Synperiodic filter bank alone or Synperiodic filter bank +  new structure?
Evidence: We adopt two evaluation metrics: segment-based and event-based metric. Segment-based metric is a widely adopted evaluation metric, it couples semantic label and spatial location together: a semantic-correctly detected sound source needs to be spatially close enough to its ground truth location in order to be regarded as a true positive detection. Event-based based metric is newly proposed by to comprehensively evaluate under different confidence scores. Like object detection from images, it computes mean average precision (mAP) and mean average recall (mAR) score.
We call our framework SoundSynp and compare it with four most recent methods: SELD-Net, EIN, SoundDet and Utsc-Iflytek. SELDNet is the baseline model and it jointly trains sound source's semantic label and spatial location with a convolutional recurrent neural network (CRNN). EIN) is a very recent work. It adopts multi-heads self-attention to model temporal dependency and trackwise permutation-invariant training to train the model. SoundDet directly learns from raw waveform with MaxCorr kernels, followed by an encoder-decoder neural network. Utsc-Iflytek is ranked first in DECASE2020 challenge leaderboard 1 , it combines MIC and FOA features and ensembles different models like ResNet and Xception(Chollet, 2017) to give final prediction.
The network architecture is shown in table in Appendix material. To train the neural network, we evenly divide the one minute long four-channel raw waveform into non-overlapping 4s short snippets. The raw waveform is first normalized to r´1, 1s before feeding the neural network. We adopt Adam optimizer with an initial learning rate 0.0002 in the first 100 epochs and 0.00007 in the following 50 epochs. Batchsize is 16. The loss combination weight between classification head and regression head is 1 : 2. During training, data augmentation method SpecAugment is applied. For DoA task, we regress direction of arrival angle in Cartesian coordinates rx, y, zs. In synperiodic filter bank groups, the filter length is 1025, each group's filter number is 256 and the step size is 600. Particularly, we have observed the initialized learnable synperiodic filter banks update its parameters intensively during the very early several epochs, and then gradually becomes stable. The final learned parameters are close to their initialization, one group's learned parameters (like frequency response and filter window length) is different from those learned by other groups, although they are initialized as the same in the start. We train each model five times independently and report the average score. The standard deviation is within 0.04 (for recall) and 0.2 ˝for angle, 0.003 for mAP and mAR.
The result is given in Table1, from which we see that SoundSynp achieves the best performance over all comparing methods by a large margin, under both segment-based and event-based metrics. Both SELDNet, EIN and UTSC-Iflytek use pre-extracted hand-engineered sound features, such as Logmel, GCC-Phat and Intensity Vector. SoundDet and SoundSynp are the only two methods that directly learn from raw waveforms. At the same time, SoundSynp obtains better performance on FOA than MIC format, the same phenomena has been observed by all other methods. It thus shows FOA better fits for sound source detection than MIC. It is worth noting that Utsc-Iflytek ensembles different powerful image-based 2D models to detect sound sources. However, our proposed SoundSynp still outperforms Utsc-Iflytek by a large margin. We don't report of mAP/mAR value for Utsc-Iflytek because it is a complex system and no detail about their system is available.
Ablation Study To disentangle the individual contribution of each part of our SoundSynp framework to the whole performance improvement, we conduct three ablation studies. First, the individual contribution of Synperiodic filter bank. We replace hand-engineered sound feature head of SELDNet, EIN and learnable MaxCorr filter bank with our proposed Synperiodic filter bank to test their corresponding performance. It helps to remove the influence of the backbone neural network of different models and thus helps to get direct comparison of synperiodic filter bank with other front-end. Second, we replace SoundSynp's synperiodic filter bank group with widely-used MFCC and Log-Mel that is used by and EIN, respectively. It disentangles synperiodic filter bank group with Transformer-like backbone, thus helps to figure out if the performance gain is simply brought by backbone network.
Third, internally, we test five synperiodic variants: (1) synperiodic filter bank with just multiscale perception in frequency domain (SoundSynp MSFreq), (2) just multi-scale perception in time domain (SoundSynp MSTime), (3) Synperiodic filter bank with frequency responses linearly initialized in Nyquist frequency range (SoundSynp Linear, compare with our mel-scale initialization), (4) just one synperiodic filter bank without multi-scale perception neither in time nor frequency domain (SoundSynp SingleScale). () synperiodic filter bank with rectangular band-pass frequency response initialization (SoundSynp Sinc), like SincNet does. The internal comparison helps us to figure out the necessity of each part of synperiodic filter bank design.
The ablation study result is shown in Table. We can observe that: First, using synperiodic filter bank as a replacement of existing filter bank can help to improve the corresponding performance. Second, replacing SoundSynp's synperiodic filter bank with classic hand-engineered features inevitably reduces the performance under all evaluation metrics, its thus shows learning from pre-extracted fixed and single-scale sound representation leads to inferior performance than our proposed multiscale synperiodic filter bank. Third, the absence of multi-scale perception in either frequency domain or time domain inevitably reduces the performance. We find sound source semantic label detection suffers more in single-scale perception in time domain than in frequency domain (see better performance on ER 20 ˝, and F 20 ˝score), which shows frequency domain multi-scale perception is vital for semantic label estimation. Similarly, we can observe that multi-scale perception in time domain is vital for sound source spatial location estimation (see better performance on LE and LR score). Linearly initialized filter bank frequency response reduces the performance, which shows assigning more filters to the lower frequency range is important. But this conclusion might be data-dependent because we find DCASE dataset contains many low-frequency sounds like burning fire and footsteps. Moreover, reducing the synperiodic filter bank group to one groups with just singlescale perception leads to comparably the worst performance, it thus shows multi-scale perception in both time and frequency domain is essential for DoA-based sound source detection. Lastly, SoundSynp Sinc leads to slightly inferior performance than our used mel-scale initialization strategy, it shows our proposed synperiodic filter bank is a general filter bank that can be adopted to other frequency sensitive filter bank.  One qualitative comparison is shown in Fig.. We can clearly see that SELDNet generates mixed prediction at different time steps and DoA locations. SoundDet and EIN give non-existing sound sources (orange color). When multiple sound sources happen at the same time (polyphonicity), SoundDet and EIN are easily failed to predict the right spatial location (discretized blue and red color). Our method predicts more spatially and temporally consistent sound sources by maximally keeping sound source's continuity and consistency.
We adopt two evaluation metrics: segment-based and event-based metric. Segment-based metric is a widely adopted evaluation metric, it couples semantic label and spatial location together: a semantic-correctly detected sound source needs to be spatially close enough to its ground truth location in order to be regarded as a true positive detection. Event-based based metric is newly proposed by to comprehensively evaluate under different confidence scores. Like object detection from images, it computes mean average precision (mAP) and mean average recall (mAR) score.
We call our framework SoundSynp and compare it with four most recent methods: SELD-Net, EIN, SoundDet and Utsc-Iflytek. SELDNet is the baseline model and it jointly trains sound source's semantic label and spatial location with a convolutional recurrent neural network (CRNN). EIN) is a very recent work. It adopts multi-heads self-attention to model temporal dependency and trackwise permutation-invariant training to train the model. SoundDet directly learns from raw waveform with MaxCorr kernels, followed by an encoder-decoder neural network. Utsc-Iflytek is ranked first in DECASE2020 challenge leaderboard 1 , it combines MIC and FOA features and ensembles different models like ResNet and Xception(Chollet, 2017) to give final prediction.
The network architecture is shown in table in Appendix material. To train the neural network, we evenly divide the one minute long four-channel raw waveform into non-overlapping 4s short snippets. The raw waveform is first normalized to r´1, 1s before feeding the neural network. We adopt Adam optimizer with an initial learning rate 0.0002 in the first 100 epochs and 0.00007 in the following 50 epochs. Batchsize is 16. The loss combination weight between classification head and regression head is 1 : 2. During training, data augmentation method SpecAugment is applied. For DoA task, we regress direction of arrival angle in Cartesian coordinates rx, y, zs. In synperiodic filter bank groups, the filter length is 1025, each group's filter number is 256 and the step size is 600. Particularly, we have observed the initialized learnable synperiodic filter banks update its parameters intensively during the very early several epochs, and then gradually becomes stable. The final learned parameters are close to their initialization, one group's learned parameters (like frequency response and filter window length) is different from those learned by other groups, although they are initialized as the same in the start. We train each model five times independently and report the average score. The standard deviation is within 0.04 (for recall) and 0.2 ˝for angle, 0.003 for mAP and mAR.
The result is given in Table1, from which we see that SoundSynp achieves the best performance over all comparing methods by a large margin, under both segment-based and event-based metrics. Both SELDNet, EIN and UTSC-Iflytek use pre-extracted hand-engineered sound features, such as Logmel, GCC-Phat and Intensity Vector. SoundDet and SoundSynp are the only two methods that directly learn from raw waveforms. At the same time, SoundSynp obtains better performance on FOA than MIC format, the same phenomena has been observed by all other methods. It thus shows FOA better fits for sound source detection than MIC. It is worth noting that Utsc-Iflytek ensembles different powerful image-based 2D models to detect sound sources. However, our proposed SoundSynp still outperforms Utsc-Iflytek by a large margin. We don't report of mAP/mAR value for Utsc-Iflytek because it is a complex system and no detail about their system is available.
Ablation Study To disentangle the individual contribution of each part of our SoundSynp framework to the whole performance improvement, we conduct three ablation studies. First, the individual contribution of Synperiodic filter bank. We replace hand-engineered sound feature head of SELDNet, EIN and learnable MaxCorr filter bank with our proposed Synperiodic filter bank to test their corresponding performance. It helps to remove the influence of the backbone neural network of different models and thus helps to get direct comparison of synperiodic filter bank with other front-end. Second, we replace SoundSynp's synperiodic filter bank group with widely-used MFCC and Log-Mel that is used by and EIN, respectively. It disentangles synperiodic filter bank group with Transformer-like backbone, thus helps to figure out if the performance gain is simply brought by backbone network.
Third, internally, we test five synperiodic variants: (1) synperiodic filter bank with just multiscale perception in frequency domain (SoundSynp MSFreq), (2) just multi-scale perception in time domain (SoundSynp MSTime), (3) Synperiodic filter bank with frequency responses linearly initialized in Nyquist frequency range (SoundSynp Linear, compare with our mel-scale initialization), (4) just one synperiodic filter bank without multi-scale perception neither in time nor frequency domain (SoundSynp SingleScale). () synperiodic filter bank with rectangular band-pass frequency response initialization (SoundSynp Sinc), like SincNet does. The internal comparison helps us to figure out the necessity of each part of synperiodic filter bank design.
The ablation study result is shown in Table. We can observe that: First, using synperiodic filter bank as a replacement of existing filter bank can help to improve the corresponding performance. Second, replacing SoundSynp's synperiodic filter bank with classic hand-engineered features inevitably reduces the performance under all evaluation metrics, its thus shows learning from pre-extracted fixed and single-scale sound representation leads to inferior performance than our proposed multiscale synperiodic filter bank. Third, the absence of multi-scale perception in either frequency domain or time domain inevitably reduces the performance. We find sound source semantic label detection suffers more in single-scale perception in time domain than in frequency domain (see better performance on ER 20 ˝, and F 20 ˝score), which shows frequency domain multi-scale perception is vital for semantic label estimation. Similarly, we can observe that multi-scale perception in time domain is vital for sound source spatial location estimation (see better performance on LE and LR score). Linearly initialized filter bank frequency response reduces the performance, which shows assigning more filters to the lower frequency range is important. But this conclusion might be data-dependent because we find DCASE dataset contains many low-frequency sounds like burning fire and footsteps. Moreover, reducing the synperiodic filter bank group to one groups with just singlescale perception leads to comparably the worst performance, it thus shows multi-scale perception in both time and frequency domain is essential for DoA-based sound source detection. Lastly, SoundSynp Sinc leads to slightly inferior performance than our used mel-scale initialization strategy, it shows our proposed synperiodic filter bank is a general filter bank that can be adopted to other frequency sensitive filter bank.  One qualitative comparison is shown in Fig.. We can clearly see that SELDNet generates mixed prediction at different time steps and DoA locations. SoundDet and EIN give non-existing sound sources (orange color). When multiple sound sources happen at the same time (polyphonicity), SoundDet and EIN are easily failed to predict the right spatial location (discretized blue and red color). Our method predicts more spatially and temporally consistent sound sources by maximally keeping sound source's continuity and consistency.
We use Pyroomacoustics simulator to simulate a shoebox like room with r7m, 5m, 3ms size. Four microphones (24kHz sampling rate) are configured in a plane with 5cm mutual distance and put in the room center (r3.5m, 2.5m, 1.5ms). We choose three commonly heard sounds from BBC sound effect site 2 : cat meowing, baby talking and dog barking. All these seed sounds last less than 10s. During simulation, we put all seed sound in the plane z " 1.5m (we constrain to a plane because we find relaxing z easily leads EIN model to get stuck in a local minima like r´1, ´1, ´1s), but with x randomly in r2m, 5ms and y randomly in r1m, 4ms, up to two sounds can happen at the same time. In sum, we have simulated 500 one-minute recordings (400 for train, 100 for test), with an average of four sounds recorded per recording. For physical location prediction, we regress the angle and range. The result is in Table. We do not report SELDNet result as we find it doesn't converge during train. We can see from the table that SoundSynp produces the best performance than over EIN and SoundDet network. Replacing their front-end with synperiodic filter bank also improves the performance. It thus shows our proposed SoundSynp is capable of learning representation for physical-location based sound source detection.
Table 2: Ablation study report. Internally, we replace SELDNet, EIN and SoundDet head with
synperiodic filter bank (denoted as Synp). Externally, we test different modified SoundSynp versions.****Third, internally, we test five synperiodic variants: (1) synperiodic filter bank with just multiscale perception in frequency domain (SoundSynp MSFreq), (2) just multi-scale perception in time domain (SoundSynp MSTime), (3) Synperiodic filter bank with frequency responses linearly initialized in Nyquist frequency range (SoundSynp Linear, compare with our mel-scale initialization), (4) just one synperiodic filter bank without multi-scale perception neither in time nor frequency domain (SoundSynp SingleScale). (5) synperiodic filter bank with rectangular band-pass frequency response initialization (SoundSynp Sinc), like SincNet(Ravanelli & Bengio, 2018) does. The internal comparison helps us to figure out the necessity of each part of synperiodic filter bank design.****Replacing their front-end with synperiodic filter bank also improves the performance. It thus shows our proposed SoundSynp is capable of learning representation for physical-location based sound source detection.****The computation time of all comparing methods is given in Table 5, from which we can see that our proposed SoundSynp framework requires between 2x-3x more computation time than SELDNet, EIN and SoundDet due to its large Transformer-like backbone. However, SoundSynp computation time is still within a controllable range.
Answer: In our first version submission, we have conducted two ablation studies (shown in Table 2) 1. We replace SELDNet, EIN and SoundDet front-end head with our proposed synperiodic filter bank. Experimental result shows that introducing synperiodic filter bank to existing approaches helps to improve their corresponding performance. It thus shows our proposed synperiodic filter bank can be used as a plug-and-play front-end by existing approaches to improve their performance on sound source detection task, without introducing explicit extra parameters and computational burden. 2. Moreover, we test five variants of synperiodic filter bank. For example, we find out that reducing the multi-scale perception strategy of synperiodic filter bank to single-scale perception in either time domain or frequency domain reduces the performance. This internal synperiodic filter bank variants ablation study shows the necessity of each part of synperiodic filter bank design. To disentangle the synperiodic filter bank front-end and Transformer-like backbone, in our revised paper, we have added another ablation study (also given in Table 2), in which we replace our proposed SoundSynp's synperiodic filter bank with traditional hand-engineered features including MFCC and LogMel. The corresponding experimental result shows that using traditional sound features leads to inferior performance. In sum, we can conclude from the three ablation studies that: *  The performance improvement is due to the joint contribution of synperiodic filter bank and Transformer-like backbone network structure. *  Removing either Transformer-like backbone network (ablation study 1) or synperiodic filter bank (ablation study 2) inevitably reduces the performance. * Transformer-like backbone network and synperiodic filter bank can be disentangled to solve other sound related tasks separately.