Question: how reliable  are these numbers?
Evidence: We adopt two evaluation metrics: segment-based and event-based metric. Segment-based metric is a widely adopted evaluation metric, it couples semantic label and spatial location together: a semantic-correctly detected sound source needs to be spatially close enough to its ground truth location in order to be regarded as a true positive detection. Event-based based metric is newly proposed by to comprehensively evaluate under different confidence scores. Like object detection from images, it computes mean average precision (mAP) and mean average recall (mAR) score.
We call our framework SoundSynp and compare it with four most recent methods: SELD-Net, EIN, SoundDet and Utsc-Iflytek. SELDNet is the baseline model and it jointly trains sound source's semantic label and spatial location with a convolutional recurrent neural network (CRNN). EIN) is a very recent work. It adopts multi-heads self-attention to model temporal dependency and trackwise permutation-invariant training to train the model. SoundDet directly learns from raw waveform with MaxCorr kernels, followed by an encoder-decoder neural network. Utsc-Iflytek is ranked first in DECASE2020 challenge leaderboard 1 , it combines MIC and FOA features and ensembles different models like ResNet and Xception(Chollet, 2017) to give final prediction.
The network architecture is shown in table in Appendix material. To train the neural network, we evenly divide the one minute long four-channel raw waveform into non-overlapping 4s short snippets. The raw waveform is first normalized to r´1, 1s before feeding the neural network. We adopt Adam optimizer with an initial learning rate 0.0002 in the first 100 epochs and 0.00007 in the following 50 epochs. Batchsize is 16. The loss combination weight between classification head and regression head is 1 : 2. During training, data augmentation method SpecAugment is applied. For DoA task, we regress direction of arrival angle in Cartesian coordinates rx, y, zs. In synperiodic filter bank groups, the filter length is 1025, each group's filter number is 256 and the step size is 600. Particularly, we have observed the initialized learnable synperiodic filter banks update its parameters intensively during the very early several epochs, and then gradually becomes stable. The final learned parameters are close to their initialization, one group's learned parameters (like frequency response and filter window length) is different from those learned by other groups, although they are initialized as the same in the start. We train each model five times independently and report the average score. The standard deviation is within 0.04 (for recall) and 0.2 ˝for angle, 0.003 for mAP and mAR.
The result is given in Table1, from which we see that SoundSynp achieves the best performance over all comparing methods by a large margin, under both segment-based and event-based metrics. Both SELDNet, EIN and UTSC-Iflytek use pre-extracted hand-engineered sound features, such as Logmel, GCC-Phat and Intensity Vector. SoundDet and SoundSynp are the only two methods that directly learn from raw waveforms. At the same time, SoundSynp obtains better performance on FOA than MIC format, the same phenomena has been observed by all other methods. It thus shows FOA better fits for sound source detection than MIC. It is worth noting that Utsc-Iflytek ensembles different powerful image-based 2D models to detect sound sources. However, our proposed SoundSynp still outperforms Utsc-Iflytek by a large margin. We don't report of mAP/mAR value for Utsc-Iflytek because it is a complex system and no detail about their system is available.
Ablation Study To disentangle the individual contribution of each part of our SoundSynp framework to the whole performance improvement, we conduct three ablation studies. First, the individual contribution of Synperiodic filter bank. We replace hand-engineered sound feature head of SELDNet, EIN and learnable MaxCorr filter bank with our proposed Synperiodic filter bank to test their corresponding performance. It helps to remove the influence of the backbone neural network of different models and thus helps to get direct comparison of synperiodic filter bank with other front-end. Second, we replace SoundSynp's synperiodic filter bank group with widely-used MFCC and Log-Mel that is used by and EIN, respectively. It disentangles synperiodic filter bank group with Transformer-like backbone, thus helps to figure out if the performance gain is simply brought by backbone network.
Third, internally, we test five synperiodic variants: (1) synperiodic filter bank with just multiscale perception in frequency domain (SoundSynp MSFreq), (2) just multi-scale perception in time domain (SoundSynp MSTime), (3) Synperiodic filter bank with frequency responses linearly initialized in Nyquist frequency range (SoundSynp Linear, compare with our mel-scale initialization), (4) just one synperiodic filter bank without multi-scale perception neither in time nor frequency domain (SoundSynp SingleScale). () synperiodic filter bank with rectangular band-pass frequency response initialization (SoundSynp Sinc), like SincNet does. The internal comparison helps us to figure out the necessity of each part of synperiodic filter bank design.
The ablation study result is shown in Table. We can observe that: First, using synperiodic filter bank as a replacement of existing filter bank can help to improve the corresponding performance. Second, replacing SoundSynp's synperiodic filter bank with classic hand-engineered features inevitably reduces the performance under all evaluation metrics, its thus shows learning from pre-extracted fixed and single-scale sound representation leads to inferior performance than our proposed multiscale synperiodic filter bank. Third, the absence of multi-scale perception in either frequency domain or time domain inevitably reduces the performance. We find sound source semantic label detection suffers more in single-scale perception in time domain than in frequency domain (see better performance on ER 20 ˝, and F 20 ˝score), which shows frequency domain multi-scale perception is vital for semantic label estimation. Similarly, we can observe that multi-scale perception in time domain is vital for sound source spatial location estimation (see better performance on LE and LR score). Linearly initialized filter bank frequency response reduces the performance, which shows assigning more filters to the lower frequency range is important. But this conclusion might be data-dependent because we find DCASE dataset contains many low-frequency sounds like burning fire and footsteps. Moreover, reducing the synperiodic filter bank group to one groups with just singlescale perception leads to comparably the worst performance, it thus shows multi-scale perception in both time and frequency domain is essential for DoA-based sound source detection. Lastly, SoundSynp Sinc leads to slightly inferior performance than our used mel-scale initialization strategy, it shows our proposed synperiodic filter bank is a general filter bank that can be adopted to other frequency sensitive filter bank.  One qualitative comparison is shown in Fig.. We can clearly see that SELDNet generates mixed prediction at different time steps and DoA locations. SoundDet and EIN give non-existing sound sources (orange color). When multiple sound sources happen at the same time (polyphonicity), SoundDet and EIN are easily failed to predict the right spatial location (discretized blue and red color). Our method predicts more spatially and temporally consistent sound sources by maximally keeping sound source's continuity and consistency.
We train each model five times independently and report the average score. The standard deviation is within 0.04 (for recall) and 0.2 ˝ for angle, 0.003 for mAP and mAR.
Answer: In our experiment, we train each model five times independently and report the averaging performance score. We did not report the standard deviation because many comparing methods did not report it. In our revised version, we report the average standard deviation.