## 6. REFERENCES

[1] Y. Koizumi, S. Saito, H. Uematsu, and N. Harada, "Optimizing acoustic feature extractor for anomalous sound detection based on Neyman-Pearson lemma," in Proc. 25th European Signal Processing Conference (EUSIPCO), 2017, pp. 698-702.

[2] Y. Kawaguchi and T. Endo, "How can we detect anomalies from sub-sampled audio signals?" in Proc. 27th IEEE International Workshop on Machine Learning for Signal Processing (MLSP), 2017.

[3] Y. Koizumi, S. Saito, H. Uematsu, Y. Kawachi, and N. Harada, "Unsupervised detection of anomalous sound based on deep learning and the Neyman-Pearson lemma," IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 27, no. 1, pp. 212-224, Jan. 2019.

[4] Y. Kawaguchi, R. Tanabe, T. Endo, K. Ichige, and K. Hamada, "Anomaly detection based on an ensemble of dereverberation and anomalous sound extraction," in Proc. 44th IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2019, pp. 865-869.

[5] Y. Koizumi, S. Saito, M. Yamaguchi, S. Murata, and N. Harada, "Batch uniformization for minimizing maximum anomaly score of DNN-based anomaly detection in sounds," in Proc. IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WAS-PAA), 2019, pp. 6-10.

[6] K. Suefusa, T. Nishida, H. Purohit, R. Tanabe, T. Endo, and Y. Kawaguchi, "Anomalous sound detection based on interpolation deep neural network," in Proc. 45th IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020, pp. 271-275.

[7] H. Purohit, R. Tanabe, T. Endo, K. Suefusa, Y. Nikaido, and Y. Kawaguchi, "Deep autoencoding GMM-based unsupervised anomaly detection in acoustic signals and its hyper-parameter optimization," in Proc. 5th Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE), 2020, pp. 175-179.

[8] Y. Koizumi, Y. Kawaguchi, K. Imoto, T. Nakamura, Y. Nikaido, R. Tanabe, H. Purohit, K. Suefusa, T. Endo, M. Yasuda, and N. Harada, "Description and discussion on DCASE2020 challenge task2: Unsupervised anomalous sound detection for machine condition monitoring," in Proc. 5th Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE), 2020, pp. 81-85.

[9] Y. Kawaguchi, K. Imoto, Y. Koizumi, N. Harada, D. Niizumi, K. Dohi, R. Tanabe, H. Purohit, and T. Endo, "Description and discussion on DCASE 2021 challenge task 2: Unsupervised anomalous detection for machine condition monitoring under domain shifted conditions," in Proc. 6th Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE), 2021, pp. 186-190.

[10] N. Harada, D. Niizumi, D. Takeuchi, Y. Ohishi, M. Yasuda, and S. Saito, "Toyadmos2: Another dataset of miniature-machine operating sounds for anomalous sound detection under domain shift conditions," in Proc. 6th Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE), 2021, pp. 1-5.

[11] K. Dohi, T. Nishida, H. Purohit, R. Tanabe, T. Endo, M. Yamamoto, Y. Nikaido, and Y. Kawaguchi, "MIMII DG: Sound dataset for malfunctioning industrial machine investigation and inspection for domain generalization task," arXiv preprint arXiv:2205.13879, 2022.

[12] R. Giri, S. V. Tenneti, F. Cheng, K. Helwani, U. Isik, and A. Krishnaswamy, "Self-supervised classification for detecting anomalous sounds," in Proc. 5th Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE), 2020, pp. 46-50.

[13] P. Primus, V. Haunschmid, P. Praher, and G. Widmer, "Anomalous sound detection as a simple binary classification problem with careful selection of proxy outlier examples," in Proc. 5th Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE), 2020, pp. 170-174.

[14] T. Inoue, P. Vinayavekhin, S. Morikuni, S. Wang, T. H. Trong, D. Wood, M. Tatsubori, and R. Tachibana, "Detection of anomalous

sounds for machine condition monitoring using classification confidence," in Proc. 5th Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE), 2020, pp. 66-70.

[15] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, "MobileNetV2: Inverted residuals and linear bottlenecks," in Proc. 31st IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 4510-4520.

[16] Y. Zeng, H. Liu, L. Xu, Y. Zhou, and L. Gan, "Robust anomaly sound detection framework for machine condition monitoring," DCASE2022 Challenge, Tech. Rep., 2022.

[17] I. Kuroyanagi, T. Hayashi, K. Takeda, and T. Toda, "Two-stage anomalous sound detection systems using domain generalization and specialization techniques," DCASE2022 Challenge, Tech. Rep., 2022.

[18] F. Xiao, Y. Liu, Y. Wei, J. Guan, Q. Zhu, T. Zheng, and J. Han, "The dcase2022 challenge task 2 system: Anomalous sound detection with self-supervised attribute classification and gmm-based clustering," DCASE2022 Challenge, Tech. Rep., 2022.

[19] Y. Deng, J. Liu, and W.-Q. Zhang, "Aithu system for unsupervised anomalous detection of machine working status via sounding," DCASE2022 Challenge, Tech. Rep., 2022.

[20] S. Venkatesh, G. Wichern, A. Subramanian, and J. Le Roux, "Disentangled surrogate task learning for improved domain generalization in unsupervised anomalous sound detection," DCASE2022 Challenge, Tech. Rep., 2022.

[21] Y. Wei, J. Guan, H. Lan, and W. Wang, "Anomalous sound detection system with self-challenge and metric evaluation for dcase2022 challenge task 2," DCASE2022 Challenge, Tech. Rep., 2022.

[22] K. Morita, T. Yano, and K. Tran, "Comparative experiments on spectrogram representation for anomalous sound detection," DCASE2022 Challenge, Tech. Rep., 2022.

[23] J. Bai, Y. Jia, and S. Huang, "Jless submission to dcase2022 task2: Batch mixing strategy based method with anomaly detector for anomalous sound detection," DCASE2022 Challenge, Tech. Rep., 2022.

[24] S. Verbitskiy, M. Shkhanukova, and V. Vyshegorodtsev, "Unsupervised anomalous sound detection using multiple time-frequency representations," DCASE2022 Challenge, Tech. Rep., 2022.

[25] K. Wilkinghoff, "An outlier exposed anomalous sound detection system for domain generalization in machine condition monitoring," DCASE2022 Challenge, Tech. Rep., 2022.

[26] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, "mixup: Beyond empirical risk minimization," in International Conference on Learning Representations, 2018. [Online]. Available: https://openreview.net/forum?id=r1Ddp1-Rb

[27] I. Nejjar, J. P. J. Meunier-Pion, G. M. Frusque, and O. Fink, "Dcase challenge 2022: Self-supervised learning pre-training, training for unsupervised anomalous sound detection," DCASE2022 Challenge, Tech. Rep., 2022.
# CONVOLUTIONAL NEURAL NETWORK FOR AUDIBILITY ASSESSMENT OF ACOUSTIC ALARMS

François Effa¹,²,³, Romain Serizel³, Jean-Pierre Arz¹, Nicolas Grimault²,

¹ Institut National de Recherche et de Sécurité, F-54000 Nancy, France

² Centre de Recherche en Neurosciences de Lyon, CNRS, F-69500 Bron, France

³ Université de Lorraine, CNRS, Inria, Loria, F-54000 Nancy, France

## ABSTRACT

In noisy workplaces, the audibility of acoustic alarms is essential to ensure worker safety. In practice, some criteria are required in international standards to make sure that the alarms are "clearly audible". However, the recommendations may lead to overly loud alarms, thereby exposing workers to unnecessary high sound levels, especially when ambient sound levels are high themselves. For this reason, it appears necessary to properly assess the audibility of alarms at design stage. Existing psychoacoustical methods rely on repeated subjective measurements at different sound levels and therefore require time-consuming procedures. In addition, they must be repeated each time the alarm or sound environment changes. To overcome this issue, we propose a data-driven approach to estimate the audibility of new alarm signals without having to test each new condition experimentally. In this study, a convolutional neural network model is trained to perform a binary classification task on short sound clips labeled with the outcomes of psychoacoustical experiments. We propose a proof of concept of this approach and analyze its performance depending on the data used at training and the temporal context used by the networks to predict the audibility of the alarm.

**Index Terms—** Acoustic Scene Classification, Warning signals, Psychophysics, Machine learning

## 1. INTRODUCTION

The audibility of acoustic signals indicating a danger – for instance, the reverse alarm of a construction machine – is essential to ensure the safety of workers and to prevent the risk of accidents. The international standard dedicated to auditory danger signals for public and work areas requires for the alarms to be “clearly audible” [1]. In order to meet this requirement, several criteria related to the ambient noise levels are proposed. In particular an overall level criterion imposes a minimum signal-to-noise ratio (SNR) of 15 dB. This recommendation is questionable on two counts. First, the proposed criteria appear to lead in practice to excessive sound levels when the ambient noise levels are high. Žera and Nagórski have found that, when asking listeners to adjust the level of alarms so that they are judged to be clearly audible, the required SNR varies continuously from 15 to –2 dB as the noise level increases from 60 dB to 90 dB [2]. An SNR of 15 dB could therefore be too conservative or even harmful at high noise levels. Second, there is no scientific or formal definition of what "clearly audible" means. We can easily understand the difference between a sound that is simply detectable and another, qualified as clearly audible. The alarm must not only be perceptible but also "loud" enough to provide an effective warn-

ing of danger. This judgment can however vary greatly from one individual to another. It should also be noted that such consideration is dependent on the sound environment in which the alarm occurs.

In the presence of background noise, our ability to detect a given sound is reduced. It translates into an increase in the audibility threshold of the target sound, meaning that the level at which the sound is just audible is higher in noise than in silence [3, 4]. The noise in question is called *masker*, and we refer to the audibility threshold when a masker is present as the *masked threshold*. The masking mechanism is the basis for current alarm design methods. With the understanding that the alarm level must be well above the masked threshold to ensure reliable audibility, it has been suggested that alarms should exceed the masked threshold by 10 to 15 dB [5]. This recommendation is also included in the standard [1]. Masked thresholds can be measured experimentally [6], but such approach is not the most convenient, since a psychoacoustical experiment requires the involvement of multiple human subjects, and measuring an audibility threshold implies covering a range of different sound levels, which can be time-consuming. Furthermore, the measurements strongly depend on the acoustic properties of the sounds. Consequently, the experiment must be repeated for any new condition of interest such as a new alarm signal, or different ambient noise type or level. In response to this problem, some models have been developed to predict masked thresholds [7, 8]. However, these are based on the explicit estimation of the masked thresholds, and therefore only efficient in well-controlled conditions.

The main motivation of our work is to propose a model capable of accurately estimating the audibility of acoustic alarms in noise. We intend this model to be applicable to a large variety of sound environments, including fluctuating noises and different types of warning signals. In that regard, we suggest that a deep learning approach, which is data-driven, would be suitable. A few studies have already begun to pave the way for connecting psychophysics and deep learning methods, for instance, by implementing a psychophysically inspired methodology for model evaluation [9]. More recently, in the context of handwritten document transcription, a loss function was reformulated to account for perceptual data [10]. The authors used it to train a CNN to perform character recognition and obtained consistent and repeatable performance improvement on standard datasets. In the field of audition, recent works have replicated human perceptual judgments with a high level of likeness in word recognition, genre recognition [11] and sound localization in natural environments [12].

In this paper, we propose to adapt methods used in ambient sound analysis [13, 14] to the specific topic of audibility assessment of auditory alarms. To that end, we introduce a new dataset consist-
ing of sound clips made up of auditory warning signals embedded in noise labelled through psychoacoustical experiments. Based on this dataset, we train a model to reproduce human judgment regarding the audibility of the alarms.

## 2. METHOD

Ambient sound analysis is a broad field of research that encompasses several sub-areas such as Acoustic Scene Classification (ASC) [15], Sound Event Detection (SED) [16, 17], or audio tagging [18]. Although our approach does not fully fit into any of these categories, it does share certain aspects with some of them. First, similarly to what is done in ASC and audio tagging, we want the model to produce file-level estimations. Besides, the analysis focuses on the alarms, which are temporally localized sound events. This is pretty similar to SED or tagging. Despite this, we are not directly interested in the ability of the model to detect the alarm. Yet, if the model can predict that an alarm is audible, it gives an indication that the model did initially detect it. The opposite, however, is not true. It is quite possible that the model predicts that an alarm is not audible although it has detected the alarm itself.

To assess whether a particular alarm can be considered clearly audible in a given environment, we propose to frame the problem as a binary classification task. The proposed method uses 5.5-second audio clips containing auditory warning signals embedded in background noises as input. After extracting acoustical features from these signals, we used a CNN to produce a binary estimate of whether the alarm present in the clip is clearly audible or not. More details about the audio clip generation and the annotation process are provided in the next section.

## 3. PSYCHOACOUSTICAL DATA COLLECTION

The need for relatively large amounts of data with a sufficient variability is a well-known constraint associated with deep learning. To develop our model with psychoacoustical data, no ready-to-use dataset was available from external sources. Therefore, we had to collect one. To do this, our choices were guided by two major considerations. First, we had to find a way to collect the maximum amount of data at the lowest time cost. Second, we still wanted to keep the possibility of explaining all or part of our results within the psychoacoustical framework. We decided to divide the psychoacoustical experiments in three parts. The first part is following within-subject design, closer to psychoacoustical procedures. The second and third parts follow lighter procedures and lower data collection time cost. In this section, we describe the psychoacoustical experiments that have been carried out to collect the dataset. The dataset in this paper is a preliminary version used for a proof of concept. The final dataset is expected to be larger.

### 3.1. Stimuli and material

Stimuli were made of short alarm sounds (between 0.243 and 1.763 s long) embedded in background noises. The alarms were mostly synthetic signals, but some of them were clean recordings. Backgrounds were field recordings, taken to be industry-related (factory, roadworks, construction) or captured in noisy public spaces. Both alarms and noises were mono signals collected from different sources, mainly the Freesound database [19], Big-SoundBank [20], and to a lesser extent, a published set of medical alarms [21] or self-recorded railway warning signals [22].

In the experiments, 5.5-second clips were generated, each containing a background noise and a single alarm with a random onset temporal location. The level of the noise varied between 60 dBA and 80 dBA. The SNRs were all taken between -30 dB and 15 dB. The stimuli were played at a sample rate of 44.1 kHz using an RME Babyface Pro sound card, and presented over Beyerdynamic DT 770 Pro headphones calibrated with Larson Davis AEC101 artificial ear and Model 824 sonometer.

### 3.2. Procedure and datasets

Twelve volunteers aged from 18 to 43 and free of reported hearing problems took part in this study. They came for multiple sessions of one or two hours each. All the participants were compensated for the time spent on the experiments. To evaluate the audibility, the subjects were presented with a clip made up of an alarm embedded in a background noise. At the end of the presentation, they had to answer the question "Was the alarm clearly audible?" by simply clicking Yes or No. Three different experiments were carried out and most of the subjects took part in each of the three experiments. Each experiment served to collect a separate subset designed for a specific purpose.

The first set consists of 6 audio clips, each declined in 2 different noise levels (60 dBA and 80 dBA) and 10 SNRs linearly spaced between -30 and 15 dB, making a total of 120 signals. In each of the 6 clips, the alarm signals and backgrounds are different from the other clips. All these stimuli have been annotated once by the 10 participants. As a result, it contains psychoacoustical data that are quite close to what would have been obtained through a standard procedure, yet it is often recommended to make more than one repetition per subject [1]. This set is the most controlled set and is selected as the evaluation set. The data collection process on this subset will eventually allow for further comparison with more standard psycho-acoustic experiments. The labels (0 or 1) have been obtained by setting a 0.5 threshold on the proportion of "Yes" across participant answers. This subset is referred to as *subA* in the remainder of the paper.

The second set contains 1800 audio clips, made with the same noises and alarms as in the first set, except that the 6 formerly used alarm-noise combinations were avoided. As a consequence, there was a total of 30 possible alarm-noise combinations. Six different SNRs ([-25, -10, -5, 0, 5, 12.5] in dB) and six noise levels ([60, 64, 68, 72, 76, 80] in dBA) were used and uniformly distributed among the clips. Each of the 1800 clips had a unique alarm onset location. Ten participants were involved in this experiment. Each of them listened to 180 clips. The distribution of the clips among subjects was done in such a way that each subject listened to the same number of clips per SNR and per noise level. No repetition was made across subjects, meaning each stimulus has been annotated just once. This subset is referred to as *subB*.

The data eventually collected in these experiments will be used to form a large training subset. However, the third set introduced in this paper is also composed of 1800 different clips. This is motivated by the desire to keep comparable size between this set and *subB*. There are 70 alarms and 52 background noises in this set, all different from the 6 used in the first and second sets. Two noise levels were used (60 dBA and 80 dBA). There were 46 different SNRs ranging from -30 to 15 dB with a step of 1 dB. Ten subjects contributed to the annotation. Some of the conditions have been randomly repeated among subjects, making a total of around 11500 annotation points. For clips with a single annotation point,
the subject's answer was kept as the final annotation. For clips with multiple annotation points, the labels were derived by setting a 0.5 on the Yes-rate. We refer to this subset as *subC*.

Subsets *subB* and *subC* were used separately for development, using 1440 training clips and 360 validation clips, which corresponds to a 80%/20% ratio. The training/validation split was performed randomly and kept fixed for all the experiments.

# 4. EXPERIMENTAL SETUP

## 4.1. Acoustic features

The signals were sampled at 44.1 kHz. The features we used are mel-spectrograms with 64 coefficients. They were extracted using a 1024-sample short-time Fourier transform (STFT), with 50% overlap and a Hamming window.

We did explore the idea of using more perceptually relevant representations such as cochleagrams [12] or spectro-temporal excitation patterns [23]. However, preliminary experiments with these features did not provide any significant performance improvement in terms of accuracy on the development sets. Therefore, in this paper we use only mel-spectrograms as input features.

For the experiments, the input representations were standardized to zero mean and unit variance along mel frequency bins. The standardization coefficients were computed over the whole training set.

## 4.2. Convolutional Neural Network

The architecture of the CNN used in this paper is inspired by models used in SED [14] and Bird Audio Detection [13]. The model is composed of 4 convolutional layers with [32, 64, 64, 128] filters per layer. Each filter has a 3-by-3 receptive field. Each convolutional layer is followed by ReLU activations and max pooling along the frequency axis ([1, 4], [1, 4], [1, 2] and [1, 2], respectively). The activation outputs from the last convolutional layer are stacked along frequency axis [13]. Preliminary experiments with recurrent layers did not lead to significant improvement. Therefore they are not used in this paper. Instead, we directly operate $L_p$ aggregation over the time axis on the stacked representations. $L_p$ aggregation with $p = 2$ was preferred over max pooling which is not differentiable and may lead to instability [24]. In addition, it has been shown to be more robust to variations in the relative duration of the alarm compared to the clip length [25]. The aggregation layer is followed by the classification layer that has one single neuron with sigmoid activation. The neuron is intended to produce an activation which is close to 1 when the alarm present in the clip is clearly audible, and close to 0 when the alarm is not clearly audible.

For training, back-propagation was performed using a binary cross-entropy loss function and Adam optimizer [26] with a learning rate of 0.0001. To reduce overfitting, dropout was applied on the outputs of all the convolutional layers with a rate of 0.25 and regularization was employed by fixing a 0.0001 weight decay in Adam. The model was trained for a maximum of 250 epochs and the epoch giving the best accuracy on validation set was kept.

## 4.3. Model evaluation

For the experiments, the model was evaluated from the area under the receiver operating characteristic curve (AUC) and the F1-score.

We trained the models with 10 randomized initializations. The metrics were computed on the outputs obtained with these 10 models. We report the mean and 95% confidence intervals of the metrics.

# 5. RESULTS AND DISCUSSION

In this section, we report the two series of experiments that have been conducted on the model. The first series of experiments focuses on the model performance depending on the data used at training. The second series of experiments assesses the potential effects of the temporal context used in the model to predict the audibility.

## 5.1. Impact of the training data

Our first series of experiments investigates the performance of the models trained on *subB* and *subC*. As described in Section 3.2, the clips in *subA* and *subB* were made with the same alarms and background noises. For this reason, we expect the model to perform better when it is trained on *subB* than on *subC*.

At first, the model was trained on *subB* or *subC* data while *subB* validation data were used to select the model. Table 1 shows the AUC and F1-score on development and test sets. As we can observe, performance on the test set is better when *subB* is used for training. There are two potential causes for this. The difference in performance can be due to the fact that the alarm signals and backgrounds in *subC* are different from those in *subB* and in the test set or to the fact the task addressed in *subC* is more difficult than in *subB* (or both). We evaluated models trained on *subC* and validated on either *subB* or *subC* to verify this second hypothesis (see Table 2). The results show a significant difference in performance on development set depending on whether the model was validated on *subB* or *subC*. The high development score when *subC* is used for validation suggests that the model can be fitted to *subC* data, which indicates that the task addressed in *subC* is in fact not more difficult than in *subB*. However, the performance on test set shows that the model gives better results on test data when the alarms and background noises have been seen during training. This raises the question whether collecting more training data with a larger set of alarms and backgrounds can help to compensate for this performance gap. If not, it would induce the need to see test alarms or test backgrounds or both during training. In practice, such a scenario would not be realistic. Despite the difference in performance, it should be noted that when training on different alarms and backgrounds, the model performance does not actually collapse.

<table><caption>Table 1: AUC and F1-scores on development and evaluation sets with 95% confidence intervals. *subB* is used for validation.</caption><thead><tr><th>Subset</th><th></th><th>Development score</th><th>Test score</th></tr></thead><tbody><tr><td rowspan="2">subB</td><td>AUC</td><td>94.4 ± 0.3</td><td>95.3 ± 0.7</td></tr><tr><td>F1</td><td>91.9 ± 0.3</td><td>89.2 ± 1.6</td></tr><tr><td rowspan="2">subC</td><td>AUC</td><td>78.9 ± 1.0</td><td>87.9 ± 1.9</td></tr><tr><td>F1</td><td>79.1 ± 1.2</td><td>79.3 ± 2.4</td></tr></tbody></table>

## 5.2. Impact of the clip duration

The alarms that are present in the different clips have variable lengths. Since the longest alarm is less than 1.8 s long, all the 5.5-second clips contain both portions of noise alone and the whole section where the alarm occurs. It is still to be determined whether the model bases its predictions on the entire alarm or on a specific region of the alarm. This region could be the onset, for instance.
<table><caption>Table 2: Performance of the model trained on *subC* with either *subB* or *subC* used for validation.</caption><thead><tr><th>Validation set</th><th></th><th>Development score</th><th>Test score</th></tr></thead><tbody><tr><td rowspan="2">subB</td><td>AUC</td><td>78.9 ± 1.0</td><td>87.9 ± 1.9</td></tr><tr><td>F1</td><td>79.1 ± 1.2</td><td>79.3 ± 2.4</td></tr><tr><td rowspan="2">subC</td><td>AUC</td><td>94.4 ± 0.4</td><td>88.3 ± 1.8</td></tr><tr><td>F1</td><td>87.2 ± 0.4</td><td>79.5 ± 2.0</td></tr></tbody></table>

Moreover, it is not sure whether it also relies on the information present in the parts of the signal where there is no alarm. In this second series of experiments, we are interested in observing how the model uses the temporal context to produce estimations of the audibility of the alarms.

The model was trained on *subB* since it resulted to better performance in the previous experiment. We varied the duration of the clips used to train the model. Four different durations were experimented: 5.5, 1.0, 0.5, and 0.1 seconds. For this, each input representation was shortened to the desired length around the alarm position. Every time, the model was tested on *subA* using all four clip lengths. The results are reported in Table 3.

As a first observation, when tested with the same clip length as the one used for training, the model shows relatively good performance. This result is true whatever the clip length. However, the model is only able to perform well for all test clip durations when it is trained on 5.5-second clips. This result suggests that the model needs temporal context at training time but not necessarily to make predictions at inference time. Finally, when 5.5-second clips are used for training, the performance of the model weakens slightly as the duration of the test clips is reduced, though it is still quite high. It would therefore be reasonable to train on long clips if the model is then to make predictions over shorter time periods. These observations are based on the AUC. The-F1 score shows some unexplained effects such as a lower value observed when the model is trained on 5.5-second clips and tested on 1-second clips. The investigation of these effects may require a more detailed analysis.

<table><caption>Table 3: Performance on test set depending on the clip length used for training and evaluation.</caption><thead><tr><th>Test<br>Training</th><th></th><th>0.1</th><th>0.5</th><th>1.0</th><th>5.5</th></tr></thead><tbody><tr><td rowspan="2">0.1</td><td>AUC</td><td>88.6 ± 1.4</td><td>53.2 ± 11.1</td><td>51.8 ± 11.9</td><td>50.1 ± 7.5</td></tr><tr><td>F1</td><td>81.8 ± 1.6</td><td>36.0 ± 19.5</td><td>32.9 ± 20.6</td><td>32.2 ± 21.0</td></tr><tr><td rowspan="2">0.5</td><td>AUC</td><td>53.3 ± 13.2</td><td>92.1 ± 1.5</td><td>55.5 ± 14.9</td><td>50.1 ± 6.7</td></tr><tr><td>F1</td><td>44.4 ± 17.7</td><td>85.0 ± 1.3</td><td>41.9 ± 18.7</td><td>33.8 ± 20.2</td></tr><tr><td rowspan="2">1.0</td><td>AUC</td><td>43.9 ± 15.2</td><td>53.5 ± 13.5</td><td>89.5 ± 1.8</td><td>47.7 ± 6.0</td></tr><tr><td>F1</td><td>36.7 ± 15.5</td><td>45.5 ± 14.7</td><td>81.6 ± 1.6</td><td>40.6 ± 14.3</td></tr><tr><td rowspan="2">5.5</td><td>AUC</td><td>88.8 ± 6.0</td><td>92.3 ± 1.9</td><td>93.9 ± 1.6</td><td>95.0 ± 1.0</td></tr><tr><td>F1</td><td>75.3 ± 6.5</td><td>80.0 ± 5.7</td><td>75.7 ± 4.2</td><td>87.0 ± 1.2</td></tr></tbody></table>

### 5.3. The model's output as a psychometric value

As previously mentioned, psychoacoustical experiments are usually conducted in a repeated measures design. For example, in order to evaluate the audibility of an alarm through a *Yes-No* task, a common approach consists in presenting same clip once or several times to every participant. With such procedure, we can measure the proportion of *Yes* responses over all trials. Then by varying a given attribute of the stimulus, it is possible to establish a relationship between this specific attribute and the subjects' responses. Such a relationship is called a psychometric function.

<figure><img src="image_35.png" /><figcaption>Figure 1: Psychometric function of a clip from *subA*. The rate of positive responses averaged across all participants as a function of the SNR is represented by the plain curve. The dotted curve with round markers shows the values taken by the last neuron.</figcaption></figure>

For instance, consider the procedure described in Section 3.2 to collect data for *subA*. Different clips were generated from the 6 initial clips by varying the SNR and the noise level. By taking a given clip at a single ambient noise level, we can represent the evolution of the proportion of *Yes* responses as the SNR increases. This approach is quite different from what we do when we train a CNN to perform a binary classification task. Indeed, we use binary labels for training. This means that the model is trained to produce outputs as close to 1 as possible when the alarm in the clip is judged to be clearly audible and close to 0 when the alarm is not clearly audible. However, information such as an actual *Yes*-rate is totally absent from the data seen by the model. As a consequence, we do not necessarily expect a match between the output of the model and a psychometric function when varying the SNR of the alarm present in a clip. Yet, we did try to observe the output of the model when the inputs were the same clips of *subA* with different SNRs. The activation of the last neuron was found to roughly follow the evolution of what could be interpreted as a psychometric curve. This result opens up analytical perspectives for future studies. An example is shown in Figure 1.

## 6. CONCLUSION

In this paper, we proposed a proof of concept of a new approach to assess the audibility of acoustic alarms. We presented an experimental procedure that was specifically designed to collect a dataset with perceptual annotation. This dataset was used to develop a model that gave auspicious results on a binary classification task. Both the influence of the training data and the importance of the temporal context have been investigated. Our results showed that it is possible to predict the audibility of acoustic alarms in relative accordance with human perception, even if training was made on a dataset that was collected using a much lighter procedure than usual psychoacoustical tests. However, we are aware of the lack of a baseline to compare the results of the present work, and therefore plan to collect new perceptual data with different annotators whose "performance" will serve as a basis for comparison with the model. Lastly, the psychoacoustical experiments presented in this article are part of a broader experimental method that includes the numerical rating of the audibility and a detection task that have not been detailed here. The data collected on this occasion will be used in future developments.
7. REFERENCES

[1] I. O. for standardization (ISO), "Ergonomics — danger signals for public and work areas — auditory danger signals," 2008.

[2] J. Żera and A. Nagórski, "Preferred levels of auditory danger signals," *Int. J. Occup. Saf. Ergon.*, vol. 6:sup1, pp. 111–117, 2004.

[3] E. Zwicker and H. Fastl, *Psychoacoustics: Facts and Models*. Berlin: Springer, 1999.

[4] B. C. J. Moore, *An Introduction to the Psychology of Hearing*, 6th ed. Leiden, Netherlands: Brill, 2013.

[5] R. Patterson, "Auditory warning sounds in the work environment," *Phil. Trans. R. Soc. Lond.*, vol. B 327, pp. 485–492, 1990.

[6] M. R. Leek, "Adaptive procedures in psychophysical research," *Perception & Psychophysics*, vol. 63, pp. 1279–1292, 2001.

[7] Y. Zheng, C. Giguère, C. Laroche, C. Sabourin, A. Gagné, and M. Elyea, "A Psychoacoustical Model for Specifying the Level and Spectrum of Acoustic Warning Signals in the Workplace," *Journal of Occupational and Environmental Hygiene*, vol. 4, no. 2, pp. 87–98, Jan. 2007.

[8] B. R. Glasberg and B. C. J. Moore, "Development and Evaluation of a Model for Predicting the Audibility of Time-Varying Sounds in the Presence of Background Sounds," *J. Audio Eng. Soc*, vol. 53, no. 10, pp. 906–918, 2005.

[9] B. RichardWebster, S. E. Anthony, and W. J. Scheirer, "Psychophy: A psychophysics driven evaluation framework for visual recognition," *IEEE Transactions on Pattern Analysis and Machine Intelligence*, vol. 41, no. 9, pp. 2280–2286, 2019.

[10] S. Grieggs, B. Shen, G. Rauch, P. Li, J. Ma, D. Chiang, B. Price, and W. Scheirer, "Measuring human perception to improve handwritten document transcription," *IEEE Transactions on Pattern Analysis and Machine Intelligence*, 2021.

[11] A. J. Kell, D. L. Yamins, E. N. Shook, S. V. Norman-Haignere, and J. H. McDermott, "A task-optimized neural network replicates human auditory behavior, predicts brain responses, and reveals a cortical processing hierarchy," *Neuron*, vol. 98, no. 3, pp. 630–644.e16, 2018.

[12] A. Francl and J. McDermott, "Deep neural network models of sound localization reveal how perception is adapted to real-world environments," *Nat. Hum. Behav.*, vol. 6, pp. 11–13, 2022.

[13] E. Cakir, S. Adavanne, G. Parascandolo, K. Drossos, and T. Virtanen, "Convolutional recurrent neural networks for bird audio detection," in *2017 25th European Signal Processing Conference (EUSIPCO)*, 2017, pp. 1744–1748.

[14] N. Turpault and R. Serizel, "Training sound event detection in a heterogenous dataset," in *Proc. DCASE Workshop*, 2020.

[15] J. Abeßer, "A review of deep learning based methods for acoustic scene classification," *Appl. Sci.*, vol. 10, no. 6, 2020.

[16] E. Çakir and T. Virtanen, "End-to-end polyphonic sound event detection using convolutional recurrent neural networks with learned time-frequency representation input," in *2018 International Joint Conference on Neural Networks (IJCNN)*, 2018.

[17] X. Xia, R. Togneri, F. Sohel, Y. Zhao, and D. Huang, "A survey: Neural network-based deep learning for acoustic event detection," *Circuits Syst. Signal Process.*, vol. 38, pp. 3433–3453, 2019.

[18] E. Fonseca, M. Plakal, F. Font, D. P. W. Ellis, and X. Serra, "Audio tagging with noisy labels and minimal supervision," in *Proc. DCASE Workshop*, 2019.

[19] The Freesound project, https://www.freesound.org.

[20] BigSoundBank by Joseph SARDIN, https://www.bigsoundbank.com.

[21] J. Atyeo and P. M. Sanderson, "Comparison of the identification and ease of use of two alarm sound sets by critical and acute care nurses with little or no music training: a laboratory study," *Anaesthesia*, vol. 70, no. 7, pp. 818–827, 2015.

[22] J.-P. Arz, N. Grimault, and O. ElSawaf, "Experimental assessment of the effect of wearing hearing protectors on the audibility of railway warning signals for normal hearing and hearing impaired listeners," *International Journal of Occupational Safety and Ergonomics*, 2021. [Online]. Available: https://doi.org/10.1080/10803548.2021.1991681

[23] B. R. Glasberg and B. C. J. Moore, "A model of loudness applicable to time-varying sounds," *Journal of the Audio Engineering Society*, vol. 50, no. 5, pp. 331–342, 2002.

[24] B. McFee, J. Salamon, and J. P. Bello, "Adaptive pooling operators for weakly labeled sound event detection," *IEEE/ACM Transactions on Audio, Speech, and Language Processing*, vol. 26, no. 11, pp. 2180–2193, 2018.

[25] N. Turpault, R. Serizel, and E. Vincent, "Analysis of weak labels for sound event tagging," 2021, working paper or preprint. [Online]. Available: https://hal.inria.fr/hal-03203692

[26] D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization," *arXiv:1412.6980*, 2014.
# DETECTION AND IDENTIFICATION OF BEEHIVE PIPING AUDIO SIGNALS

Dominique Fourier and Agnieszka Orlowska

IBISC (EA 4526) - University of Evry / Paris-Saclay*
Evry-Courcouronnes, France
dominique.fourer@univ-evry.fr

## ABSTRACT

Piping signals are particular sounds emitted by honey bees during the swarming season or sometimes when bees are exposed to specific factors during the life of the colony. Such sounds are of interest for beekeepers for predicting an imminent swarming of a beehive. The present study introduces a novel publicly available dataset made of several honey bee piping recordings allowing for the evaluation of future audio-based detection and recognition methods. First, we propose an analysis of the most relevant timbre features for discriminating between tooting and quacking sounds which are two distinct types of piping signals. Second, we comparatively assess several machine-learning-based methods designed for the detection and the identification of piping signals through a beehive-independent 3-fold cross-validation methodology.

***Index Terms***— bees piping signals, quacking, tooting, audio signal recognition, smart beekeeping

## 1. INTRODUCTION

Nowadays, smart beekeeping is gaining interest since it aims at developing innovative methods for enhancing the monitoring of beehives using AI techniques. To this end, the audio-based approach [1, 2] is promising since it allows to use low-cost sensors for monitoring a bee colony. Recent work pioneered the bee sound analysis problem through a machine learning approach to predict the different health states of a beehive. For example, the task of predicting the bee queen presence is investigated in [3, 4] and could help beekeepers to reduce the number of inspections which are stressful for a beehive. The prediction of colony swarming from audio signal is investigated [5, 6] and can be related to specific sounds emitted by the bees. Several studies analyze different piping sounds and show their interest for beekeepers [7, 5, 6]. Other studies explain that piping signals can also have other functions for synchronizing the colony activity [7, 8]. Such particular sounds can respectively be emitted by bee workers or by a queen and can easily be distinguished from classical background beehive sounds. A more recent study [9] proposes an acoustic analysis of piping signals which can be segregated into two classes with specific audio signatures: tooting and quacking. Both tooting and quacking signals can occur about 1 day before swarming and their occurrences can increase every 10 minutes during approximately 6 hours.

The present study pursues the piping sounds investigation with an analysis of the most relevant audio features using a machine learning-based methodology. Our contributions are manifold. First, we introduce a new publicly available audio piping dataset made of

several recordings collected from various beekeepers which were manually segmented and annotated as tooting or quacking. Second, we present an acoustic analysis through timbre features to discriminate between the tooting and the quacking signals. Finally, we assess several methods for a supervised detection and classification of audio field recordings of beehive sounds. This paper is organized as follows. In Section 2, we explain the differences between piping signals and we introduce our new proposed dataset. In Section 3, we perform an acoustic analysis of piping sounds using timbre features. Section 4 presents our audio detection and classification results using several proposed methods. Finally, the paper is concluded by a discussion with future work directions in Section 5.

## 2. MATERIALS

### 2.1. Tooting and Quacking

Piping sounds (cf. Fig. 1) are among the most noticeable signs of swarming. *Tooting* corresponds to the sound emitted by a virgin queen bee who announces her presence by releasing pheromones and by tooting. *Tooting* corresponds to a series of pulsed, high-pitched sounds produced by pressing her thorax and operating her wing-beating mechanism without spreading her wings [10]. Mature queens still confined within their queen cells answer the tooting with a distinct piping sound, called *Quacking*. A chorus of synchronized quacking follows each tooting, and those specific swarming sounds are broadcasting in the bee nest as vibrations of the combs and perceived by vibration detectors in the workers’ tarsi [11]. *Toots* and *quacks* are made of different varying pulses: during the process of tooting, the queen produces a one-second-long pipe immediately followed by several bursts of less than half a second. The fundamental frequency increases with the age of queens, ranging from 200 to 550 Hz, and is usually observed around 400 Hz [12]. *Quacks* are made of several short pulses which are typically less than 0.2 seconds at a lower fundamental frequency around 350 Hz [13]. Piping sounds are not only emitted by queens but also by workers in queenless colonies: laying-workers and guarding-workers [14]. More recent studies show that workers could emit piping sounds to prepare a synchronized liftoff [7]. This prompts a conclusion that workers pipe in a variety of circumstances, while queens pipe only in the context of colony reproduction [15]. The queens’ toots and quacks last several seconds and are broken up into syllables [12]. Piping sounds emitted by workers come from several sources and have a duration below one second. It often consists of a single pulse [14].

### 2.2. New Proposed Piping Dataset

We introduce a novel dataset of natural honey bee piping audio signals which was built by collecting 44 different recordings pub-

<footnote>*This work is partly supported by the French ANR ASCETE project (ANR-19-CE48-0001).</footnote>
<figure><img src="image_36.png" /><figcaption>Figure 1: Spectrograms with highlighted $F_0$ and waveforms with RMS envelope of two distinct piping signals.</figcaption></figure>

lished on the YouTube platform by various beekeepers around the world. These audio recordings were obtained in field conditions using various non-professional microphones located close to the beehive when a piping signal is emitted. Each recording has a duration varying from 2 to 13 seconds and is annotated according to the beekeeper comment respectively as *Tooting* or *Quacking*. We extracted and segmented the audio from 14 distinct videos from which the signal is recorded without a loss of quality into WAVE files with a sampling frequency of $F_s = 22.05$ kHz and a sample precision of 16 bits. After manually removing the silent and spurious frames, the resulting dataset contains 36 tooting signals and 8 quacking signals which correspond to a duration of 145 seconds for tooting and 60 seconds for quacking (total 205 seconds). To avoid possible copyright issues, we only made publicly available the Short-Time Fourier Transform (STFT) matrices and the timbre descriptors computed using a matlab implementation of the timbre toolbox [16] from the post-processed signals used in our experiments. We propose a more detailed description of the dataset containing the links of the original Youtube videos with our matlab loader codes published on IEEE DataPort [17].

# 3. ACOUSTIC ANALYSIS

## 3.1. Signal analysis

We present in Fig. 1 the waveform of a tooting and of a quacking signal both extracted from our proposed dataset (Toot1 and Quack1) with almost the same duration of about 6 seconds. Colored in red, we plot the Root Mean Square (RMS) envelope computed for a window length of 23ms. We also display the spectrograms of the same signals where the fundamental frequency ($F_0$) estimated using the SWIPE method [18] is highlighted. From these observations, one can notice that tooting and quacking are both harmonic signals but with very different temporal and spectral structures. The tooting signal contains longer pulses with a higher $F_0$ (mean value of $\mu_T = 382.97$Hz with a standard deviation $\sigma_T = 61.45$ Hz) and a slightly lower number of pulses for the same observation duration. For the comparison, the quacking signal contains more pulses with a lower $F_0$ ($\mu_Q = 306.60$ Hz, $\sigma_Q = 23.98$ Hz). We also notice that the $F_0$ decreases at the end of each pulse for both tooting and quacking signals.

<table><caption>Table 1: Top-10 most relevant timbre descriptors selected using a mutual information criterion.</caption><thead><tr><td></td><td>Timbre feature</td><td>Relevance score</td></tr></thead><tbody><tr><td>1</td><td>ERB-gammatone Spectral Centroid</td><td>0.428</td></tr><tr><td>2</td><td>ERB-gammatone Spectral Kurtosis</td><td>0.419</td></tr><tr><td>3</td><td>ERB-fft Spectral Kurtosis</td><td>0.402</td></tr><tr><td>4</td><td>ERB-gammatone Spectral Skewness</td><td>0.373</td></tr><tr><td>5</td><td>ERB-fft Spectral Skewness</td><td>0.373</td></tr><tr><td>6</td><td>ERB-fft Spectral Centroid</td><td>0.371</td></tr><tr><td>7</td><td>ERB-fft Spectral Spread</td><td>0.334</td></tr><tr><td>8</td><td>Zero-crossing rate</td><td>0.321</td></tr><tr><td>9</td><td>STFT Spectral Kurtosis</td><td>0.314</td></tr><tr><td>10</td><td>STFT Spectral Roll-Off</td><td>0.311</td></tr></tbody></table>

## 3.2. Timbre Feature Selection

The timbre toolbox proposed by Peeters et al. [16] proposes a large set of hand-crafted audio features used in various audio recognition tasks. These features are expected to convey information about the perceived timbre of an arbitrary sound. They include temporal, spectral, harmonic and perceptual descriptors which are directly computed from the waveform and from the time-frequency representation of the analyzed signal. In this study, we investigate a total of 164 timbre features (cf. [19] Table. 2 for details) summarized by median and Inter Quartile Range (IQR) statistics related to the signal acoustic parameters. In Table 1, we present the top-10 most relevant features sorted by descending order of relevance according to the mutual information (MI) criterion [20] by considering the tooting/quacking classification problem. Our computation uses the scikit-learn MI python implementation which shows that perceptual-based Equivalent-Rectangular-Bandwith (ERB) spectral features appear to be the most relevant. Fig. 2a plots in 3 dimensions the whole dataset where each individual corresponds to a one-second-long frame where the axes correspond to the top-3 most relevant features. This figure shows that the components can almost be separated into two distinct clusters corresponding to tooting and quacking signals (plotted with different colors) using only 3 relevant features. In Fig. 2b, we plot a whole dataset projection using Principal Component Analysis (PCA) which is a dimension reduction method reducing the redundancy between the features while
preserving original data inertia. This second projection shows that the separation between tooting and quacking sounds is not trivial despite each cluster seem located in a different area. Finally, we perform a Linear Discriminant Analysis (LDA) [21] which can be viewed as a supervised PCA providing the optimal linear projection of the dataset which maximizes the Euclidean distance between individuals of different classes while minimizing the distance between individuals of the same class. Fig. 2c shows that there exists a linear combination of the original timbre features enabling to perfectly separate tooting and quacking sounds. This result paves the way of a supervised classification investigated in Section 4.

# 4. DETECTION AND CLASSIFICATION RESULTS

## 4.1. Experimental Setup

We focus on two distinct tasks which consist of the detection of piping signals and the discrimination between tooting and quacking piping signals. To this end, we consider three distinct experiments. **Experiment 1** focuses on the detection of piping signals from bee-hives recordings. We address this problem through a binary classification problem involving samples from our proposed dataset and beehive recordings from the OSBH dataset<fnref n="1" /> made of several bee-hives sounds. **Experiment 2** focuses on the binary piping audio classification problem which consists in identifying respectively tooting and quacking signals where 145 recordings are labeled as *tooting* and 60 recordings as *quacking*. **Experiment 3** considers both the detection and the classification problem that is addressed through a 3-label supervised classification approach consisting in predicting if a signal is a *tooting*, a *quacking* or a *non-piping* signal. For each experiment, datasets are preprocessed by splitting signals into one-second-long chunks sampled at $F_s = 22.05$ kHz. Each signal is centered by subtracting the mean and the amplitude is normalized by dividing each sample by $\max(|x|)$. Our evaluation uses a 3-fold cross-validation methodology (2 training folds and 1 testing fold) where the recordings are beehive-independent to avoid overfitting and to assess over the whole dataset the generalizing capability of the trained models. Hence, all recordings from the same Youtube video are only present into a unique fold and cannot simultaneously appear in both the training and testing sets. In experiments 1 and 3 involving *non-piping* signals, we randomly add bee signals from the OSBH dataset to obtain the same number of *piping* and *non-piping* signals in each fold.

## 4.2. Methods

### 4.2.1. Classification

We comparatively assess four distinct supervised classification methods suitable for beehive audio signals. The **TTB+SVM** method uses the 164 timbre descriptors investigated in Section 3 combined with a support vector machines (SVM) classifier with a Gaussian radial basis function kernel [22]. The proposed **1D-CNN** method uses the modulus of the discrete Fourier transform of the signal as input of a 1D-convolutional neural network (CNN) with residual connections. This architecture (total: 7,684,226 trainable parameters) is made of 4 residual blocks with a different number of kernel filters (sequentially: 16, 32, 64, 128). Each residual block is made of 3 one-dimensional convolutional layers interspersed by the addition of the input followed by a Rectified Linear Unit (ReLU) activation and a max-pooling. Output of the last residual block is average-pooled and connected to 3 fully-connected

<figure><img src="image_37.png" /><figcaption>(a) Top-3 most relevant timbre features</figcaption></figure>

<figure><img src="image_38.png" /><figcaption>(b) PCA</figcaption></figure>

<figure><img src="image_39.png" /><figcaption>(c) LDA</figcaption></figure>

Figure 2: Three-dimensional projections of our proposed piping dataset where each point corresponds to a one-second-long excerpt.

(FC) layers including flatten and with ReLU and softmax activation for the final output. The **MFCC+CNN** and the **STFT+CNN** are based on the same 2D-CNN architecture (total: 404,770 trainable

<footnote n="1">¹https://zenodo.org/record/1321278</footnote>
parameters) with 2 distinct inputs: Mel-Frequency Cepstral Coefficients (MFCC) and the spectrogram defined as the squared modulus of the short-time Fourier transform (STFT). The proposed 2D-CNN architecture is inspired from [2] and consists of 4 convolutional blocks containing 16 kernel filters of size 3×3, a 2×2 max-pooling layer and a 25% dropout layer. The output is connected to a 3 FC layers including 2 dropout layers of respectively 25% and 50% followed by a softmax activation function to compute the output predicted label. Convolutional and FC layers both use a LeakyReLU activation function defined as $LeakyReLU(x) = max(\alpha x, x)$, with $\alpha = 0.1$.

### 4.2.2. Detection

For detecting piping in an arbitrary audio signal as proposed in **Experiment 1**, we also consider the 4 proposed classification methods using a binary piping/non-piping taxonomy. We also consider two additional methods based on the stochastic modeling of the estimated $F_0$ distribution respectively for piping and non-piping signals. This later approach is motivated by the harmonic property of piping signals described in Section 3. The **F0 Gaussian model** estimates the parameters $\theta = [\mu, \sigma^2]$ of a Gaussian probability distribution used to model respectively piping and non-piping signals. Thus, given the estimated $F_0$ denoted $f_x$ of a signal, the decision to detect a piping signal is made when $p(f_x|\theta_{piping}) > p(f_x|\theta_{non-piping})$. The **F0 kernel model** is a variant of the **F0 Gaussian model** where $p(f_x|\theta_{piping})$ is estimated using the empirical distribution (i.e. histogram) of the estimated $F_0$ smoothed by a convolution product using a Gaussian kernel [23]. Our experiments used the SWIPE $F_0$ estimator [18] for which the median function is used to summarize a frame of signal with an arbitrary length.

## 4.3. Implementation details

The 17 first cepstral coefficients of the **MFCC+CNN** method are computed each 20 ms. The STFT is computed using a Hann analysis window with a 50% overlap and a FFT size of $M = 1025$ (i.e. 512 positive frequency bins). The input of the **1D-CNN** is set to half of the frequency sampling due to the Fourier transform Hermitian symmetry of a real signal (i.e. 11,025 real-valued coefficients). During the testing of each of the 3 folds, we use data augmentation (DA) [24] to artificially increase the number of training recordings by generating new samples from the original ones by the addition of a white Gaussian noise (SNR = 25dB) and by the application of temporal random circular shifts. The results reported in Tables 2, 3 and 4 correspond to the best ones obtained after several iterations (no significant improvement is shown by data augmentation). The training of our CNN methods is configured for a constant number of 25 epochs for the **1D-CNN** and 50 epochs for the **2D-CNN**, with a batch size of 16. The overall evaluation framework and the **TTB+SVM** method are implemented in matlab. The deep learning methods are implemented in Python using Keras with Tensorflow frameworks. Our codes are freely available online<fnref n="2" /> for the sake of reproducible research.

## 4.4. Comparative results

According to Table 2, the best detection results in terms of accuracy for **Experiment 1** are obtained using the **TTB+SVM** and the **MFCC+CNN** method which both obtain 94%. The best piping classification (**Experiment 2**) results (cf. Table 3) are obtained

using the **STFT+CNN** method with an overall accuracy of 95%, followed from far by the **MFCC+CNN** method which obtains an accuracy of 78%. Despite efforts, the two other techniques fail to identify quacking sounds and obtain poorer results with a quacking F-measure below 0.5. These poor quacking recognition results are confirmed in **Experiment 3** (cf. Table 4) where the best method remain **STFT+CNN** for which the results are poorer than in **Experiment 2**. This suggests the best pipeline which detects piping signals using **MFCC+CNN** or **TTB+SVM** before attempting to discriminate between tooting and quacking signals using **STFT+CNN**.

<table><caption>Table 2: Experiment 1: Piping signals detection comparative results.</caption><thead><tr><th>Method</th><th>Feat. dimension</th><th>Label</th><th>Recall</th><th>Precision</th><th>F-score</th><th>Accuracy</th></tr></thead><tbody><tr><td rowspan="2">F0 kern. model</td><td rowspan="2">1</td><td>Piping</td><td>0.68</td><td>0.96</td><td>0.79</td><td rowspan="2">0.84</td></tr><tr><td>Non-piping</td><td>0.97</td><td>0.78</td><td>0.87</td></tr><tr><td rowspan="2">F0 Gauss. model</td><td rowspan="2">1</td><td>Piping</td><td>0.69</td><td>0.99</td><td>0.81</td><td rowspan="2">0.85</td></tr><tr><td>Non-piping</td><td>1</td><td>0.79</td><td>0.88</td></tr><tr><td rowspan="2">TTB+SVM</td><td rowspan="2">164</td><td>Piping</td><td><strong>0.91</strong></td><td>0.96</td><td><strong>0.94</strong></td><td rowspan="2">0.94</td></tr><tr><td>Non-piping</td><td>0.97</td><td><strong>0.93</strong></td><td><strong>0.95</strong></td></tr><tr><td rowspan="2">1D-CNN</td><td rowspan="2">11,025</td><td>Piping</td><td>0.84</td><td><strong>1.00</strong></td><td>0.91</td><td rowspan="2">0.93</td></tr><tr><td>Non-piping</td><td><strong>1.00</strong></td><td>0.88</td><td>0.93</td></tr><tr><td rowspan="2">MFCC+CNN</td><td rowspan="2">17×47</td><td>Piping</td><td>0.87</td><td><strong>1.00</strong></td><td>0.93</td><td rowspan="2"><strong>0.94</strong></td></tr><tr><td>Non-piping</td><td><strong>1.00</strong></td><td><strong>0.90</strong></td><td>0.94</td></tr><tr><td rowspan="2">STFT+CNN</td><td rowspan="2">512×42</td><td>Piping</td><td>0.86</td><td>0.96</td><td>0.91</td><td rowspan="2">0.92</td></tr><tr><td>Non-piping</td><td>0.97</td><td>0.89</td><td>0.93</td></tr></tbody></table>

<table><caption>Table 3: Experiment 2: Piping signals binary classification comparative results.</caption><thead><tr><th>Method</th><th>Feat. dimension</th><th>Label</th><th>Recall</th><th>Precision</th><th>F-score</th><th>Accuracy</th></tr></thead><tbody><tr><td rowspan="2">TTB+SVM</td><td rowspan="2">164</td><td>Tooting</td><td>0.78</td><td>0.85</td><td>0.71</td><td rowspan="2">0.66</td></tr><tr><td>Quacking</td><td>0.24</td><td>0.18</td><td>0.38</td></tr><tr><td rowspan="2">1D-CNN</td><td rowspan="2">11,025</td><td>Tooting</td><td><strong>0.97</strong></td><td>0.72</td><td>0.82</td><td rowspan="2">0.71</td></tr><tr><td>Quacking</td><td>0.08</td><td>0.50</td><td>0.14</td></tr><tr><td rowspan="2">MFCC+CNN</td><td rowspan="2">17×47</td><td>Tooting</td><td>0.93</td><td>0.79</td><td>0.86</td><td rowspan="2">0.78</td></tr><tr><td>Quacking</td><td>0.42</td><td>0.71</td><td>0.53</td></tr><tr><td rowspan="2">STFT+CNN</td><td rowspan="2">512×42</td><td>Tooting</td><td>0.94</td><td><strong>0.98</strong></td><td><strong>0.96</strong></td><td rowspan="2"><strong>0.95</strong></td></tr><tr><td>Quacking</td><td><strong>0.96</strong></td><td><strong>0.87</strong></td><td><strong>0.92</strong></td></tr></tbody></table>

<table><caption>Table 4: Experiment 3: Simultaneously Detection and classification classification comparative results.</caption><thead><tr><th>Method</th><th>Feat. dimension</th><th>Label</th><th>Recall</th><th>Precision</th><th>F-score</th><th>Accuracy</th></tr></thead><tbody><tr><td rowspan="3">TTB+SVM</td><td rowspan="3">164</td><td>Tooting</td><td>0.88</td><td>0.78</td><td>0.83</td><td rowspan="3">0.82</td></tr><tr><td>Quacking</td><td>0.03</td><td>0.12</td><td>0.05</td></tr><tr><td>Non-piping</td><td>0.99</td><td>0.89</td><td>0.94</td></tr><tr><td rowspan="3">1D-CNN</td><td rowspan="3">11,025</td><td>Tooting</td><td>0.93</td><td>0.84</td><td>0.88</td><td rowspan="3">0.85</td></tr><tr><td>Quacking</td><td>0.10</td><td>0.54</td><td>0.16</td></tr><tr><td>Non-piping</td><td>0.99</td><td>0.86</td><td>0.92</td></tr><tr><td rowspan="3">MFCC+CNN</td><td rowspan="3">17×47</td><td>Tooting</td><td>0.88</td><td>0.81</td><td>0.84</td><td rowspan="3">0.84</td></tr><tr><td>Quacking</td><td>0.18</td><td>0.45</td><td>0.26</td></tr><tr><td>Non-piping</td><td><strong>0.99</strong></td><td><strong>0.90</strong></td><td><strong>0.95</strong></td></tr><tr><td rowspan="3">STFT+CNN</td><td rowspan="3">512×42</td><td>Tooting</td><td><strong>0.94</strong></td><td><strong>0.97</strong></td><td><strong>0.95</strong></td><td rowspan="3"><strong>0.91</strong></td></tr><tr><td>Quacking</td><td><strong>0.50</strong></td><td><strong>0.76</strong></td><td><strong>0.60</strong></td></tr><tr><td>Non-piping</td><td><strong>0.99</strong></td><td>0.89</td><td>0.94</td></tr></tbody></table>

# 5. CONCLUSION

We introduced a new dataset made of beehive piping sounds designed for identifying tooting and quacking signals emitted by bees. The most relevant timbre features were presented and reveal a link with perceptual spectral features. Our numerical experiments involving several state-of-the-art approaches show that a time-frequency representation combined with a 2D-CNN is currently the most promising approach for addressing the tooting/quacking binary classification problem and can obtain an accuracy above 85%. Future work consists in evaluating new methods in more realistic application scenarios involving embedded systems.

<footnote n="2">²https://fourer.fr/dcase22</footnote>
## 6. REFERENCES

[1] S. Cecchi, A. Terenzi, S. Orcioni, P. Riolo, S. Ruschioni, and N. Isidoro, "A preliminary study of sounds emitted by honey bees in a beehive," in *Audio Engineering Society Convention 144*, Milan, Italy, May 2018.

[2] S. Cecchi, A. Terenzi, S. Orcioni, and F. Piazza, "Analysis of the sound emitted by honey bees in a beehive," in *Audio Engineering Society Convention 147*, 2019.

[3] I. Nolasco and E. Benetos, "To bee or not to bee: Investigating machine learning approaches for beehive sound recognition," in *Proc. DCASE*, Nov. 2018.

[4] I. Nolasco, A. Terenzi, S. Cecchi, S. Orcioni, H. L. Bear, and E. Benetos, "Audio-based identification of beehive states," in *Proc. IEEE ICASSP*, 2019, pp. 8256–8260.

[5] A. Zgank, "Bee swarm activity acoustic classification for an iot-based farm service," *Sensors*, vol. 20, no. 1, p. 21, 2020.

[6] M.-T. Ramsey, M. Bencsik, M. I. Newton, M. Reyes, M. Pioz, D. Crauser, N. S. Delso, and Y. Le Conte, "The prediction of swarming in honeybee colonies using vibrational spectra," *Scientific reports*, vol. 10, no. 1, pp. 1–17, 2020.

[7] T. D. Seeley and J. Tautz, "Worker piping in honey bee swarms and its role in preparing for liftoff," *Journal of Comparative Physiology A*, vol. 187, no. 8, pp. 667–676, 2001.

[8] C. Thom, D. C. Gilley, and J. Tautz, "Worker piping in honey bees (apis mellifera): the behavior of piping nectar foragers," *Behavioral Ecology and Sociobiology*, vol. 53, no. 4, pp. 199–205, 2003.

[9] T. Yamamoto, M. Sugahara, R. Okada, and H. Ikeno, "Differences between queen piping temporal structures of two honeybee species, apis cerana and apis mellifera," *Apidologie*, vol. 52, no. 2, pp. 524–534, 2021.

[10] J. Simpson, "The mechanism of honey-bee queen piping," *Zeitschrift für vergleichende Physiologie*, vol. 48, no. 3, pp. 277–282, 1964.

[11] J. Simpson and S. M. Cherry, "Queen confinement, queen piping and swarming in apis mellifera colonies," *Animal Behaviour*, vol. 17, pp. 271–278, 1969.

[12] A. Michelsen, W. H. Kirchner, B. B. Andersen, and M. Lindauer, "The tooting and quacking vibration signals of honey-bee queens: a quantitative analysis," *Journal of Comparative Physiology A*, vol. 158, no. 5, pp. 605–611, 1986.

[13] W. Kirchner, "Acoustical communication in honeybees," *Apidologie*, vol. 24, no. 3, pp. 297–307, 1993.

[14] T. Ohtani and T. Kamada, "worker piping': The piping sounds produced by laying and guarding worker honeybees," *Journal of Apicultural Research*, vol. 19, no. 3, pp. 154–163, 1980.

[15] S. Pratt, S. Kühnholz, T. D. Seeley, and A. Weidenmüller, "Worker piping associated with foraging in undisturbed queenright colonies of honey bees," *Apidologie*, vol. 27, no. 1, pp. 13–20, 1996.

[16] G. Peeters, B. L. Giordano, P. Susini, N. Misdariis, and S. McAdams, "The timbre toolbox: Extracting audio descriptors from musical signals," *The Journal of the Acoustical Society of America*, vol. 130, no. 5, pp. 2902–2916, 2011.

[17] A. Orlowska and D. Fourer, "Identification of beehive piping audio signals," in *IEEE Dataport (doi:10.21227/53mq-g936)*, 2021. [Online]. Available: https://dx.doi.org/10.21227/53mq-g936

[18] A. Camacho and J. G. Harris, "A sawtooth waveform inspired pitch estimator for speech and music," *The Journal of the Acoustical Society of America*, vol. 124, no. 3, pp. 1638–1652, 2008.

[19] D. Fourer, J.-L. Rouas, P. Hanna, and M. Robine, "Automatic timbre classification of ethnomusicological audio recordings," in *Proc. ISMIR*, Taipei, Taiwan, Oct. 2014.

[20] N. Hoque, D. K. Bhattacharyya, and J. K. Kalita, "Mifs-nd: A mutual information-based feature selection method," *Expert Systems with Applications*, vol. 41, no. 14, pp. 6371–6385, 2014.

[21] T. W. Anderson, *An Introduction to Multivariate Statistical Analysis*. New York, USA: Wiley-Blackwell, 1958.

[22] A. J. Smola and B. Schölkopf, "A tutorial on support vector regression," *Statistics and computing*, vol. 14, no. 3, pp. 199–222, 2004.

[23] A. W. Bowman and A. Azzalini, *Applied smoothing techniques for data analysis: the kernel approach with S-Plus illustrations*. OUP Oxford, 1997, vol. 18.

[24] D. A. Van D. and X.-L. Meng, "The art of data augmentation," *Journal of Computational and Graphical Statistics*, vol. 10, no. 1, pp. 1–50, 2001.
# SOUND EVENT LOCALIZATION AND DETECTION FOR REAL SPATIAL SOUND SCENES: EVENT-INDEPENDENT NETWORK AND DATA AUGMENTATION CHAINS

Jinbo Hu<fnref n="1" />,², Yin Cao³, Ming Wu<fnref n="1" />, Qiuqiang Kong⁴, Feiran Yang<fnref n="1" />, Mark D. Plumbley⁵, Jun Yang<fnref n="1" />,²

¹Key Laboratory of Noise and Vibration Research, Institute of Acoustics, Chinese Academy of Sciences, Beijing, China, {hujinbo, mingwu, feiran, jyang}@mail.ioa.ac.cn

²University of Chinese Academy of Sciences, Beijing, China

³Xi’an Jiaotong Liverpool University, Suzhou, China, yin.k.cao@gmail.com

⁴ByteDance Shanghai, China, kongqiuqiang@bytedance.com

⁵Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey, UK m.plumbley@surrey.ac.uk

## ABSTRACT

Sound event localization and detection (SELD) is a joint task of sound event detection and direction-of-arrival estimation. In DCASE 2022 Task 3, types of data transform from computationally generated spatial recordings to recordings of real-sound scenes. Our system submitted to the DCASE 2022 Task 3 is based on our previous proposed Event-Independent Network V2 (EINV2) with a novel data augmentation method. Our method employs EINV2 with a track-wise output format, permutation-invariant training, and a soft parameter-sharing strategy, to detect different sound events of the same class but in different locations. The Conformer structure is used for extending EINV2 to learn local and global features. A data augmentation method, which contains several data augmentation chains composed of stochastic combinations of several different data augmentation operations, is utilized to generalize the model. To mitigate the lack of real-scene recordings in the development dataset and the presence of sound events being unbalanced, we exploit FSD50K, AudioSet, and TAU Spatial Room Impulse Response Database (TAU-SRIR DB) to generate simulated datasets for training. We present results on the validation set of Sony-TAu Realistic Spatial Soundscapes 2022 (STARSS22) in detail. Experimental results indicate that the ability to generalize to different environments and unbalanced performance among different classes are two main challenges. We evaluate our proposed method in Task 3 of the DCASE 2022 challenge and obtain the second rank in the teams ranking. Source code is released<fnref n="1" />.

**Index Terms—** Sound event localization and detection, real spatial sound scenes, Event-Independent Network, data augmentation chains, simulated datasets

## 1. INTRODUCTION

Sound event localization and detection (SELD) consists of sound event detection (SED) and direction-of-arrival (DoA) estimation. SED aims to detect the presence and types of sound events, and DoA estimation predicts the spatial locations of different sound sources. SELD characterizes sound sources in a spatial-temporal manner. SELD plays an important role in a wide range of applications, such as robot auditory and surveillance of intelligent home.

SELD has received broad attention recently. Adavanne et al. [1] proposed a polyphonic SELD approach using an end-to-end network, SELDnet, which was utilized for a joint task of SED and regression-based DoA estimation. SELD was then introduced in Task 3 of the Detection and Classification of Acoustics Scenes and Events (DCASE) 2019 Challenge for the first time, which uses the TAU Spatial Sound Events 2019 dataset [2]. Most datasets of spatial sound events are computationally simulated and these recordings are generated by convolving randomly chosen sound event examples with a corresponding random real-life spatial room impulse response (SRIR) to spatially place them at a given position [2–4]. To bring each iteration of Task 3 of DCASE Challenge closer to real conditions, stronger reverberation, diversity of environment, dynamic scenes with both moving and static sound sources, ambient noise, sound events of the same type, and unknown directional interfering events out of the target classes were added into datasets to complicate the SELD task. In 2022, the challenge transforms from computationally simulated spatial recordings to real spatial sound scene recordings. The Sony-TAu Realistic Spatial Soundscapes 2022 (STARSS22) dataset is manually annotated and released to serve as the development and evaluation dataset of DCASE2022 Task 3 this year [5].

SELDnet is unable to detect sound events of the same type but with different locations [1], which is also called homogeneous overlap. An event-independent network (EIN) with a track-wise output format was proposed to detect the homogeneous overlap problem [6–8]. In EIN, there are several event-independent tracks, and each track can be of any event. The number of tracks needs to be pre-determined according to the maximum number of overlapping events. EINV2, an improved version of EIN, utilizes multi-head self-attention (MHSA) and a soft parameter-sharing strategy of multi-task learning to achieve better performance [7].

The training set often deviates from real-scene spatial and acoustical environments, and mismatched distribution of locations and sound types between the training set and test set is common. A novel data augmentation method is used to generalize the model [8, 9]. The data augmentation method contains several data augmentation chains. These data augmentation chains consist of some randomly sampled data augmentation operations. The augmentation method can increase the diversity of augmented features.

In this study, our system is based on our previous proposed EINV2 with data augmentation chains. EINV2 is extended by

<footnote n="1">¹https://github.com/Jinbo-Hu/DCASE2022-TASK3</footnote>
Conformer, which is a combination structure of self-attention and convolution. The data augmentation method is composed of several augmentation operations. These data augmentation operations are sampled and layered randomly to combine to several data augmentation chains [8]. External data is allowed in this challenge. We generate simulated data by randomly convolving chosen samples of sound events from AudioSet [10] and FSD50K [11] with measured SRIRs from TAU Spatial Room Impulse Responses Database<fnref n="2" /> (TAU-SRIR DB). The experimental results show the proposed model with the novel data augmentation method, which was trained on our simulated data, outperforms the DCASE2022 challenge Task 3 baseline model which was trained on official synthetic SELD mixtures<fnref n="3" />. In addition, we present class-wise and room-wise metric scores of the validation set of STARSS22 in detail. The proposed system obtains the second rank in Task 3 of DCASE 2022 Challenge<fnref n="4" />.

## 2. THE METHOD

### 2.1. Input features

In this method, log-mel spectrograms and intensity vectors (IV) in log-mel space are used for features of the SELD task. First order ambisonics (FOA) include four-channel signals, i.e., omnidirectional channel w, and three directional channels x, y, and z. Log-mel spectrograms are computed from the mel filter banks and the short-time Fourier transform spectrograms, and IVs are cross-correlation of log-mel spectrograms of w with x, y and z [12]. These features are directly calculated online using a 1-D convolutional layer, which supports data augmentation on raw waveform.

### 2.2. Network Architecture

The track-wise output format was introduced in our previous works [6–8]. It can be defined as

$$Y_{\text{Trackwise}} = \{(y_{\text{SED}}, y_{\text{DoA}}) \mid y_{\text{SED}} \in \mathbb{O}_{\mathbf{S}}^{M \times K}, y_{\text{DoA}} \in \mathbb{R}^{M \times 3}\} \quad (1)$$

where M is the number of tracks, K is the number of sound-event types, $\mathbb{O}_{\mathbf{S}}^{M \times K}$ is one-hot encoding of K classes, and S is the set of sound events. Cartesian DoA estimation is used here.

The number of tracks is determined by the maximum polyphony. Each track can only detect a sound event with a corresponding direction of arrival. While a model with a track-wise output format is trained, sound events may be predicted in any track, instead of a fixed track. It may cause the track permutation problem that sound events predicted and their ground truth may not be aligned in a fixed track. Permutation-invariant training (PIT) is proposed to tackle the problem effectively. The PIT loss is defined as

$$\mathcal{L}_{\text{PIT}}(t) = \min_{\alpha \in \mathbf{P}(t)} \sum_{M} \left\{ \lambda \cdot \ell_{\alpha}^{\text{SED}}(t) + (1 - \lambda) \cdot \ell_{\alpha}^{\text{DoA}}(t) \right\} \quad (2)$$

where $\alpha \in \mathbf{P}(t)$ indicates one of the possible permutations and $\lambda$ is a loss weight between SED and DoA. $\ell_{\alpha}^{\text{SED}}$ is binary cross entropy loss for the SED task, and $\ell_{\alpha}^{\text{DoA}}$ is mean square error for the DoA task. The lowest loss will be chosen by finding a possible permutation, and the back-propagation is then performed.

<figure><img src="image_40.png" /><figcaption>Figure 1: The architecture of the SELD network, which is a Conv-Conformer network. The upper half (yellow boxes) is the SED task. The lower half (blue boxes) is the DoA estimation task. The green boxes sandwiched between SED branch and DoA branch indicate soft connections between SED and DoA estimation.</figcaption></figure>

From multi-task learning (MTL) perspective, joint SELD learning can be mutually beneficial. Hard parameter-sharing (PS) and soft PS are two typical methods to implement MTL. Hard PS means subtasks use the same feature layers, while soft PS means subtasks use their own feature layers with connections existing among those feature layers. In [7], experimental results show that soft PS using cross-stitch is more effective.

EINV2, which combines the track-wise output format, PIT, and soft PS, is utilized in our system. Three tracks are adopted to address up to three overlapped sound events. Multi-head self-attention (MHSA) blocks are replaced with Conformer blocks. Conformer consists of two feed-forward layers with residual connections sandwiching the MHSA and convolution modules, and hence has the ability to capture global and local patterns. [8, 13]. Our proposed network is shown in Fig. 1.

### 2.3. Data Augmentation Chains

The main characteristic of our data augmentation method is using some augmentation chains [8,9,14]. These augmentation chains are combined by some augmentation operations, which are randomly selected and linked in chain. We randomly sample $k = 3$ augmentation chains. Augmentation operations that are used here include Mixup [15], Cutout [16], SpecAugment [17], and frequency shifting [18]. Rotation of FOA signals [19] is an additional augmentation method, but excluded by data augmentation chains. The diagram of data augmentation chains is shown in Fig. 2.

Mixup utilize convex combinations of pairs of feature vectors and their labels to train the model. Mixup on both raw waveform and spectrograms is used here to improve the ability of detecting overlapping sound events. While random Cutout produces several rectangular masks on spectrograms, SpecAugment produces stripes masks on time and frequency dimension of spectrograms. Frequency shifting in the frequency domain is similar to pitch shift in the time domain, and it randomly shifts input features of all the channels up or down along the frequency dimension by several bands. We also use a spatial augmentation method, rotation of FOA signals. It rotates FOA format signals by channel swap to enrich DoA labels. This method does not lose physical relationships

<footnote n="2">²https://doi.org/10.5281/zenodo.6408611</footnote>

<footnote n="3">³https://doi.org/10.5281/zenodo.6406873</footnote>

<footnote n="4">⁴https://dcase.community/challenge2022</footnote>
<table><caption>Table 1: The SELD performance of our proposed system. The training set of STARSS22 is mixed into synthetic training set by default.</caption><thead><tr><th rowspan="2">System</th><th rowspan="2">Datasets</th><th colspan="4">Validation set</th><th colspan="4">Evaluation (Blind test) set</th></tr><tr><th>ER<sub>20°</sub></th><th>F<sub>20°</sub></th><th>LE<sub>CD</sub></th><th>LR<sub>CD</sub></th><th>ER<sub>20°</sub></th><th>F<sub>20°</sub></th><th>LE<sub>CD</sub></th><th>LR<sub>CD</sub></th></tr></thead><tbody><tr><td>Baseline FOA [5]</td><td>Official</td><td>0.71</td><td>21.0%</td><td>29.3°</td><td>46.0%</td><td>0.61</td><td>23.7%</td><td>22.9°</td><td>51.4%</td></tr><tr><td>EINV2 w/o dataAug chains</td><td>Official</td><td>0.75</td><td>32.3%</td><td>24.0°</td><td>56.1%</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>EINV2 w/ dataAug chains</td><td>Official</td><td>0.56</td><td>42.4%</td><td>19.3°</td><td>61.4%</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>System #1</td><td>A+B+C</td><td>0.50</td><td>48.4%</td><td>19.5°</td><td>65.7%</td><td>0.44</td><td>49.2%</td><td>16.6°</td><td>70.4%</td></tr><tr><td>System #2</td><td>A+B</td><td>0.50</td><td>51.0%</td><td>16.4°</td><td>65.9%</td><td>0.40</td><td>57.4%</td><td>15.1°</td><td>70.6%</td></tr><tr><td>System #3</td><td>A</td><td>0.53</td><td>48.1%</td><td>17.8°</td><td>62.6%</td><td>0.39</td><td>55.8%</td><td>16.2°</td><td>72.4%</td></tr><tr><td>System #4</td><td>B</td><td>0.53</td><td>45.4%</td><td>17.4°</td><td>62.5%</td><td>0.40</td><td>50.9%</td><td>15.9°</td><td>69.4%</td></tr></tbody></table>

<figure><img src="image_41.png" /><figcaption>Figure 2: Diagram of data augmentation chains</figcaption></figure>

between sound sources and observers. We use z-axis as the rotation axis to swap directional channel x and y, which leads to 16 types of channel rotation.

## 2.4. Simulated Data

Manual annotations are expensive and the duration of STARSS22 (about 5 hours of the development set) is limited compared with the synthetic datasets (about 13 hours synthetic recordings in DCASE 2021) used in previous years, therefore, external datasets are used to improve the model performance. We generated simulated data using the generator code<fnref n="5" /> provided by DCASE 2022.

Samples of sound events are mainly sourced from FSD50K dataset, based on affinity of the labels in that dataset to the target classes. The target class *background music* and the interference class *shuffling cards* are not in FSD50K dataset, therefore, we use AudioSet as a supplement. Spatial events were spatialized in 9 unique rooms, using collected SRIRs from the TAU-SRIR DB dataset. The ambient noise from the same room was additionally mixed at varying signal-to-noise ratios (SNR) ranging from 30 dB to 6 dB. The maximum polyphony of target classes is 3, excluding additional polyphony of interference classes.

We select sound event samples whose labels significantly corresponded to the target classes. Each sound event sample also has a different energy gain for mixing. By setting different ranges of gain and choosing different samples, we generate three datasets, A, B, and C. All of these synthetic datasets have 2700 1-minute clips.

# 3. EXPERIMENTS

## 3.1. Datasets

The STARSS22 dataset contains recordings of real scenes, and the density of sound event samples and the presence of each class varies

greatly. The maximum number of the overlaps is 5, but those cases are very rare [5]. The overlap of 4 and 5 accounts for the proportion of 1.8% in total. Occurrences of up to 3 simultaneous events are fairly common, so we ignore the case scenarios that the number of overlapping events is more than 3. During the development stage, we train our proposed model on mixed datasets of synthetic recordings and the training set of STARSS22, and evaluate those systems using the validation set of STARSS22. During the evaluation stage, both synthetic recordings and all of the development set of STARSS22 are used for training.

## 3.2. Hyper-parameters

Audio clips are segmented to have a fixed length of 5 seconds with no overlap for training and inference. Log-mel spectrograms and intensity vectors features, with 24 kHz sampling rate, a 1024-point Hanning window with a hop size of 400, and 128 mel bins, are extracted from these audio segments. AdamW optimizer is used. The learning rate is set to 0.0003 for the first 70 epochs and then decreased to 0.00003 for the following 20 epochs. The threshold for SED is set to 0.5 to binarize predictions. The loss weight λ is 0.5.

## 3.3. Evaluation Metrics

We use the official evaluation metrics to evaluate the SELD performance [20, 21]. The evaluation metrics use a joint metric of localization and detection: location-sensitive F-score ($F_{<T°}$), error rate ($ER_{<T°}$), and class-sensitive localization recall ($LR_{CD}$), localization error ($LE_{CD}$). $T°$ means spatial threshold and is set to 20° in this challenge. $F_{<T°}$ and $ER_{<T°}$ consider true positives predicted under a spatial threshold $T°$ from the ground truth. For $LE_{CD}$ and $LR_{CD}$, the detected sound class has to be correct in order to count the corresponding localization predictions.

In the previous challenges, the evaluation metrics were micro-averaged, which gives equal weight to each individual decision and the performance is affected by the classes with more samples. In this challenge, macro-averaging of evaluation metrics is used. Macro-averaging gives equal weight to each class and emphasizes the system performance on the smaller classes [22].

## 3.4. Experimental Results

Table 1 summarizes the performance of our proposed systems. The official dataset means the synthetic mixtures for baseline training. The system baseline, EINV2 without dataAug chains, and EINV2 with dataAug chains all use the same dataset for training. EINV2

<footnote n="5">⁵https://github.com/danielkrause/DCASE2022-data-generator</footnote>
<figure><img src="image_42.png" /><figcaption>Figure 3: Metric scores of System #2 on validation set of STARSS22 in detail. The first column shows metric scores of the whole validation set. The following columns present metric scores of each room of validation set.</figcaption></figure>

without data augmentation chains outperforms the baseline model, whereas EINV2 with data augmentation chains performs better.

All configurations of systems #1 - #4 are the same as system EINV2 with dataAug chains, except for the training set used. The results also demonstrate the effectiveness of our simulated data over the official dataset.

The first column of Fig. 3 shows class-wise metric scores of System #2 on the validation set of STARSS22. The class-wise performance on the whole validation set is highly skewed, with $F_{\le 20^{\circ}}$ of knock class being 80.0%, whereas $F_{\le 20^{\circ}}$ of water tap and faucet class being 2.2%. $LE_{CD}$ of female speech class and water tap and faucet class is a lot higher than average. Other columns of Fig. 3 present class-wise performance for each room. Unbalanced class-wise performance among different rooms results in the skewed class-wise performance on the whole validation set.

The performance of the localization in room 2 is the worst among all the rooms, resulting in a directly significant increase of $LE_{CD}$ of female speech class. It may be attributed to small room size of room 2 compared with other rooms. $LR_{CD}$ of walk, footsteps (0.0%) class and water tap and faucet (2.4%) class in room 24 is very low. A possible reason is the low quality of synthetic training samples, because we ignore the natural temporal occurrences and spatial connections of some types of sounds happening in real scenes when simulating data [5]. For example, the target class water tap and faucet and the directional interference class dishes, pots, and pans often occur simultaneously in room 24, which leads to many observed false negatives of the class water tap and faucet in the system output. It is difficult to synthesis training samples that contains the temporal and spatial relationships of sound events in real scenes. These factors can lead to performance degradation.

## 4. CONCLUSION

We have presented an approach using an Event-Independent Network V2 (EINV2) with a novel data augmentation method for real-life sound event localization and detection. EINV2 is extended by conformer blocks. The novel data augmentation method contains several augmentation chains, which are stochastic combinations of data augmentation operations. For this challenge, we synthesized more training samples which are convolved using sound events from FSD50k and AudioSet with measured room impulse responses from TAU-SRIR DB. Our model with data augmentation chains performs better than the baseline model. Furthermore, experimental results show further improvement with our synthetic datasets. We also show results on the validation set of STARSS22 in detail. Our proposed method is evaluated in the evaluation set of STARSS22, and obtained the second best team in Task 3 of DCASE 2022 Challenge. The study of the generalization ability to different environments and the performance for unbalanced classes will be analyzed further in the future work.

## 5. ACKNOWLEDGEMENT

This work was partly supported by Frontier Exploration project independently deployed by Institute of Acoustics, Chinese Academy of Sciences (No. QYTS202009), UK Engineering and Physical Sciences Research Council (EPSRC) grant EP/T019751/1 "AI for Sound". For the purpose of open access, the authors have applied a Creative Commons Attribution (CC BY) licence to any Author Accepted Manuscript version arising.
## 6. REFERENCES

[1] S. Adavanne, A. Politis, J. Nikunen, and T. Virtanen, "Sound event localization and detection of overlapping sources using convolutional recurrent neural networks," *IEEE J. Sel. Top. Signal Process.*, vol. 13, pp. 34–48, 2018.

[2] S. Adavanne, A. Politis, and T. Virtanen, "A multi-room reverberant dataset for sound event localization and detection," in *Proc. DCASE 2019 Workshop*, 2019, pp. 10–14.

[3] A. Politis, S. Adavanne, and T. Virtanen, "A dataset of reverberant spatial sound scenes with moving sources for sound event localization and detection," in *Proc. DCASE 2020 Workshop*, 2020, pp. 165–169.

[4] A. Politis, S. Adavanne, D. Krause, A. Deleforge, P. Srivastava, and T. Virtanen, "A dataset of dynamic reverberant sound scenes with directional interferers for sound event localization and detection," in *Proc. DCASE 2021 Workshop*, 2021, pp. 125–129.

[5] A. Politis, K. Shimada, P. Sudarsanam, S. Adavanne, D. Krause, Y. Koyama, N. Takahashi, S. Takahashi, Y. Mitsufuji, and T. Virtanen, "STARSS22: A dataset of spatial recordings of real scenes with spatiotemporal annotations of sound events," in *arXiv:2206.01948*, 2022.

[6] Y. Cao, T. Iqbal, Q. Kong, Y. Zhong, W. Wang, and M. D. Plumbley, "Event-independent network for polyphonic sound event localization and detection," in *Proc. DCASE 2020 Workshop*, 2020, pp. 11–15.

[7] Y. Cao, T. Iqbal, Q. Kong, F. An, W. Wang, and M. D. Plumbley, "An improved event-independent network for polyphonic sound event localization and detection," in *Proc. IEEE ICASSP 2021*, 2021, pp. 885–889.

[8] J. Hu, Y. Cao, M. Wu, Q. Kong, F. Yang, M. D. Plumbley, and J. Yang, "A track-wise ensemble event independent network for polyphonic sound event localization and detection," in *Proc. IEEE ICASSP 2022*, 2022, pp. 9196–9200.

[9] D. Hendrycks, N. Mu, E. D. Cubuk, B. Zoph, J. Gilmer, and B. Lakshminarayanan, "AugMix: A simple data processing method to improve robustness and uncertainty," in *Proc. ICLR 2020*, 2020.

[10] J. F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, "Audio Set: An ontology and human-labeled dataset for audio events," in *Proc. IEEE ICASSP 2017*, 2017, pp. 776–780.

[11] E. Fonseca, X. Favory, J. Pons, F. Font, and X. Serra, "FSD50K: an open dataset of human-labeled sound events," *IEEE/ACM Trans. on Audio, Speech, and Lang. Process.*, vol. 30, pp. 829–852, 2021.

[12] P.-A. Grumiaux, S. Kitić, L. Girin, and A. Guérin, "A survey of sound source localization with deep learning methods," *The Journal of the Acoustical Society of America*, vol. 152, no. 1, pp. 107–151, 2022.

[13] A. Gulati, J. Qin, C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu, and R. Pang, "Conformer: Convolution-augmented transformer for speech recognition," in *Proc. Interspeech 2020*, 2020, pp. 5036–5040.

[14] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, "A simple framework for contrastive learning of visual representations," in *Proc. ICML 2020*, 2020, pp. 1597–1607.

[15] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, "mixup: Beyond empirical risk minimization," in *Proc. ICLR 2018*, 2018.

[16] Z. Zhong, L. Zheng, G. Kang, S. Li, and Y. Yang, "Random erasing data augmentation," in *Proc. of AAAI 2020*, vol. 34, no. 07, 2020, pp. 13 001–13 008.

[17] D. S. Park, W. Chan, Y. Zhang, C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le, "SpecAugment: A simple data augmentation method for automatic speech recognition," in *Proc. Interspeech 2019*, 2019, pp. 2613–2617.

[18] T. T. N. Nguyen, K. N. Watcharasupat, K. N. Nguyen, D. L. Jones, and W.-S. Gan, "SALSA: Spatial cue-augmented log-spectrogram features for polyphonic sound event localization and detection," *IEEE/ACM Trans. on Audio, Speech, and Lang. Process.*, vol. 30, pp. 1749–1762, 2022.

[19] L. Mazzon, Y. Koizumi, M. Yasuda, and N. Harada, "First order ambisonics domain spatial augmentation for DNN-based direction of arrival estimation," in *Proc. DCASE 2019 Workshop*, 2019, pp. 154–158.

[20] A. Politis, A. Mesaros, S. Adavanne, T. Heittola, and T. Virtanen, "Overview and evaluation of sound event localization and detection in DCASE 2019," *IEEE/ACM Trans. on Audio, Speech, and Lang. Process.*, vol. 29, pp. 684–698, 2020.

[21] A. Mesaros, S. Adavanne, A. Politis, T. Heittola, and T. Virtanen, "Joint measurement of localization and detection of sound events," in *Proc. IEEE WASPAA 2019*, 2019, pp. 333–337.

[22] A. Mesaros, T. Heittola, and T. Virtanen, "Metrics for polyphonic sound event detection," *Appl. Sci.*, vol. 6, no. 6, p. 162, 2016.
# COLOC: CONDITIONED LOCALIZER AND CLASSIFIER FOR SOUND EVENT LOCALIZATION AND DETECTION

Sławomir Kapka*, Jakub Tkaczuk

Samsung R&D Institute Poland

Warsaw, Poland

s.kapka@samsung.com, j.tkaczuk@samsung.com

## ABSTRACT

In this article we describe Conditioned Localizer and Classifier (CoLoC) which is a novel solution for Sound Event Localization and Detection (SELD). The solution constitutes of two stages: the localization is done first and is followed by classification conditioned by the output of the localizer. In order to resolve the problem of unknown number of sources we incorporate the idea borrowed from Sequential Set Generation (SSG). Models from both stages are SELDnet-like CRNNs, but with single outputs. Conducted reasoning shows that such two single output models are fit for SELD task. We show that our solution improves on the baseline system in most metrics on the STARSS22 Dataset.

**Index Terms—** DCASE2022 Challenge Task3, Sound Event Localization and Detection, CRNN, Ambisonics

## 1. INTRODUCTION

Sound Event Localization and Detection (SELD) is a complex task with many applications in robotics and surveillance. Since 2019, DCASE host annual Challenge in which Task3 is precisely SELD. This allows for gradual improvement of SELD systems over time.

The original SELDNet [1] has a drawback of being unable to detect multiple overlapping occurrences of events from the same class. We follow [2] and henceforth we call this problem homogeneous overlap. Thus, a new track-wise output format has been introduced in [2] incorporating Permutation Invariant Training (PIT) which precisely tackles the problem of homogeneous overlap. Since then, PIT was used in the improved version of DCASE2021 Top solution [3] as well as in the DCASE2022 Task3 Baseline System [4].

In this paper we propose a novel two-stage solution which incorporates class agnostic localizer based on Sequential Set Generation (SSG) and classifier conditioned on the output of the localizer. We will explain in the Section 2 how our solution solves the problem of homogeneous overlap without PIT. Our solution can be seen as an improved and refined version of our previous system from DCASE2019 [5]. In this solution the estimator of the number of active sources is included as a part of a conditional localizer, i.e. using SSG localizer we can retrieve the number of active sound sources in each time frame.

In the following Section 2 we describe the main components and the inference process of our method. In Section 3 we present a way how to train the components. Finally, in Section 4 we describe detailed results obtained on the STARSS22 Dataset [6] and compare our solution with the baseline system.

## 2. PROPOSED METHOD

### 2.1. Motivation

From the perspective of statistical learning theory, the optimal solutions to the problem of Empirical Risk Minimalisation (ERM) are conditional probability $P(Y|X)$ and conditional expectation $E(Y|X)$ for classification with cross-entropy loss and for regression with $L^2$ loss respectively [7]. By abuse of language, we will call the general solution to the ERM problem simply as conditional probability and we will use notations $P(Y|X)$ and $E(Y|X)$ rather frivolously.

SELD is the problem of simultaneous localization and detection, so the optimal solution in each time frame may be modeled as joint probability:

$$P(\{c_i \wedge l_i\}_{i=1..k}|X),$$

where $c_i$ and $l_i$ denote the class and the location of a detected event $e_i$, and $k \le N$, where $N$ is the maximal number of overlapping events. The first problem rises from the fact that models do not output sets. The most popular workaround is to force a model to output a list of length $N$ denoting the individual tracks with class and location information, where some tracks may be empty.

In our solution we first localize all audio events using the Sequential Set Generation method, which simultaneously allows us to estimate the number of active sound sources. More precisely, the localizer returns, in a sequential manner, directions of arrivals (DOAs) conditioned by DOAs which it already returned starting from the empty set $\emptyset$:

$$l_1 = E(l|X, \emptyset)$$

$$l_2 = E(l|X, \{l_1\})$$

$$l_3 = E(l|X, \{l_1, l_2\})$$

...

$$l_k = E(l|X, \{l_i\}_{i=1..k-1})$$

$$\tau = E(l|X, \{l_i\}_{i=1..k}),$$

where $\tau$ is a special token denoting that there is no more events. In our case $\tau$ is set to be an origin 0 from $\mathbb{R}^3$.

Based on the output from the localizer we then classify the event corresponding to this particular DOA. Thus informally

$$P(c_i \wedge l_i|X) = P(c_i|X, l_i) \cdot P(l_i|X),$$

which indicates that we could resolve the SELD task given an SSG localizer $E(l|X, \{l_i\}_{i=1..k})$ and location-conditioned classifier $P(c|X, l)$.

<footnote>*Corresponding author.</footnote>
The only edge case where the above solution may fail is when two or more events overlap spatially. However, in practice outputs from models have temporal context which should resolve this issue.

## 2.2. Stacked-Tracks

Let us consider a chunk of First Order Ambisonics (FOA) audio format in which there are at most *N* overlapping audio events. With such audio we associate meta information about the location and classes of the occurrences in each time frame. By location we mean *xyz* Cartesian coordinates on a unit sphere, and by class we mean one of the *K* predefined classes. We aim to construct an *N*×*T*×4 tensor *stacked-tracks*, where *N* is the maximal number of overlapping events, *T* denotes the number of time bins and the last dimension contains information about locations and classes. The tensor contains all available meta information in a convenient form. To obtain stacked-tracks, we iterate sequentially over occurrences and stack them from bottom to top. If an event terminates in some track, then all events from the above tracks are stacked down. All remaining empty cells are filled with zeros in *xyz* coordinates and with a new class index *K* which is interpreted as the lack of any of predefined event, i.e. silence or unknown event. Figure 1 presents an example of how to obtain stacked-tracks. Since the tracks can be permuted before stacking, the stacked-tracks are not unique; we will exploit this during training later.

<table><thead><tr><th rowspan="2">Tracks</th><th colspan="8">Time Frames</th></tr><tr><th>0</th><th>1</th><th>2</th><th>3</th><th>4</th><th>5</th><th>6</th><th>7</th></tr></thead><tbody><tr><td rowspan="4">T4</td><td>0.2</td><td>0.2</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>0.7</td><td>0.8</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>-0.2</td><td>-0.1</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>3</td><td>3</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td rowspan="4">T3</td><td></td><td></td><td>0.5</td><td>0.5</td><td>0.5</td><td>0.6</td><td>0.6</td><td></td></tr><tr><td></td><td></td><td>-0.7</td><td>-0.7</td><td>-0.7</td><td>-0.7</td><td>-0.7</td><td></td></tr><tr><td></td><td></td><td>0.5</td><td>0.5</td><td>0.5</td><td>0.4</td><td>0.4</td><td></td></tr><tr><td></td><td></td><td>7</td><td>7</td><td>7</td><td>7</td><td>7</td><td></td></tr><tr><td rowspan="4">T2</td><td>-0.5</td><td>-0.4</td><td>-0.4</td><td>-0.4</td><td>-0.3</td><td></td><td></td><td></td></tr><tr><td>0.6</td><td>0.7</td><td>0.7</td><td>0.8</td><td>0.8</td><td></td><td></td><td></td></tr><tr><td>0.3</td><td>0.3</td><td>0.3</td><td>0.3</td><td>0.4</td><td></td><td></td><td></td></tr><tr><td>3</td><td>3</td><td>3</td><td>3</td><td>3</td><td></td><td></td><td></td></tr><tr><td rowspan="3">T1</td><td></td><td></td><td></td><td></td><td></td><td></td><td>0.7</td><td>0.7</td></tr><tr><td></td><td></td><td></td><td></td><td></td><td></td><td>0.5</td><td>0.5</td></tr><tr><td></td><td></td><td></td><td></td><td></td><td></td><td>-0.5</td><td>-0.5</td></tr><tr><td rowspan="4">T0</td><td></td><td></td><td></td><td>-0.9</td><td>-0.9</td><td>-0.8</td><td></td><td></td></tr><tr><td></td><td></td><td></td><td>0.2</td><td>0.2</td><td>0.2</td><td></td><td></td></tr><tr><td></td><td></td><td></td><td>0.1</td><td>0.1</td><td>0.2</td><td></td><td></td></tr><tr><td></td><td></td><td></td><td>8</td><td>8</td><td>8</td><td></td><td></td></tr></tbody></table>

↓↓↓ Stacking ↓↓↓

<table><caption>Figure 1: An illustration of obtaining stacked-tracks from regular tracks. In this example there are five tracks with maximal overlap of three events. In this example there are 13 predefined classes, thus cells with coordinates at origin get a class label with index 13.</caption><thead><tr><th rowspan="2">Tracks</th><th colspan="8">Time Frames</th></tr><tr><th>0</th><th>1</th><th>2</th><th>3</th><th>4</th><th>5</th><th>6</th><th>7</th></tr></thead><tbody><tr><td rowspan="4">ST2</td><td>0.0</td><td>0.0</td><td>0.2</td><td>0.5</td><td>0.5</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>0.0</td><td>0.0</td><td>0.8</td><td>-0.7</td><td>-0.7</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>0.0</td><td>0.0</td><td>-0.1</td><td>0.5</td><td>0.5</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>13</td><td>13</td><td>3</td><td>7</td><td>7</td><td>13</td><td>13</td><td>13</td></tr><tr><td rowspan="4">ST1</td><td>0.0</td><td>0.2</td><td>0.5</td><td>-0.4</td><td>-0.3</td><td>0.6</td><td>0.6</td><td>0.0</td></tr><tr><td>0.0</td><td>0.7</td><td>-0.7</td><td>0.8</td><td>0.8</td><td>-0.7</td><td>-0.7</td><td>0.0</td></tr><tr><td>0.0</td><td>-0.2</td><td>0.5</td><td>0.3</td><td>0.4</td><td>0.4</td><td>0.4</td><td>0.0</td></tr><tr><td>13</td><td>3</td><td>7</td><td>3</td><td>3</td><td>7</td><td>7</td><td>13</td></tr><tr><td rowspan="4">ST0</td><td>-0.5</td><td>-0.4</td><td>-0.4</td><td>-0.9</td><td>-0.9</td><td>-0.8</td><td>0.7</td><td>0.7</td></tr><tr><td>0.6</td><td>0.7</td><td>0.7</td><td>0.2</td><td>0.2</td><td>0.2</td><td>0.5</td><td>0.5</td></tr><tr><td>0.3</td><td>0.3</td><td>0.3</td><td>0.1</td><td>0.1</td><td>0.2</td><td>-0.5</td><td>-0.5</td></tr><tr><td>3</td><td>3</td><td>3</td><td>8</td><td>8</td><td>8</td><td>11</td><td>11</td></tr></tbody></table>

## 2.3. Self-Conditioned SSG Localizer

The idea behind self-conditioned SSG localizer is to recursively localize all events from known classes without specifying class labels, i.e. it is class-agnostic. We will call this module simply as localizer. Our localizer consists of two trainable components:

* Localizer Encoder $L_{enc} : \mathbb{R}^3 \to \mathbb{R}^c$, where $c$ is the hyperparameter denoting the number of new channels.

* Localizer Network $L_{net} : \mathbb{R}^{t \times F \times C} \to \mathbb{R}^{T \times 3}$, where $C = c + c_f$ given that $c_f$ is the number of feature channels, where $t$, $F$ are numbers of time and frequency bins respectively, and where $T$ is the number of label time bins with meta resolution (in our case $t = 5 \cdot T$).

The core idea behind our self-conditioned SSG localizer is to recursively obtain all DOAs in all time frames. We start from the blank stacked-tracks and in each step we successively fill tracks in such a way that $L_{enc}$ encodes previously detected DOAs which $L_{net}$ should ignore. For that, we introduce below a modified version of the SSG method:

**Step 0.** During the first step we set *xyz* to zeros in each time frame (is convenient to look at it as putting empty row below ST0 in the Figure 1). Hence, we end up with the vector of size $t \times 3$. Then we apply $L_{enc}$ to its frame-wise to obtain tensor of size $T \times c$. Next, we repeat the tensor so many times to obtain vector which could be stacked with extracted features from audio channels. Thus, we obtain tensor of size $t \times F \times c$ which we concatenate with audio features tensor of size $t \times F \times c_f$ to obtain $t \times F \times C$ tensor. Finally, we feed it to $L_{net}$ to get the tensor of size $T \times 3$ denoting the predicted DOAs in each frame. When there is no active sound sources in a frame we expect it to return origin (i.e. $xyz = 0$) similarly as it was done in [8]. On the other hand, when there are some events we expect the model to return *xyz* coordinates of any of the occurring events. In the end we simply threshold length of vectors to decide whether there has been any event. If the length is greater than 0.5, then we conclude that there is an event, otherwise we put zeros. In summary, we obtain information about DOAs in each time-frame which we put in a first row ST0 in the blank stacked-tracks.

**Step 1.** During the second iteration, the output from the first step is fed into the $L_{enc}$. If in the first step in some frames an event was detected, then we encode these predicted DOAs, 0 otherwise. In the same way as in the first step, the encoded tensor is repeated and stacked with features channels. Now, we expect the $L_{net}$ model to output in each time-frame the DOA of second event if there is any and origin if there is none, or the already detected one is the only one. Analogously as in the first step, we threshold the length of output to decide if there is a new event. We stack the obtained results in stacked-track ST1 on top of ST0.

**Step n.** Lets say we already have $n-1$ stacked-tracks. The aim of this step is to get the DOAs of events which haven't been already localized. For these frames where stacked-track no. $n-1$ denotes that there is less than $n-1$ events, we encode the origin via $L_{enc}$. Otherwise, for these frames where we acquired $n-1$ DOAs we encode them individually using $L_{enc}$ and for each such frame we average $n-1$ obtained embeddings from $\mathbb{R}^c$. Note that thanks to average-pooling (in contrast to max-pooling as it was resolved in [9]) we potentially preserve some additional information about DOAs count. So, we encoded the set of all previous DOAs which $L_{net}$ should ignore. Analogously as in previous steps, encoded embeddings are repeated and stack with extracted audio features. Next,
we predict DOAs of new events if there are some, threshold lengths, and finally obtain a new stacked-track.

One may ask in which order the localizer should return DOAs. We did not impose any restrictions on that and let the model learn its own internal hidden order. It will be evident later from the training process how it is done. In essence, in each frame the localizer returns DOAs by DOAs in its own fashion until all sound sources have been localized.

## 2.4. Location-Conditioned Classifier

Self-Conditioned SSG Localizer outputs information about DOAs in each time frame written in the stacked-frame format. Location-Conditioned Classifier simply takes each row from stacked-tracks and in each time-frame outputs probabilities of predicted classes conditioned by DOAs. If the localizer predicted that there is no event in a frame, the classifier is conditioned by the origin and it is expected to predict additional special class with index K, where K is the number of classes.

Our classifier similarly as localizer consists of two trainable components:

* Classifier Encoder $C_{enc}: \mathbb{R}^3 \to \mathbb{R}^c$, where c is the hyperparameter denoting the number of new channels (may be different than the one in localizer, but for simplicity we set it to be the same)

* Classifier Network $C_{net}: \mathbb{R}^{t \times F \times C} \to \mathbb{R}^{T \times (K+1)}$, where notation is the same as in the localizer above and where K denotes the number of classes.

# 3. TRAINING PROCESS

In our solution localizer and classifier are trained completely separately.

## 3.1. Self-Conditioned SSG Localizer

Let's say we have an audio chunk and associated meta with N stacked-tracks, where N denotes the maximal number of overlapping events. We select a random integer r from 0 to N - 2 in a uniform way and split stacked-tracks into two parts:

* Conditioning part containing tracks from 0 to r - 1,

* Target part containing tracks from r to N - 1.

The aim of this splitting is to imitate the r'th iteration from the inference, by hiding stacked-tracks with indices $\ge r$.

We feed the Conditioning part into $L_{enc}$ frame-wise in the following way: if there is some event in r - 1'th stacked-track, then we encode all DOAs via $L_{enc}$ and average pool the embeddings. Otherwise, we encode 0 via $L_{enc}$. We then repeat obtained embeddings so many times to be able to stack them with audio features and we feed an acquired tensor through $L_{net}$. Thus, in each time-frame s we obtain new xyz coordinates which we denote by $l_{pred}^s$. We compare the predicted coordinates with the ones from the Target part. In each time frame we compute a $L_{1.5}$ distance between predicted DOAs and the ground truth DOAs from Target part and set the minimum value as our loss. If there are no more active sound sources in Target part we enforce the target to be a zero vector by minimizing $L_{1.5}$ norm of the predicted DOA. I.e.

$$Loss_{loc}^{r,k}(l_{gt}^s, l_{pred}^s) = \begin{cases} \min_{i=r..k} \|l_{gti}^s - l_{pred}^s\|_{1.5} & \text{if } k \ge r \\ \|l_{pred}^s\|_{1.5} & \text{if } k < r \end{cases}$$

where k is the index of the last nonzero DOAs from the Target part of stacked-tracks. During training, given a batch of size B of audio-meta pairs we select random r for each item in the batch, compute $Loss_{loc}^{r,k}$ loss (frame-wise and item-wise) and average it over all k $\le N$ and r $\le k$. I.e. the final loss between $[l_{gt}]_b^t$ and $[l_{pred}]_b^t$ is

$$\frac{2}{N(N+1)+2} \sum_{k \le N} \sum_{r \le k} \frac{1}{|T_{r,k}|} \sum_{s \in T_{r,k}} Loss_{loc}^{r,k}(l_{gt}^s, l_{pred}^s),$$

where $T_{r,k}$ stand for time-frames among whole batch and where k, r were sampled according to the rule described above.

One may ask why we decided to select $L_p$ norm with $p = 1.5$. Since for $p = 2$ the optimal solution is the expected value, there is a possible risk of ignoring the conditioning $L_{enc}$ and averaging xyz outputs in the case of multiple overlapping sound sources. Conversely, for $p = 1$ the optimal solution is the median which may be too "sharp" decision making. Thus $p = 1.5$.

## 3.2. Location-Conditioned Classifier

In the same manner as in the case of the localizer, let's assume that we have some audio-meta pair from a chunk of the sound signal and let N denote the maximal number of overlapping events. We select random number r from 0 to N - 1 and we select r'th row from stacked-meta containing DOA information. We aim to output classes associated with these DOAs. In each time-frame we encode the DOAs from the selected row via $C_{enc}$. Then we repeat the encoded tensor and stack it with audio features. We forward it through $C_{net}$ to obtain K + 1 scores in each time-frame s associated with K classes and one score denoting unknown class or lack of any event. We then simply compare predicted probability scores $p_{pred}^s$ with ground truth classes $c_{gt}^s$ from the selected stacked-track. Since there is great imbalance of classes due to the fact that class associated with unknown event is over-represented, we used focal loss [10] with $\gamma = 1$ instead of regular cross-entropy. I.e.

$$Loss_{cls}(c_{gt}^s, p_{pred}^s) = -(1 - p_{cls,pred}^s) \log(p_{cls,pred}^s).$$

For a batch of size B and T time-frames per item we simply average everything to obtain the final loss for a back-propagation.

## 3.3. Angle perturbation

One drawback of our approach is the error propagation. Since localizer and classifier are conditioned by the output of the localizer, the error in DOA predictions downgrade the quality of outputs in the next steps. To partly resolve that issue we decided to perturb angles fed to $L_{enc}$ and $C_{enc}$ during the training as well during the inference. We decided to randomly perturb azimuth and elevation by 5 degrees in each time-frame from meta.

## 3.4. Stack-track permutation

In order to increase the number of training samples we could permute ordinary tracks before stacking into stacked-tracks. However, in our case it was more convenient to operate on stacked-tracks. Thus, for each stacked-track tensor we permute the tracks and stack them back.

In our solution we constrained to N = 3, i.e. we allow up to 3 overlapping audio events. We simply ignore the rest.

## 3.5. Model Architecture

For $L_{net}$ and $C_{net}$ we used the SELDnet-like architecture introduced in [1]. More precisely, for localizer and classifier we took
the baseline model [6] and changed the number of filters in convolutional blocks from 64 to 128. After the last time-distributed dense layer we put at the end another time-distributed dense layer with 3 outputs with tanh activation for localizer and with 13 outputs with softmax activation for classifier.

We set $L_{enc}$ and $C_{enc}$ to be single dense layers with 3 inputs and $c = 5$ outputs with tanh activation.

As for the models complexity, $L_{net}$ and $C_{net}$ have slightly above 2.3 million parameters each while $L_{enc}$ and $C_{enc}$ have just 20 parameters each.

### 3.6. Complexity

During the training of the localizer, we need to compute up to $\frac{N \cdot (N+1)}{2} + 1$ components to the loss given that the maximal number of overlapping sound sources is $N$. For the classifier, the complexity scales linearly with $N$. This contrasts with PIT, where in principle we need to compute $N!$ components to the loss. During the inference as described in Section 2, our method requires up to $N$ steps for both localizer and classifier.

### 3.7. Features and augmentations

We used 24kHz FOA format for our nets. We extracted complex spectrograms from each four FOA channels using Short Time Fourier Transform (STFT) with $n_{fft} = 1024$ and Hanning window and hop length of 960 and 480 respectively. From each obtained spectrogram we acquire the log-power spectrogram and the phase spectrogram. For the last three channels we used intensity vectors as it was done in [11]. In summary, from each FOA audio signal we acquire 11 audio features of size $t \times 513$, where $t$ is the number of time bins from the STFT. In our case we randomly selected 5s audio chunks from the recordings which constitutes of 250 time bins.

For data augmentation we used volume perturbation by selecting a random number between 0.5 and 1.5 and multiplying all audio channels by that number. We also used FOA domain spatial augmentation [12] to augment every fourth audio-meta pair.

We trained localizer and classifier using Adam optimizer [13] with default parameters except learning rate which we set to be 0.0005. We trained both models for half a million batches of size 96.

## 4. EVALUATION

### 4.1. Metrics

The DCASE2022 Challenge Task3 organizers provided two types of datasets for the development stage [6]:

* Synth: 1200 one-minute synthesized mixtures from collected SRIRs and selected sound events from FSD50K [14]

* STARSS22: 292 minutes of real recordings simulating real life scenarios gathered in 11 rooms in Tokyo and Tempere.

STARSS22 is further split into train and test folds. In this section we will discuss results on the test fold. During the training, for every batch we sampled half of the recordings from Synth dataset and half from the train split from STARSS22, utilizing all the data. We noticed that if we do not use synthetic dataset the scores drop drastically.

For localizer we first report the average DOA error in angles. In Table 1 we show average DOA errors in multiple cases differentiating between number of active sound sources and number of DOAs in conditioning. As excepted, the more DOAs in conditioning, the larger the error. However, what is interesting is the fact that the more sound sources, the more accurate the model is in detecting the first few DOAs.

<table><caption>Table 1: Dependence of the localizer's average DOA error (in degrees) on the number of active sources (noas) and the number of DOAs in conditioning (#cond) on STARSS22 test split.</caption><thead><tr><th rowspan="2">noas</th><th colspan="3">#cond</th></tr><tr><th>0</th><th>1</th><th>2</th></tr></thead><tbody><tr><td>1</td><td>23</td><td></td><td></td></tr><tr><td>2</td><td>20</td><td>33</td><td></td></tr><tr><td>3</td><td>14</td><td>19</td><td>51</td></tr></tbody></table>

For the classifier we report conditional accuracy (CAcc) in which we count the class $c$ with maximal probability $P(c|X, l)$ correct if that class corresponds to the conditioning DOA $l$. Otherwise we treat it as an incorrect prediction. On the STARSS22 test split we achieved CAcc of 68%. We also kept track of the number of frames where the classifier misses known classes and of frames where the classifier predicted a known class where there is none, but for those cases the classifier achieve almost perfect scores.

Finally, we report our results on official DCASE2022 Task3 metrics [6], namely: the localization-dependent error rate $ER_{20\circ}$, F1-score $F_{20\circ}$, the localization error $LE_{CD}$ and the localization recall $LR_{CD}$. We compare the baseline system with two versions of our solution. In the first one we will predict up to 3 DOAs and corresponding classes, and in the second one we terminate inference on 2 events. We denote these solutions as $max_{ov3}$ and $max_{ov2}$ respectively. Note that we used the very same models in both solutions, only the inference changes. We summarise the results in Table 2. In most metrics our solution outperforms the baseline system. The only one when our solution is lacking is the error rate. Furthermore, the error rate is worse when we try to infer more events. We speculate that this is due to the fact that the localizer is very inaccurate in later steps (see Table 1), which may generate many false positives.

<table><caption>Table 2: Official metrics; the boldface denotes the best scores.</caption><thead><tr><th></th><th>ER<sub>20°</sub></th><th>F<sub>20°</sub></th><th>LE<sub>CD</sub></th><th>LR<sub>CD</sub></th></tr></thead><tbody><tr><td>Baseline</td><td><strong>0.71</strong></td><td>21%</td><td>29.3°</td><td>46%</td></tr><tr><td>max_ov3</td><td>0.85</td><td>32%</td><td>24.7°</td><td><strong>51%</strong></td></tr><tr><td>max_ov2</td><td>0.76</td><td><strong>33%</strong></td><td><strong>24.6°</strong></td><td>49%</td></tr></tbody></table>

## 5. CONCLUSION

In this paper we presented alternative solution to the SELD problem. Our solution uses our custom SSG method to determine DOAs one by one and as a result to determine number of sound sources in each time frame. The localizer is followed by the location-conditioned classifier. The performance of our method is comparable to the DCASE2022 Task3 baseline system which uses PIT. Thus, we imagine that in the future SSG and PIT may be combined together to obtain the best of both worlds.
## 6. REFERENCES

[1] S. Adavanne, A. Politis, J. Nikunen, and T. Virtanen, "Sound event localization and detection of overlapping sources using convolutional recurrent neural networks," *IEEE Journal of Selected Topics in Signal Processing*, vol. 13, no. 1, pp. 34–48, 2018.

[2] Y. CAO, T. IQBAL, Q. Kong, Y. ZHONG, W. WANG, and M. D. PLUMBLEY, "Event-independent network for polyphonic sound event localization and detection," in *Detection and Classification of Acoustic Scenes and Events 2020 Workshop (DCASE 2020)*. University of Surrey.

[3] K. Shimada, Y. Koyama, S. Takahashi, N. Takahashi, E. Tsunoo, and Y. Mitsufuji, "Multi-accdoa: Localizing and detecting overlapping sounds from the same class with auxiliary duplicating permutation invariant training," in *ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. IEEE, 2022, pp. 316–320.

[4] https://dcase.community/challenge2022.

[5] S. Kapka and M. Lewandowski, "Sound source detection, localization and classification using consecutive ensemble of crnn models," in *Acoustic Scenes and Events 2019 Workshop (DCASE2019)*, 2019, p. 119.

[6] A. Politis, K. Shimada, P. Sudarsanam, S. Adavanne, D. Krause, Y. Koyama, N. Takahashi, S. Takahashi, Y. Mitsufuji, and T. Virtanen, "Starss22: A dataset of spatial recordings of real scenes with spatiotemporal annotations of sound events," 2022. [Online]. Available: https://arxiv.org/abs/2206.01948

[7] V. Vapnik, *The nature of statistical learning theory*. Springer science & business media, 1999.

[8] K. Shimada, Y. Koyama, N. Takahashi, S. Takahashi, and Y. Mitsufuji, "Accdoa: Activity-coupled cartesian direction of arrival representation for sound event localization and detection," in *ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. IEEE, 2021, pp. 915–919.

[9] M. Zaheer, S. Kottur, S. Ravanbakhsh, B. Poczos, R. R. Salakhutdinov, and A. J. Smola, "Deep sets," *Advances in neural information processing systems*, vol. 30, 2017.

[10] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, "Focal loss for dense object detection," in *Proceedings of the IEEE international conference on computer vision*, 2017, pp. 2980–2988.

[11] Y. Cao, T. Iqbal, Q. Kong, M. Galindo, W. Wang, and M. Plumbley, "Two-stage sound event localization and detection using intensity vector and generalized cross-correlation," DCASE2019 Challenge, Tech. Rep., June 2019.

[12] L. Mazzon, M. Yasuda, Y. Koizumi, and N. Harada, "Sound event localization and detection using foa domain spatial augmentation," in *Proc. of the 4th Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE)*, 2019.

[13] D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization," 2014. [Online]. Available: https://arxiv.org/abs/1412.6980

[14] E. Fonseca, X. Favory, J. Pons, F. Font, and X. Serra, "Fsd50k: an open dataset of human-labeled sound events," *IEEE/ACM Transactions on Audio, Speech, and Language Processing*, vol. 30, pp. 829–852, 2021.
# MODEL TRAINING THAT PRIORITIZES RARE OVERLAPPED LABELS FOR POLYPHONIC SOUND EVENT DETECTION

Rie Koga¹, Sawa Takamuku¹, Keisuke Imoto², Naotake Natori¹

¹ AISIN CORPORATION, 1-1-20 Aomi, Koto-ku, Tokyo, Japan,
{rie.yamada, sawa.takamuku, naotake.natori}@aisin.co.jp

² Doshisha University, 1-3 Tatara Miyakodani, Kyotanabe, Kyoto, Japan,
keisuke.imoto@ieee.org

## ABSTRACT

In this study, we propose a model training method for polyphonic sound event detection (polyphonic SED) that prioritizes rare event label frames during multiple overlapping sound events. Multi-label classification typically utilized in polyphonic SED often fails to recognize such events. To overcome this problem, the proposed method is designed to represent event overlaps of rare labels easily without a complicated network structure. During model training, we periodically apply either binary cross-entropy loss (BCE) for multi-label classification or softmax cross-entropy loss (Softmax-CE) for multi-class classification. When multi-class classification is performed using Softmax-CE, the labels of the overlapping frames are reconstructed from the target labels to include the rarest ones and exclude the frequent ones. The model was evaluated on strongly labeled AudioSet data, from which only human voice segments were extracted. The proposed method achieves an improvement of 0.23 percentage points over the baseline, which only used the BCE, in terms of the mean average precision. In particular, the proposed method outperforms the baseline with respect to rare labels, with an average precision of 1.18 percentage points. The experimental results also demonstrate the effectiveness of the proposed method for both overlap of sound events and rare labels.

**Index Terms—** Polyphonic sound event detection, multi-label classification, multi-class classification

## 1. INTRODUCTION

Due to the advancements in deep learning, sound event detection (SED), which is a technique used for estimating the type and interval (onset and offset times) of sound events present in an acoustic signal, has recently attracted attention. Additionally, shared mobility services have become ubiquitous in many cities worldwide. For safety, they require surveillance of both the drivers and passengers inside the vehicles [1, 2]. In an in-vehicle surveillance system, various sound events must be detected to understand what is occurring inside the vehicle. Therefore, this study focused on human voice SED for an in-vehicle surveillance system based on human voice signals.

Some DCASE competitions [3, 4] have previously dealt with the sound of human speech or crying babies, where target sounds are often overlapped, whereas real-world data often suffer from extreme imbalances between classes as well as overlapping sound events. For example, as shown in Fig. 1, in the strongly labeled AudioSet [5] dataset annotated using real-world data, despite focusing on the top seven classes with the highest number of event frames

<figure><img src="image_43.png" /><figcaption>Figure 1: Number of frames in training data.</figcaption></figure>

among the 14 “human voice” classes, the number of frames among the classes is imbalanced, with a ratio of approximately 100 to 1 between the most common and the rarest.

Many studies have treated SED as a multi-label classification problem for handling overlapping events that often cause detection errors [6, 7, 8]. [9] represented event overlaps by linking a bivariate probability distribution based on time and frequency with class-wise hidden Markov models. In [10], a non-negative matrix decomposition-based method that jointly trained a dictionary and a multinomial logistic regression classifier was used to manage the overlap of sounds. In [11], an event independent network for SED and localization was developed with a track-wise output. In polyphonic sound event detection (polyphonic SED) with deep learning, [12] performed multi-class classification by considering all possible overlapping event combinations as classes. However, the model architecture requires significant modification to manage a multi-class multi-tasking problem. Therefore, the conventional method [12] cannot be applied to the current polyphonic SED system without any modification of the network architecture.

Binary cross-entropy (BCE) loss is often employed as the loss function of multi-label classification in polyphonic SED. However, SED using BCE often falls into imbalance between sound event classes when training an SED model. Therefore, when applied to real-world data, the accuracy of rare class event detection decreases. Specifically, accurately detecting anomalous or rare sounds such as “Screaming” is more important than detecting common sounds such as “Speech,” as shown in Fig. 1. Several loss functions that are effective for imbalanced data have been proposed in polyphonic SED. [13] proposed asymmetric focal loss and focal batch Tversky loss; however, these mainly address the imbalance problem between negative and positive samples. [14] proposed time-balanced focal loss, which is highly dependent on the dataset because the class weights used in the loss function are adjusted as hyperparameters.

Therefore, without modifying the original model architecture
or preparing the class weights, we propose a method that periodically uses multi-label classification based on BCE and multi-class classification to prioritize rare classes as target labels when sound events overlap.

The contributions of this study can be summarized as follows:

* We propose a new model training method for detecting overlapped and rare sound events. The proposed method combines multi-class classification, in which rare classes are preferentially learned as target labels, along with multi-label classification. We then confirm the efficacy of this method.

* We reconstructed a strongly labeled AudioSet using seven sound event classes with “Human voice” at the upper level. We conducted a baseline evaluation for an SED task covering multiple types of human vocalization with these classes.

## 2. DATASET

Based on AudioSet's strong labels [5], we created a new dataset comprising 10 seconds of audio taken from the soundtrack of a YouTube video, with approximately 67,000 clips for training and 18,000 clips for evaluation. The strongly labeled AudioSet ontology is a hierarchy of 356 sound event classes. The sound classes selected for this study were the following seven event classes within the “Human voice” class: “Speech,” “Singing,” “Laughter,” “Shout,” “Crying, sobbing (Crying),” “Screaming,” and “Whispering.” In cases where the selected classes have subclasses, the subclasses are merged into the superclass. For example, subclasses “Baby cry, infant cry” and “Whimper” are merged into a superclass “Crying.” Sound clips with other sound events in “Human voice,” such as “Humming” or “Yawn,” were not used in the dataset because there were few events in each class.

When sound clips contain other sound events from the category non-“Human voice,” such as “Music” or “Hands,” the clips were still used. However, these sound events were only background noise, that is, they were not used as target labels. After extracting the dataset to contain the selected sound class for each audio clip, the dataset contained 50,650 sound clips for training and 8,747 sound clips for evaluation. Note that in this study, rare labels (“Screaming” and “Whispering”) were defined as appearing with approximately 1% of the frequency of the most frequent label.

## 3. PROPOSED METHOD

In this section, we first describe the loss functions used in this study for the multi-label and multi-class classification tasks. Next, we discuss a new model training method combining those loss functions. Finally, we describe a method of label selection for multi-class classification using polyphonic SED.

### 3.1. Loss function

Generally, a sigmoid activation function-based BCE is employed in training polyphonic SED models.

$$f(s)_{i,j} = \frac{1}{1 + e^{-s_{i,j}}} \qquad (1)$$

$$\mathcal{L}_{\text{BCE}} = - \sum_{i}^{C} \sum_{j}^{T} \{y_{i,j} \log(f(s)_{i,j}) + (1 - y_{i,j}) \log(1 - f(s)_{i,j})\}, \quad (2)$$

<table><thead><tr><th colspan="5">Predicted probabilities at epoch i</th></tr><tr><th>Class</th><th>Time [s]</th><th>0.0</th><th>0.1</th><th>0.2</th><th>10.0</th></tr></thead><tbody><tr><td>No sound</td><td>0.8</td><td>0.9</td><td>0.4</td><td>...</td><td>0.3</td></tr><tr><td>Speech</td><td>0.7</td><td>0.1</td><td>0.3</td><td>...</td><td>0.6</td></tr><tr><td>Singing</td><td>0.5</td><td>0.7</td><td>0.9</td><td>...</td><td>0.9</td></tr><tr><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td></tr><tr><td>Whispering</td><td>0.3</td><td>0.5</td><td>0.1</td><td>...</td><td>0.1</td></tr></tbody></table>

If *i* is NOT divisible by *N*, Otherwise

<table><caption>Actual labels for multi-label classification</caption><thead><tr><th>Class</th><th>Time [s]</th><th>0.0</th><th>0.1</th><th>0.2</th><th>...</th><th>10.0</th></tr></thead><tbody><tr><td>No sound</td><td>0</td><td>0</td><td>1</td><td>0</td><td>...</td><td>0</td></tr><tr><td>Speech</td><td>1</td><td>0</td><td>0</td><td>0</td><td>...</td><td>0</td></tr><tr><td>Singing</td><td>1</td><td>0</td><td>1</td><td>...</td><td>0</td><td>0</td></tr><tr><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td></tr><tr><td>Whispering</td><td>1</td><td>0</td><td>1</td><td>...</td><td>1</td><td>0</td></tr></tbody></table>

Multi-label classification for each class at each timestep

<table><caption>Actual labels for multi-class classification</caption><thead><tr><th>Class</th><th>Time [s]</th><th>0.0</th><th>0.1</th><th>0.2</th><th>...</th><th>10.0</th></tr></thead><tbody><tr><td>No sound</td><td>0</td><td>0</td><td>1</td><td>0</td><td>...</td><td>0</td></tr><tr><td>Speech</td><td>0</td><td>0</td><td>0</td><td>0</td><td>...</td><td>0</td></tr><tr><td>Singing</td><td>0</td><td>0</td><td>0</td><td>0</td><td>...</td><td>0</td></tr><tr><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td></tr><tr><td>Whispering</td><td>1</td><td>0</td><td>1</td><td>...</td><td>1</td><td>0</td></tr></tbody></table>

Multi-class classification at each timestep

<caption>Figure 2: Overview of the proposed model training method.</caption>

where *f* is the sigmoid function, $s_{i,j}$ is the *i*th class's *j*th time frame logit, $y_{i,j}$ is the *i*th class's *j*th time frame's target label, *C* is the total number of classes, and *T* is the total number of time frames.

Conversely, a softmax activation function-based cross-entropy (Softmax-CE) loss for a multi-class classification is employed in monophonic SEDs, to choose one event from multiple sound event classes.

$$g(s)_{i,j} = \frac{e^{s_{i,j}}}{\sum_{k}^{C} e^{s_{k,j}}} \qquad (3)$$

$$\mathcal{L}_{\text{Softmax-CE}} = - \sum_{i}^{C} \sum_{j}^{T} y_{i,j} \log(g(s)_{i,j}), \quad (4)$$

where *g* is the softmax function.

### 3.2. Training process

In this study, we applied alternately either BCE or Softmax-CE within a defined period, as shown as Fig. 2. Specifically, Softmax-CE was applied while using multi-class classification every *N* epoch. When the model was trained by Softmax-CE, it was trained by BCE and saved at the next epoch. Then, the model for evaluation was only used at the epoch with the minimum validation loss. For example, when multi-class classification was performed at every third epoch, i.e., at the 3, 6, 9, ..., $3^i$ epochs, the validation loss was monitored at the 4, 7, 10, ..., ($3^i+1$) epochs, with *i* being a positive integer.

### 3.3. Label selection for multi-class classification

Two problems are often encountered when performing multi-class classification for polyphonic SED because multi-class classification always requires one target label during loss computation. The first problem is determining which label to allocate when multiple sound events occur simultaneously. The second problem is determining which label to allocate when none of the target sound events occur. To solve these problems, we propose a new method of label selection using the sparsity of sound events. Specifically, we prioritize the rarest label for multiple events and define a new class label for no events. The details are discussed below.
<figure><img src="image_44.png" /><figcaption>(a) multi-label classification.</figcaption></figure>

<figure><img src="image_45.png" /><figcaption>(b) multi-class classification.</figcaption></figure>

<caption>Figure 3: Overview of the target labels. (a) multi-label classification and (b) multi-class classification.</caption>

<table><caption>Table 1: Number of frames for each sound event.</caption><thead><tr><th rowspan="2">Event class</th><th colspan="2">The number of frames</th></tr><tr><th>Multi-label</th><th>Multi-class</th></tr></thead><tbody><tr><td>Speech</td><td>9,083,336</td><td>8,627,958</td></tr><tr><td>Singing</td><td>2,781,533</td><td>2,690,509</td></tr><tr><td>Laughter</td><td>683,036</td><td>666,186</td></tr><tr><td>Shout</td><td>612,058</td><td>607,208</td></tr><tr><td>Crying</td><td>174,120</td><td>173,731</td></tr><tr><td>Screaming</td><td>85,578</td><td>85,578</td></tr><tr><td>Whispering</td><td>80,194</td><td>80,194</td></tr></tbody></table>

**Multiple events** This section describes the method to allocate sound event labels when multiple overlapping sound events occur concurrently during a single clip, as shown in Fig. 3. In Fig. 3, multiple sound events overlap as follows.

* 1.6 - 3.2 seconds : Speech and Shout

* 7.1 - 8.3 seconds : Laughter and Shout

* 8.3 - 8.9 seconds : Speech, Laughter, and Shout

* 8.9 - 10.0 seconds : Speech and Laughter

We adopted the rarest label for each time frame in the clip. In the example in Fig. 3, when multi-class classification was performed, the labels in bold were used. The number of frames for each label in the training data when changing from the labels used in multi-label classification to those used in multi-class classification is shown in Table 1. Frequent labels such as “Speech” and “Singing” show a large decrease in the number of frames when compared with the number of rare labels.

**No events** In multi-label classification, BCE originally includes calculation of inactive frames. However, in multi-class classification, even when none of the seven target labels exist in a frame, one class must be set as the target label. Therefore, a new class “No sound” was created and allocated to time frames containing no target class. Fig. 3(b) shows the “No sound” class with thick black lines between 3.3 - 3.5 seconds and 5.2 - 5.5 seconds.

# 4. EXPERIMENT

## 4.1. Experimental setups

The AudioSet sound clips were downloaded from YouTube. These sounds were mostly monaural. The left and right sides of the stereophonic sounds were averaged to produce monaural sounds. The

Table 2: Model architecture. The kernel sizes of the convolutional and pooling layer are denoted as “Conv (kernel size)” and “Max Pooling (kernel size),” respectively. The number of attention heads is denoted as “Transformer Encoder (number of attention heads).”

<table><thead><tr><th>Conv3</th><th>RB</th></tr></thead><tbody><tr><td colspan="2">Log-mel spectrogram<br>500 frames × 64 mel bins</td></tr><tr><td>Conv (3 × 3)<br>BN, ReLU, Dr<br>Max Pooling (8 × 1)</td><td>Conv (3 × 3)<br>BN, ReLU<br>ResBlock</td></tr><tr><td>Conv (3 × 3)<br>BN, ReLU, Dr<br>Max Pooling (4 × 1)</td><td>ResBlock</td></tr><tr><td>Conv (3 × 3)<br>BN, ReLU, Dr<br>Max Pooling (2 × 1)</td><td>ResBlock</td></tr><tr><td colspan="2">(Transformer Encoder (32)) × 2</td></tr><tr><td colspan="2">FC, Sigmoid</td></tr></tbody></table>

sounds were resampled to 44.1 kHz, as previously configured [7]. The sounds were then converted into a logmel scale of $F = 64$ filters calculated every 40 milliseconds with a hop size of 20 milliseconds.

Inspired by [13], a convolutional neural network (CNN)-transformer-based network was used as the model architecture. This architecture performs better than the CNN-biGRU-based network, which is widely used in SED [7, 15, 16]. The model architecture is shown in Table 2. The system has two types of CNN backbones: one with three Convolution layers (Conv3) and the other with three ResBlocks (RB). The parameters of the convolutional layers in RB are the same as those of Conv3.

The models were trained using the RAdam optimizer [17] with a learning rate of 0.001. Early stopping was implemented after 50 epochs if no improvement on validation loss was noted.

As evaluation metrics, we used the mean average precision (mAP), the micro average precision (micro-AP), and the frame-based macro- and micro-Fscores with a threshold for prediction of 0.5. Even when the proposed method was deployed with eight event classes including “No sound,” we evaluated them using only the original seven classes. In this study, we used eight classes when performing both the multi-label and multi-class classification using the proposed method.
<table><caption>Table 3: Average SED performance for two backbones.</caption><thead><tr><th rowspan="2">Method</th><th colspan="4">Conv3</th><th colspan="4">RB</th></tr><tr><th>mAP</th><th>macro-Fscore</th><th>micro-AP</th><th>micro-Fscore</th><th>mAP</th><th>macro-Fscore</th><th>micro-AP</th><th>micro-Fscore</th></tr></thead><tbody><tr><td>baseline</td><td>51.96%</td><td>38.16%</td><td>83.39%</td><td>73.83%</td><td>58.04%</td><td>50.06%</td><td>85.48%</td><td>77.11%</td></tr><tr><td>AFL</td><td>50.92%</td><td>36.92%</td><td>82.90%</td><td>73.19%</td><td>59.05%</td><td>52.15%</td><td>85.66%</td><td>77.56%</td></tr><tr><td>e0</td><td>51.60%</td><td>39.15%</td><td>83.25%</td><td>72.72%</td><td>58.22%</td><td>49.71%</td><td>85.35%</td><td>76.82%</td></tr><tr><td>e1</td><td>51.51%</td><td>37.03%</td><td>79.59%</td><td>50.88%</td><td>56.28%</td><td>32.97%</td><td>80.39%</td><td>47.36%</td></tr><tr><td>e2 (proposed)</td><td><strong>52.19%</strong></td><td><strong>40.41%</strong></td><td><strong>83.55%</strong></td><td><strong>74.65%</strong></td><td><strong>60.32%</strong></td><td><strong>53.40%</strong></td><td><strong>86.65%</strong></td><td>78.52%</td></tr><tr><td>e3 (proposed)</td><td>51.63%</td><td>38.20%</td><td>83.11%</td><td>73.49%</td><td>59.51%</td><td>50.96%</td><td>86.47%</td><td>78.38%</td></tr><tr><td>e4 (proposed)</td><td>51.54%</td><td>38.59%</td><td>83.27%</td><td>73.48%</td><td>59.89%</td><td>52.70%</td><td>86.59%</td><td><strong>78.58%</strong></td></tr><tr><td>e5 (proposed)</td><td>51.28%</td><td>38.25%</td><td>83.14%</td><td>73.49%</td><td>59.66%</td><td>51.35%</td><td>86.36%</td><td>78.28%</td></tr></tbody></table>

<table><caption>Table 4: Average SED performance of the rare-event labels for the two backbones.</caption><thead><tr><th rowspan="2"></th><th rowspan="2">Method</th><th colspan="2">Conv3</th><th colspan="2">RB</th></tr><tr><th>AP</th><th>Fscore</th><th>AP</th><th>Fscore</th></tr></thead><tbody><tr><td rowspan="3">Screaming</td><td>baseline</td><td>17.29%</td><td>3.57%</td><td>27.04%</td><td><strong>17.03%</strong></td></tr><tr><td>AFL</td><td>16.46%</td><td>2.92%</td><td>28.08%</td><td>15.14%</td></tr><tr><td>e2 (proposed)</td><td><strong>18.73%</strong></td><td><strong>4.01%</strong></td><td><strong>28.37%</strong></td><td>13.43%</td></tr><tr><td rowspan="3">Whispering</td><td>baseline</td><td>55.86%</td><td>42.91%</td><td>64.68%</td><td>61.64%</td></tr><tr><td>AFL</td><td>55.03%</td><td>40.55%</td><td>65.67%</td><td>60.81%</td></tr><tr><td>e2 (proposed)</td><td><strong>56.79%</strong></td><td><strong>45.19%</strong></td><td><strong>67.34%</strong></td><td><strong>65.64%</strong></td></tr><tr><td rowspan="3">Avg.</td><td>baseline</td><td>36.58%</td><td>23.24%</td><td>45.86%</td><td>39.34%</td></tr><tr><td>AFL</td><td>35.74%</td><td>21.73%</td><td>46.87%</td><td>37.98%</td></tr><tr><td>e2 (proposed)</td><td><strong>37.76%</strong></td><td><strong>24.60%</strong></td><td><strong>47.85%</strong></td><td><strong>39.54%</strong></td></tr></tbody></table>

## 4.2. Experimental results

Table 3 shows the results for the baseline using only multi-label classification and the proposed method that performed multi-label classification while using multi-class classification every *N* epochs. Each result is the average of five iterations. The value of *N* in *eN* represents frequency of switching to Softmax-CE. Here, *e0* was trained using multi-task learning, where multi-label classification and multi-class classification were performed simultaneously every epoch. Conversely, *e1* was trained using Softmax-CE every epoch and evaluated as a multi-label classification. The baseline method was performed on the original seven-class multi-label classification without the “No sound” class, and the proposed method was performed on the eight-class multi-label classification and multi-class classification including the “No sound” class. When *e2*, i.e., multi-label classification and multi-class classification, was used independently of the backbone for every other epoch, it demonstrated the best performance on several metrics. The proposed method improved mAP by 0.23 percentage points and 2.28 percentage points for Conv3 and RB, respectively. This result demonstrates that the proposed method improves the performance of rare events by using Softmax-CE and retains the performance of frequent events by using BCE. Meanwhile, the simultaneous use of Softmax-CE and BCE in *e0* could have prevented the influence of Softmax-CE. With an increase in *N*, the performance of several metrics gradually decreases, and at *e5*, the performance is comparable to the baseline values. When Softmax-CE is used less frequently, it becomes less effective.

Table 4 shows the results when focusing on rare labels (“Screaming” and “Whispering”). Comparisons were made with

asymmetric focal loss (AFL), which has an effect on the imbalanced data [13]. There is a significant difference between “Screaming” and “Whispering.” “Screaming,” which is similar to “Shout” and “Singing,” is more likely to occur in noisy environments, whereas “Whispering” is a special type of sound event where other sound events are unlikely to occur concurrently. As with the overall performance, *e2* performed highest on many measures, but the Fscore for “Screaming” dropped significantly. “Screaming” had a lower Fscore, which was based on the threshold, because underfitting caused by a rare label reduced the predicted probability of “Screaming.” However, because *e2* performed the highest, when comparing under the same conditions, the intrinsic prediction performance of *e2* is superior and is effective for rare labels. Unlike the original experimental dataset used for evaluating AFL, the number of data classes (seven) was limited due to an imbalance of approximately 1 to 100 in a class. This may have contributed to the *e2* performance being superior to that of AFL.

## 5. CONCLUSION

We proposed a method of polyphonic SED by periodically using either multi-label or multi-class classification. Based on the sparsity of sound events, multi-class classification was used to strongly train rare sound event labels, in which the rarest sound event was selected as the label representing each frame. The proposed method was evaluated on a human voice dataset extracted from the strongly labeled AudioSet data. Our approach was found to be most effective when the two loss functions were alternately applied. For the imbalanced data, regarding both the overall metrics and for rare labels, this method significantly outperformed the conventional methods.

## 6. REFERENCES

[1] G. McKenzie, “Urban mobility in the sharing economy: A spatiotemporal comparison of shared mobility services,” *Computers, Environment and Urban Systems*, vol. 79, p. 101418, 2020.

[2] A. Mishra, S. Lee, D. Kim, and S. Kim, “In-cabin monitoring system for autonomous vehicles,” *Sensors*, vol. 22, no. 12, 2022.

[3] A. Mesaros, A. Diment, B. Elizalde, T. Heittola, E. Vincent, B. Raj, and T. Virtanen, “Sound event detection in the dcase 2017 challenge,” *IEEE/ACM Transactions on Audio, Speech, and Language Processing*, vol. 27, no. 6, pp. 992–1006, 2019.
[4] A. Politis, S. Adavanne, and T. Virtanen, "A dataset of reverberant spatial sound scenes with moving sources for sound event localization and detection," in Proc. 5th Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE), 2020, pp. 165-169.

[5] S. Hershey, D. P. W. Ellis, E. Fonseca, A. Jansen, C. Liu, R. Channing Moore, and M. Plakal, "The benefit of temporally-strong labels in audio event classification," in Proc. 46th IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021, pp. 366-370.

[6] T. N. T. Nguyen, K. N. Watcharasupat, Z. J. Lee, N. K. Nguyen, D. L. Jones, and W. S. Gan, "What makes sound event localization and detection difficult? insights from error analysis," in Proc. 6th Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE), 2021, pp. 120-124.

[7] E. Çakır, G. Parascandolo, T. Heittola, H. Huttunen, and T. Virtanen, "Convolutional recurrent neural networks for polyphonic sound event detection," IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 25, no. 6, pp. 1291-1303, 2017.

[8] H. Phan, O. Y. Chén, P. Koch, L. Pham, I. McLoughlin, A. Mertins, and M. D. Vos, "Unifying isolated and overlapping audio event detection with multi-label multi-task convolutional recurrent neural networks," in Proc. 44th IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2019, pp. 51-55.

[9] E. Benetos, G. Lafay, M. Lagrange, and M. D. Plumbley, "Detection of overlapping acoustic events using a temporally-constrained probabilistic model," in Proc. 41th IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2016, pp. 6450-6454.

[10] V. Bisot, S. Essid, and G. Richard, "Overlapping sound event detection with supervised nonnegative matrix factorization," in Proc. 42th IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2017, pp. 31-35.

[11] Y. Cao, T. Iqbal, Q. Kong, F. An, W. Wang, and M. D. Plumbley, "An improved event-independent network for polyphonic sound event localization and detection," in Proc. 46th IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021, pp. 885-889.

[12] H. Phan, T. N. T. Nguyen, P. Koch, and A. Mertins, "Polyphonic audio event detection: Multi-label or multi-class multi-task classification problem?" in Proc. 47th IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022, pp. 8877-8881.

[13] K. Imoto, S. Mishima, Y. Arai, and R. Kondo, "Impact of sound duration and inactive frames on sound event detection performance," in Proc. 46th IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021, pp. 860-864.

[14] S. Park and M. Elhilali, "Time-balanced focal loss for audio event detection," in Proc. 47th IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022, pp. 311-315.

[15] R. Serizel, N. Turpault, H. Eghbal-Zadeh, and A. P. Shah, "Large-scale weakly labeled semi-supervised sound event detection in domestic environments," in Proc. 3rd Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE), 2018, pp. 19-23.

[16] D. de Benito-Gorrón, D. Ramos, and D. T. Toledano, "An analysis of sound event detection under acoustic degradation using multi-resolution systems," Applied Sciences, vol. 11, no. 23, 2021.

[17] L. Liu, H. Jiang, P. He, W. Chen, X. Liu, J. Gao, and J. Han, "On the variance of the adaptive learning rate and beyond," in Proc. 8th International Conference for Learning Representations (ICLR), 2020.
# ANALYZING THE EFFECT OF EQUAL-ANGLE SPATIAL DISCRETIZATION ON SOUND EVENT LOCALIZATION AND DETECTION

Saksham Singh Kushwaha¹, Iran R. Roman²*, Juan Pablo Bello²,³

¹ Courant Institute of Mathematical Sciences, New York University, NY, USA

² Music and Audio Research Lab, New York University, NY, USA

³ Center for Urban Science and Progress, New York University, NY, USA

## ABSTRACT

Sound event localization and detection (SELD) models detect and localize sound events in space and time. Datasets for SELD often discretize spatial sound events along the polar coordinates of azimuth (integers from -180° to 180°) and elevation (integers from -90° to 90°). This discretization, known as equal-angle, results in more dense points at the poles (±90° elevation) than at the equator (0° elevation). We first analyzed the effect of equal-angle discretization on the 2022 DCASE SELD baseline model. Since the STARSS 2022 dataset that accompanies the model shows unbalanced sampling of spatial sound events along the elevation axis, we created a synthetic dataset. Our dataset has spatial sound events uniformly distributed along the elevation axis. We created two versions: one with targets spatially discretized using equal-angle, and another one with a uniform spatial discretization (both versions had the same audio). The model trained with equal-angle showed a greater angular localization error for targets around the equator compared to the poles, while the model trained with uniform spatial discretization showed a uniform localization error along the elevation axis. To train the model with the STARSS2022 dataset and reduce the effect of its equal-angle-discretized targets, we modified the model's loss function to penalize localization errors above an angular distance threshold around each target. Using this loss we fine-tuned a model trained with the original loss, and also trained the same model from scratch. Results showed improved localization metrics in both models compared to baseline, while retaining classification metrics. Our results show that equal-angle discretization yields models with non-uniform localization errors for targets along the elevation axis. Finally, our proposed loss function penalizes the SELD model's angular localization errors, regardless of which spatial discretization was used to annotate the dataset targets.

**Index Terms—** sound event localization and detection, spatial sampling, activity-coupled Cartesian direction of arrival, DCASE

## 1. INTRODUCTION

Sound event localization and detection (SELD) consists of localizing sound events in space and time while also assigning them to a class label [1]. SELD can be applied for environmental sound classification [2], simultaneous localization and mapping for navigation without visual input or with occluded targets [3, 4], tracking of sound sources of interest [5], audio surveillance [6], and acoustic imaging [7]. As a result, there has been an increased interest in SELD modeling, and research communities have organized challenges to centralize efforts and advancements [8, 9, 10].

<figure><img src="image_46.png" /><figcaption>Figure 1: Two types of spatial discretization of points on a sphere. (A) equal-angle and (B) Fibonacci. (A) results in denser points at the poles than at the equator, while (B) does not.</figcaption></figure>

The Detection and Classification of Acoustic Scenes and Events (DCASE) community introduced its annual SELD challenge in 2019 [10]. The DCASE SELD challenge provides participants with multichannel audio recordings of categorical sound events (i.e. speech, footsteps, running water, etc.) and their spatiotemporal trajectories on the azimuth and elevation axes. Participants must develop methods to detect, localize, and classify each event. The DCASE SELD challenge also provides a baseline model, which reflects key incremental advancements from the community. For example, a significant advancement has been in the training loss function, which went from separately measuring sound event detection (SED) and direction-of-arrival (DOA) [1] to jointly carrying out these using a mean-squared-error (MSE) regressor that accounts for overlapping sound events categories [11].

SELD datasets often have spatial targets that are discretized in a sphere along the elevation ($\theta = [-90, 90] \in \mathbb{Z}$) and azimuth ($\phi = [-180, 180] \in \mathbb{Z}$) axes in units of degrees [1, 12, 13, 14]. This sampling of points in space, known as equal-angle, is easy to interpret because it yields uniform-looking grids on a 2D projection ($\phi$ vs $\theta$). However, equal-angle points on the sphere shows a larger density of points at the poles (±90° elevation) than at the equator (0° elevation) [15]. Furthermore, equal-angle sampling results in larger quantization errors around the equator.

As far as we know, current SELD research has not studied how a non-uniform density of points along the elevation axis impacts model performance. We hypothesize that training SELD models with equal-angle discretized data results in non-uniform localization performance along the elevation axis. This paper empirically analyzes the impact of equal-angle spatial discretization on SELD model performance and proposes a practical way to mitigate it.

<footnote>*corresponding author email: roman@nyu.edu</footnote>
## 2. EQUAL-ANGLE DISCRETIZATION IS IRREGULAR

A sphere can be discretized into $N$ points using a sampling function $s(\theta, \phi)$, resulting in the set of vectors $\{x_1, x_2, \dots, x_N\} \subset \mathbb{S}^2$. This set represent a lattice of directions around the sphere [16]. For the specific case of equal-angle spatial discretization, $N = N_E \times N_A$, where $N_E$ is the number of points uniformly sampled along the elevation axis $\theta \in [-\frac{\pi}{2}, \frac{\pi}{2}]$ and $N_A$ is the number points along the azimuth axis $\phi \in [0, 2\pi)$ (see supplement S1¹ for a detailed mathematical description of the equal-angle sampling function and its non-uniform density along the elevation axis).

Figure 1A shows how equal-angle spatial discretization, although regular along the axes of azimuth and elevation, results in a lattice with non-uniform distances between points, particularly noticeable along the elevation axis (i.e. poles versus equator).

It is worth noting that humans listen most events close to the equator (i.e. other speakers or ecologically-relevant sound sources on the azimuth). This introduces another sampling bias in realistic SELD datasets. Thus, equal-angle discretization can yield less-than-ideal resolution where most relevant sound sources exist.

## 3. SELD WITH EQUAL-ANGLE SPATIAL TARGETS

We want to empirically test if equal-angle discretization affects SELD model performance. We hypothesize that the model's localization error will be a function of target elevation. More specifically, we predict that the irregularities of equal-angle discretization will result in larger errors around the equator than at the poles.

### 3.1. Synthetic dataset with equal-angle spatial discretization

We wanted to do this analysis with the DCASE STARSS2022 dataset [17] (real-world events belonging to thirteen categories, spatially discretized using equal-angle). However, its distribution of events on the elevation axis is not uniform (see supplement S2). Therefore, it will be hard to determine the effect of equal-angle discretization on model performance (since the data shows non-uniform distribution of targets). Moreover, our analysis is not focused on classification, so a single sound event category would be enough. We decided to create a synthetic dataset that controls for uniform target localization along the elevation axis using a single sound category. In contrast to our dataset, DCASE STARSS2022 is more complex, so we expect SELD models to easily learn our synthetic training split and generalize to the test split.

Our synthetic dataset has no moving or overlapping events, and repeats a single alarm sound (5 seconds duration) from FSD50k [18]. We used impulse responses (IRs) from two rooms (No. 3 and No. 4) in the TAU-SRIR database [19] that we convolve with the alarm sound. The rooms were selected because they contain IRs from sources localized at elevations spanning the integers

$$ \theta \in [-33..32] \mid \theta \neq -25, -24, -3, -2, 3, 20, 21 \quad (1) $$

in units of degrees (the missing integers are elevations not present in the two rooms). To generate the data we used the DCASE2022-data-generator that accompanies the TAU-SRIR database [19].

We synthesized two data folds: training and testing with 1600 and 900 tracks, respectively. Each track was a four-channel signal ($fs = 24kHz$) with a duration of 1 minute (sequences of alarm

<figure><img src="image_47.png" /><figcaption>Figure 2: Density of sound events locations in our synthetic dataset along the elevation (A) and azimuth axes (B). Data synthesis controlled for uniform density of sound events along the elevation axis.</figcaption></figure>

sounds and silence). While both training and test sets had elevation values spanning the range described in Eq. 1, no IRs overlapped across sets. This ensured that the absolute location of simulated sound sources was different between training and test sets. Figure 2 shows the distribution of sound event locations in our dataset.

### 3.2. SELD model localization on equal-angle targets

We used our dataset to train the 2022 DCASE SELD baseline model [1], which is a convolutional recurrent neural network that maps multichannel audio features (generalized cross-correlation with phase transform) into sound event locations and classes (see section 6.1 for a description of the model's output format). The trained model detected all test set sound events and showed an average angular localization error of 1.81°. Figure 3A shows a scatter plot with the localization error for each test set prediction as a function of target elevation. To gain intuition about how the model's localization varies as a function of elevation, we fit a line and a second-order polynomial (i.e. parabola) to this plot. The coefficient that multiplies the polynomial's second-order term determines the curvature of the parabola. If its value is close to zero, this indicates that the parabola resembles a line. In contrast, a more negative (positive) coefficient indicates that the parabola is more curved, and we can interpret it as the model's error decreasing (increasing) as a function of elevation away from the equator.

The line was $\hat{y}_l = 0.89 + (3.2 \times 10^{-3})x$, while the polynomial was $\hat{y}_p = 1.03 + (2.9 \times 10^{-3})x - (0.4 \times 10^{-3})x^2$, also shown in Figure 3A. The polynomial's second order coefficient reveals an upside-down parabolic relationship between the model's angular localization error and target elevation. The Pearson's correlation coefficient for the linear regression was $r = 0.01$, and for the polynomial regression was $r = 0.09$. This indicates that the parabola better explains the model's error than the line. These results show that training a SELD model with equal-angle data results in larger localization errors around the equator.

<footnote n="1">¹the supplement and code are available at https://github.com/sakshamsingh1/dcase_seld_spatialSampling_analysis</footnote>
<figure><img src="image_48.png" /><figcaption>Figure 3: Scatter plots of SELD model angular localization error on our synthetic test data as a function of target elevation. Each dot is a sound event. (A) Trained with equal-angle points. (B) Trained with Fibonacci points. (C) Trained with equal-angle points and fine-tuned using our proposed loss function (see section 6). The red and blue lines are linear and second-order polynomial regressions, respectively.</figcaption></figure>

## 4. UNIFORM DISCRETIZATION OF THE SPHERE

Alternatives to equal-angle discretization that uniformly sample points on the sphere exist [20, 21]. Perfectly-uniform discretization is limited by using the five Platonic solids, whose vertices can be used as points on the sphere [15]. The Fibonacci lattice [16] is an- other possible method that results in neighboring points separated by a roughly equal angular distance and is obtained by sampling points along a spiral that links the two poles (see [16] for the Fi- bonacci lattice formula). Figure 1B shows Fibonacci discretization.

## 5. SELD WITH UNIFORM SPATIAL TARGETS

We want to analyze SELD model localization when trained with uniformly discretized spatial targets. We hypothesize that this will result in uniform localization error on the elevation axis.

### 5.1. Synthetic dataset with uniform spatial sampling

Our synthetic dataset generated in section 3 spatially discretized sound events using equal-angle. To generate a uniformly discretized version of our dataset, we took our dataset's equal-angle annotations and transformed them into Fibonacci discretization by converting each equal-angle target into the nearest Fibonacci point. Our Fibonacci discretization had $N=32768$ points, which is the power of two that yields an angular distance between neighboring points around $1°$ [16] (similar to the distance between integers in Eq. 1). The audio tracks were exactly the same across equal-angle and Fibonacci versions of the dataset.

### 5.2. SELD model localization on Fibonacci targets

The 2022 DCASE SELD model trained with Fibonacci targets also detected all sound events, showing an average angular localization error of $1.86°$ on the test set (Figure 3B shows this model's scatter plot). We also fit a line to this plot, which was $\hat{y}_l = 0.92 + (3.1 \times 10^{-3})x$, and a second-order polynomial, which was $\hat{y}_p = 0.96 + (3.0 \times 10^{-3})x - (0.1 \times 10^{-3})x^2$. Compared to the polynomial for the model trained with equal-angle data, this polynomial's second order term reveals a less pronounced parabola, which is also visible in Figure 3B. The Pearson's correlation coefficient for

the linear regression was $r = 0.09$, and for the polynomial regression was $r = 0.11$. This indicates that, compared to equal-angle, the parabola and the line more similarly explain the model's error as a function of target elevation after training with Fibonacci targets. In other words, the model trained with Fibonacci data shows localization errors that are uniform as a function of target elevation.

Our results clearly illustrate how SELD model performance is affected by equal-angle and Fibonacci discretization. However, we recognize that the Pearson and parabolic coefficients we observed show clear trends but are relatively weak indicators. Future work could support our observations using more robust statistical testing.

## 6. PROPOSED SOLUTION

Our empirical analysis with synthetic data revealed that equal-angle discretization can result in a SELD model with larger localization errors at the equator than at the poles. Substituting the equal-angle discretization with a uniform one (like a Fibonacci lattice) would be a simple solution. In fact, resampling the DCASE STARSS2022 dataset using a Fibonacci lattice and training the model from scratch did result in improved metrics on the test set compared to baseline (see Table 6.1). However, since equal-angle discretization is prevalent in SELD datasets, engineered SELD learning methods that reduce its impact without the need to spatially resample the data are needed. Here we propose a training loss function that, in addition to computing the mean-squared error (MSE) between targets and model predictions, penalizes the model's angular localization error uniformly for all points on the sphere.

### 6.1. The threshold angular error ADPIT (TAEADPIT) loss

The 2022 DCASE SELD model is trained with the auxiliary duplicating permutation invariant training (ADPIT) loss [11], which uses the multi-class activity-coupled cartesian direction of arrival (multi-ACCDOA) target format $\mathbf{P} \in \mathbb{R}^{3 \times N \times C \times T}$, where 3 is the dimensionality of 3D cartesian coordinates, $N$ is the maximum number of simultaneous sound events the model is trained to detect, $C$ is the number of classes and $T$ is the number of time frames. A vector $\mathbf{P}_{\text{nct}} \in \mathbb{R}^3$ has a magnitude of 1, i.e. $||\mathbf{P}_{\text{nct}}||_2 = 1$ and represents the location of a sound event for a specific track $n$, a specific
class *c* and a specific time-frame *t*. Such a vector may also represent the absence of a sound event if it has a magnitude of 0. A related term, *a* ∈ ℝ$^{N×C×T}$, indicates the activities over tracks, classes, and time, and *a*<sub>nct</sub> ∈ {0, 1}. It is worth noting that the target contains duplicated sound events along the *N* dimension when the number of simultaneous sound events for each class is less than the maximum *N*. A complete description of the ADPIT loss can be found in its original publication [11].

The multi-ACCDOA format is permuted for each time-frame and class, and all permutation are compared against the model's output $\hat{\mathbf{P}} \in \mathbb{R}^{3 \times N \times C \times T}$ using MSE, yielding the ADPIT loss

$$ \mathcal{L}^{\text{ADPIT}} = \frac{1}{CT} \sum_{c}^{C} \sum_{t}^{T} \min_{\alpha \in \text{Perm}(ct)} l_{\alpha, ct}^{\text{ACCDOA}}, \quad (2) $$

$$ l_{\alpha, ct}^{\text{ACCDOA}} = \frac{1}{N} \sum_{n}^{N} \text{MSE}(\mathbf{P}_{\alpha, nct}^{*}, \hat{\mathbf{P}}_{nct}), \quad (3) $$

where $\mathbf{P}_{\alpha, nct}^{*}$ indicates a permutation of the multi-ACCDOA format, and only the permutation that resulted in the minimum $l_{\alpha, ct}^{\text{ACCDOA}}$ term is used to average over classes and time-frames.

Due to the nature of the multi-ACCDOA format, the ADPIT loss function operates over Cartesian coordinates. We propose adding a term that penalizes the model's angular localization error on the sphere where the data is spatially discretized:

$$ \begin{aligned} l_{\alpha, ct}^{\text{ALE}} &= \max(\mathbf{a}_{\alpha, nct}\text{ALE}_{\alpha, nct}, H) \\ \text{ALE}_{\alpha, nct} &= \angle(p(\mathbf{P}_{\alpha, nct}), p(\hat{\mathbf{P}})_{nct}), \end{aligned} \quad (4) $$

where $p(x)$ is a function that converts from cartesian to polar coordinates denotes, $\angle(a, b)$ is the angular distance between inputs *a* and *b*, and *H* is a threshold. ALE stands for angular localization error. Note that the $\mathbf{a}_{\alpha, nct}$ term masks ALE so that only the model's angular localization error related to active targets counts toward the loss. Adding the ADPIT loss gives:

$$ \mathcal{L}^{\text{TAEADPIT}} = \frac{1}{CT} \sum_{c}^{C} \sum_{t}^{T} \min_{\alpha \in \text{Perm}(ct)} l_{\alpha, ct}^{\text{ACCDOATAE}}, \quad (5) $$

$$ l_{\alpha, ct}^{\text{ACCDOATAE}} = \frac{1}{N} \sum_{n}^{N} \text{MSE}(\mathbf{P}_{\alpha, nct}^{*}, \hat{\mathbf{P}}_{nct}) + \beta(l_{\alpha, ct}^{\text{ALE}} - H), \quad (6) $$

where $\beta$ is a scale factor on the new term. We call this new loss "thresholded angular error ADPIT" (TAEADPIT) loss. The new term ALE is a regularizer that uniformly penalizes angular localization errors, independent of how targets are spatially discretized.

## 7. EMPIRICAL EVALUATION OF THE TAEADPIT LOSS

We conducted experiments to assess the TAEADPIT loss. First, we used it to fine-tune the model trained with the equal-angle version of our synthetic dataset. Figure 3C shows the model's angular localization error as a function of elevation. We also fit a line to this plot, which was $\hat{y}_l = 0.49 - (1.1 \times 10^{-3})x$, and a second-order polynomial, which was $\hat{y}_q = 0.61 - (1.4 \times 10^{-3})x - (0.31 \times 10^{-3})x^2$. The parabola's second order coefficient has a value of $-0.31 \times 10^{-3}$, which is closer to zero compared to the one found before fine-tuning ($-0.42 \times 10^{-3}$). This indicates that fine-tuning with the TAEADPIT loss flattened the parabola. In other words, the model fine-tuned

<table><caption>Table 1: Comparison of SELD model performance when training with ADPIT loss versus training with the proposed TAEADPIT loss. ADPIT-base: the baseline 2022 DCASE SELD model. TAEADPIT-tune: fine-tuning the baseline model with TAEADPIT. TAEADPIT: baseline model trained from scratch with TAEADPIT. ADPIT-Fib: baseline model trained with the data spatially resampled to the Fibonacci lattice and the ADPIT loss. The metrics are the DCASE SELD challenge metrics with class-depending macro-averaging are used (see [10]).</caption><thead><tr><th>Loss</th><th>ER<sub>20°</sub></th><th>F<sub>20°</sub></th><th>LE<sub>CD</sub></th><th>LR<sub>CD</sub></th><th>SELD</th></tr></thead><tbody><tr><td>ADPIT-base</td><td>0.69</td><td>0.24</td><td>30.43</td><td>0.43</td><td>0.55</td></tr><tr><td>TAEDPIT-tune</td><td>0.71</td><td>0.23</td><td>28.86</td><td>0.47</td><td>0.54</td></tr><tr><td>TAEDPIT</td><td>0.71</td><td>0.20</td><td>26.42</td><td>0.41</td><td>0.56</td></tr><tr><td>ADPIT-Fib</td><td>0.68</td><td>0.22</td><td>26.11</td><td>0.46</td><td>0.54</td></tr></tbody></table>

with the TAEADPIT loss shows more uniform localization errors as a function of target elevation than it did before being fine-tuned.

We also wanted to assess the TAEADPIT loss using real-world data. First, we ensured that we could replicate the baseline DCASE SELD 2022 model metrics using the four-channel microphone version of the DCASE STARSS2022 dataset [14] and supplemental synthetic data [22] (see Table 1). Next, we ran a couple of experiments to assess whether the TAEADPIT loss could benefit this model's performance. In all experiments, $\beta = 1 \times 10^{-3}$ in Eq. 5 (empirically-found). First, we used the TAEADPIT loss to fine-tune it. Then, we trained it from scratch using the TAEADPIT loss. Table 6.1 shows the results on the DCASE SELD metrics, indicating that using the TAEADPIT loss to fine-tune or train the SELD model from scratch can improve its localization error while retaining or only marginally impacting the classification metrics.

Our results show that the TAEADPIT loss can be used to train a SELD model using equal-angle data and improve localization metrics, and that it does so by reducing the larger localization error around the equator produced by equal-angle discretization.

## 8. CONCLUSION

In this paper, we studied the irregularities of equal-angle spatial discretization, which results in a larger density of points at the poles than at the equator. No previous studies have shown whether a SELD model's performance is affected by training with equal-angle discretized targets. We have empirically shown that equal-angle data affects SELD model localization on the elevation axis, causing larger localization errors around the equator than at the poles.

We also studied whether discretizing targets using a uniform Fibonacci lattice resulted in the same effect. We found that training a SELD model with Fibonacci data results in more uniform localization errors along the elevation axis compared to equal-angle. We also proposed a loss function to mitigate the effect of equal-angle by adding a thresholded angular localization error term to the ADPIT loss. Empirical results using our proposed loss when training a SELD model with equal-angle showed improved localization metrics compared to when using the ADPIT loss.

Next, we would like to assess whether a thresholded angular localization error in the training loss benefits other SELD models, and whether the benefit depends on audio format (i.e. FOA, HOA, stereo) and/or localization target format (i.e. Cartesian, 3D polar).
## 9. ACKNOWLEDGMENTS

This work is supported by the National Science Foundation grant no. IIS-1955357. The authors thank the funding source and their grant collaborators. We would also like to thank Charalampos Avraam for his comments to improve Supplement Section S1.

## 10. REFERENCES

[1] S. Adavanne, A. Politis, J. Nikunen, and T. Virtanen, "Sound event localization and detection of overlapping sources using convolutional recurrent neural networks," *IEEE Journal of Selected Topics in Signal Processing*, vol. 13, no. 1, pp. 34–48, 2018.

[2] D. Barchiesi, D. Giannoulis, D. Stowell, and M. D. Plumbley, "Acoustic scene classification: Classifying environments from the sounds they produce," *IEEE Signal Processing Magazine*, vol. 32, no. 3, pp. 16–34, 2015.

[3] C. Evers and P. A. Naylor, "Acoustic slam," *IEEE/ACM Transactions on Audio, Speech, and Language Processing*, vol. 26, no. 9, pp. 1484–1498, 2018.

[4] C. Chen, U. Jain, C. Schissler, S. V. A. Gari, Z. Al-Halah, V. K. Ithapu, P. Robinson, and K. Grauman, "Soundspaces: Audio-visual navigation in 3d environments," in *European Conference on Computer Vision*. Springer, 2020, pp. 17–36.

[5] W. Mack, U. Bharadwaj, S. Chakrabarty, and E. A. Habs, "Signal-aware broadband doa estimation using attention mechanisms," in *ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. IEEE, 2020, pp. 4930–4934.

[6] G. Valenzise, L. Gerosa, M. Tagliasacchi, F. Antonacci, and A. Sarti, "Scream and gunshot detection and localization for audio-surveillance systems," in *2007 IEEE Conference on Advanced Video and Signal Based Surveillance*. IEEE, 2007, pp. 21–26.

[7] M. Simeoni, S. Kashani, P. Hurley, and M. Vetterli, "Deep-wave: a recurrent neural-network for real-time acoustic imaging," *Advances in Neural Information Processing Systems*, vol. 32, 2019.

[8] R. Stiefelhagen, K. Bernardin, R. Bowers, J. Garofolo, D. Mostefa, and P. Soundararajan, "The clear 2006 evaluation," in *International evaluation workshop on classification of events, activities and relationships*. Springer, 2006, pp. 1–44.

[9] C. Evers, H. W. Löllmann, H. Mellmann, A. Schmidt, H. Barfuss, P. A. Naylor, and W. Kellermann, "The locata challenge: Acoustic source localization and tracking," *IEEE/ACM Transactions on Audio, Speech, and Language Processing*, vol. 28, pp. 1620–1643, 2020.

[10] A. Politis, A. Mesaros, S. Adavanne, T. Heittola, and T. Virtanen, "Overview and evaluation of sound event localization and detection in dcase 2019," *IEEE/ACM Transactions on Audio, Speech, and Language Processing*, vol. 29, pp. 684–698, 2020.

[11] K. Shimada, Y. Koyama, S. Takahashi, N. Takahashi, E. Tsunoo, and Y. Mitsufuji, "Multi-accdoa: Localizing and

detecting overlapping sounds from the same class with auxiliary duplicating permutation invariant training," in *IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, Singapore, Singapore, May 2022.

[12] A. Politis, S. Adavanne, and T. Virtanen, "A dataset of reverberant spatial sound scenes with moving sources for sound event localization and detection," in *Proceedings of the Detection and Classification of Acoustic Scenes and Events 2020 Workshop (DCASE2020)*, Tokyo, Japan, November 2020, pp. 165–169. [Online]. Available: https://dcase.community/workshop2020/proceedings

[13] A. Politis, S. Adavanne, D. Krause, A. Deleforge, P. Sri vastava, and T. Virtanen, "A dataset of dynamic reverberant sound scenes with directional interferers for sound event localization and detection," in *Proceedings of the 6th Detection and Classification of Acoustic Scenes and Events 2021 Workshop (DCASE2021)*, Barcelona, Spain, November 2021, pp. 125–129. [Online]. Available: https://dcase.community/workshop2021/proceedings

[14] A. Politis, K. Shimada, P. Sudarsanam, S. Adavanne, D. Krause, Y. Koyama, N. Takahashi, S. Takahashi, Y. Mitsufuji, and T. Virtanen, "Starss22: A dataset of spatial recordings of real scenes with spatiotemporal annotations of sound events," *arXiv preprint arXiv:2206.01948*, 2022.

[15] B. Rafaely, *Fundamentals of spherical array processing*. Springer, 2015, vol. 8.

[16] B. Keinert, M. Innmann, M. Sänger, and M. Stamminger, "Spherical fibonacci mapping," *ACM Transactions on Graphics (TOG)*, vol. 34, no. 6, pp. 1–7, 2015.

[17] A. Politis, Y. Mitsufuji, P. Sudarsanam, K. Shimada, S. Adavanne, Y. Koyama, D. Krause, N. Takahashi, S. Takahashi, and T. Virtanen, "STARSS22: Sony-TAU Realistic Spatial Soundscapes 2022 dataset," May 2022. [Online]. Available: https://doi.org/10.5281/zenodo.6600531

[18] E. Fonseca, X. Favory, J. Pons, F. Font, and X. Serra, "Fsd50k: an open dataset of human-labeled sound events," *IEEE/ACM Transactions on Audio, Speech, and Language Processing*, vol. 30, pp. 829–852, 2021.

[19] A. Politis, S. Adavanne, and T. Virtanen, "TAU Spatial Room Impulse Response Database (TAU- SRIR DB)," Apr. 2022. [Online]. Available: https://doi.org/10.5281/zenodo.6408611

[20] L. Perotin, R. Serizel, E. Vincent, and A. Guérin, "Crnn-based multiple doa estimation using acoustic intensity features for ambisonics recordings," *IEEE Journal of Selected Topics in Signal Processing*, vol. 13, no. 1, pp. 22–33, 2019.

[21] P.-A. Grumiaux, S. Kitić, L. Girin, and A. Guérin, "Improved feature extraction for crnn-based multiple sound source localization," in *2021 29th European Signal Processing Conference (EUSIPCO)*. IEEE, 2021, pp. 231–235.

[22] A. Politis, "[DCASE2022 Task 3] Synthetic SELD mixtures for baseline training," Apr. 2022. [Online]. Available: https://doi.org/10.5281/zenodo.6406873
# IS MY AUTOMATIC AUDIO CAPTIONING SYSTEM SO BAD?
## SPIDEr-max: A METRIC TO CONSIDER SEVERAL CAPTION CANDIDATES

Etienne Labbé, Thomas Pellegrini, Julien Pinquier

IRIT, Université Paul Sabatier, CNRS, Toulouse, France
{etienne.labbe, thomas.pellegrini, julien.pinquier}@irit.fr

## ABSTRACT

Automatic Audio Captioning (AAC) is the task that aims to describe an audio signal using natural language. AAC systems take as input an audio signal and output a free-form text sentence, called a caption. Evaluating such systems is not trivial, since there are many ways to express the same idea. For this reason, several complementary metrics, such as BLEU, CIDEr, SPICE and SPIDEr, are used to compare a single automatic caption to one or several captions of reference, produced by a human annotator. Nevertheless, an automatic system can produce several caption candidates, either using some randomness in the sentence generation process, or by considering the various competing hypothesized captions during decoding with beam-search, for instance. If we consider an end-user of an AAC system, presenting several captions instead of a single one seems relevant to provide some diversity, similarly to information retrieval systems. In this work, we explore the possibility to consider several predicted captions in the evaluation process instead of one. For this purpose, we propose SPIDEr-max, a metric that takes the maximum SPIDEr value among the scores of several caption candidates. To advocate for our metric, we report experiments on Clotho v2.1 and AudioCaps, with a transformed-based system. On AudioCaps for example, this system reached a SPIDEr-max value (with 5 candidates) close to the SPIDEr human score of reference.

**Index Terms—** audio captioning, evaluation metric, beam search, multiple candidates

## 1. INTRODUCTION

Automated Audio Captioning (AAC) is the task, in which a system takes an audio signal as input and provides a short description of its content using natural language. AAC could be useful for hearing-impaired people, in machine-to-machine interaction, surveillance and information retrieval in general. In the last few years, the research community has developed a keen interest in AAC, in particular thanks to the Detection and Classification of Acoustic Scenes and Events (DCASE) Challenges and Workshops <fnref n="1" />, which have provided datasets and benchmarks for this task.

Most AAC systems use deep neural networks with a sequence-to-sequence encoder-decoder architecture, to build a semantic audio representation and generate a valid sentence as output [1]. They rely on models pretrained on large-scale datasets, to solve the data scarcity issue in AAC [2, 3, 4].

In this work, we are interested in the evaluation of AAC systems. AAC evaluation borrows metrics from machine translation and image captioning, and consists of comparing a candidate caption to one or several manually produced captions of reference.

Since evaluating text generated automatically is a difficult problem, several metrics are used in combination. We investigate in particular the SPIDEr metric [5], a short name used to designate the average of two metrics called Consensus-based Image Description Evaluation (CIDEr) [6] and Semantic Propositional Image Caption Evaluation (SPICE) [7]. SPIDEr is used, for instance, in the DCASE yearly challenges to rank the participant AAC systems <fnref n="2" />.

In this paper, we report experiments using the AAC system we developed to participate in the DCASE 2022 AAC task. Like most AAC systems, we use a beam search decoder that allows to generate several candidate captions. The most likely one is used to compute the SPIDEr score of our system. A strong limitation of SPIDEr, in our opinion, is that only one caption candidate is considered for evaluation. As we shall illustrate in this paper, two correct captions that differ by a single word may have very different SPIDEr scores, if one of the words happens to be in the caption(s) of reference. To overcome this issue, we propose a metric that we call SPIDEr-max, which takes into account multiple candidates for a single audio recording.

## 2. METRICS

In the literature, most AAC systems are evaluated using the CIDEr, SPICE or SPIDEr metrics. These metrics come from the field of image captioning and evaluate a single candidate caption against a reference set.

### 2.1. CIDEr

CIDEr [6] is a metric based on the TF-IDF (term frequency-inverse document frequency) scores of each n-gram of the candidate and reference sentences. TF-IDF is used to give a higher weight to infrequent n-grams and lower weight to frequent n-grams.

The CIDEr metric calculation starts by stemming all the words and compute all the n-grams of size 1 to *N* across all candidates and references. The frequency of each n-gram in references are used to compute TF-IDF of all captions. This means that the score of each candidate does not only depend on its corresponding references, but also on all the other references of the corpus being evaluated. Then, the TF-IDF scores are vectorized and used to compute cosine similarity between the candidate and each reference. The similarities are rescaled by a factor of 10 and averaged across the references to get the final score of the candidate. All the scores are averaged again to get the global score on a dataset.

The CIDEr-D metric is a more robust version of CIDEr supposed to be closer to human judgement. It removes the stemming operation to take into account the tense and plural of words, adds

<footnote n="1">¹http://dcase.community/</footnote>

<footnote n="2">²http://dcase.community/challenge2022/</footnote>
a penalty factor, and limits the maximum occurrence of candidate n-grams to penalize longer repetitive sentences. The penalty is multiplied by a similarity measure based on the length of the candidate c and the reference r:

$$ \text{Penalty}(c, r) = \exp\left(-\frac{(|c| - |r|)^2}{2\sigma^2}\right) \quad (1) $$

Some AAC papers do not specify which version of CIDEr they use, but in this paper we report CIDEr-D scores as used in the DCASE challenge. We use the default settings of CIDEr-D with the maximum n-gram size N set to 4, and the hyperparameter σ used for the penalty set to 6.

## 2.2. SPICE

SPICE [7] attempts to extract the semantic content of a sentence. Sentences are used as input to a Probabilistic Context-Free Grammar dependency parser[8], with several additional rules to build a dependency tree where each node is a word and each edge is a syntactic dependency. Custom rules are used to compute another graph, a “semantic scene graph”, comprised of three types of nodes: objects, attributes and relations. Attributes are linked to a single object, and relations connect objects between them. The reference graphs are merged into one to be compared with a candidate graph. Then, the scene graphs are converted into lists of word tuples. An object is a tuple with the object name, an attribute is a tuple of two words with the object and attribute names, and a relation is a tuple of three words containing the two objects connected and the relation names. Finally, the list is binarized for the candidate and the references, and used to compute an F-Score.

The M-SPICE metric [9] is a variant of SPICE, which takes multiple candidates for a single audio. This metric was introduced to evaluate the diversity of the words used in multiple candidates generated by stochastic decoding methods. The only difference is that the semantic graph of each candidate is merged into one, exactly as for the reference list. The other steps remain the same.

## 2.3. SPIDEr

SPIDEr [5] is a metric originally used as a cost function to optimize a model on SPICE and CIDEr-D at the same time. SPIDEr is the average of CIDEr-D and SPICE, and is supposed to have the benefits of both previous metrics. Since CIDEr-D gives a score between 0 and 10 and SPICE between 0 and 1, the SPIDEr score is between 0 and 5.5, which is quite uncommon for a metric. SPIDEr is usually the metric used in AAC papers to compare models, even if other machine-translation metrics like BLEU [10], ROUGE-L [11], and METEOR [12] scores are also reported.

# 3. SYSTEM DESCRIPTION

## 3.1. Datasets

The Clotho v2.1 dataset [13] is an audio captioning dataset containing 6974 audio files of approximately 43.6 hours from Freesound between 15 and 30 seconds. Each audio is described by 5 captions annotated by humans. The dataset is divided in 3 different splits: development, validation and evaluation, which corresponds to development-training, development-validation and development-testing, the conventional names used in the DCASE Challenge. In this paper, we use these names. The training subset contains 217362 words with a caption length between 8 and 20 words.

AudioCaps [14] is another audio captioning dataset containing 49838 training files of approximately 136.6 hours from AudioSet [15], a large audio tagging dataset with audio extracted from YouTube. AudioCaps contains only 1 caption per audio in the training subset and 5 captions for the validation and testing subsets. Since YouTube removes videos uploaded by users for various reasons, our version of AudioCaps contains only 46230 over 49838 files in training subset, 464 over 495 in validation subset and 912 over 975 files in testing subset. Our training subset contains 402482 words with a caption length between 1 and 52 words.

To extract audio features, we resample audio signals to 32 kHz and compute log-Mel spectrograms with a window size of 32 ms, a hop size of 10 ms and 64 Mel bands. All captions are put in lowercase and punctuation characters are removed. We used the spaCy tokenizer [16] to split sentences into words, resulting in a vocabulary of 4370 tokens for Clotho and 4724 words for AudioCaps.

## 3.2. Model architecture

We adopt a standard encoder-decoder structure used in most AAC systems, with a pre-trained encoder to extract audio features and a transformer decoder to generate our captions. The encoder is the CNN10 model, a convolutional network from the Pretrained Audio Neural Networks study (PANN) [2]. We used the weights available on Zenodo<fnref n="3" /> to initialize the model at the beginning of the training. An affine layer was added to project 512-dimensional to 256-dimensional embeddings. We kept the time axis of the audio embedding used as input for the decoder.

The decoder is a standard transformer decoder [17]. It takes the audio embeddings as inputs and all the previous words predicted. The word embeddings are randomly initialized and learned during training. We use teacher forcing with cross-entropy to train the model. During the testing phase, captions are generated using beam search, and we select the best candidate using the probability of the sentence P given by the model. The combination of our encoder and decoder is simply named “CNN10-Transformer”.

## 3.3. Experimental setup

We trained models for 50 epochs, on both datasets separately. To optimize our networks, we used Adam [18], with a learning rate set to $5 \cdot 10^{-4}$ at the first epoch, a $10^{-6}$ weight decay, a 0.9 $\beta_1$ and 0.999 $\beta_2$, and $\epsilon$ set to $10^{-8}$. We used a cosine learning rate scheduler with the following rule:

$$ \operatorname{lr}_k = \frac{1}{2} \left( 1 + \cos \left( \frac{k\pi}{K} \right) \right) \operatorname{lr}_0 \quad (2) $$

with k being the current epoch index, and K the total number of epochs.

The transformer decoder uses an embedding dimension $d_{model}$ of 256, four attention heads h, six stacked standard decoder layers, and a global dropout $P_{drop}$ set to 0.2. The last affine layer projects the 256-dimensional embeddings to an output of the vocabulary size of the dataset. We used label smoothing to reduce overfitting, set to 0.1 for AudioCaps and 0.2 for Clotho. In order to avoid gradient explosion, we clip gradients by a maximal L2-norm value set to 10 and 1 for AudioCaps and Clotho, respectively. During testing, beam size is set to 8 for Clotho and 2 for AudioCaps. The final encoder-decoder model results in 16M trainable

<footnote n="3">³https://zenodo.org/record/3987831</footnote>
parameters. We also used SpecAugment [19] as audio data augmentation with two bands dropped on the time axis with a maximal size of 64 bins and one band dropped on the frequency axis with a maximum size of two bins. Our implementation uses PyTorch [20], PyTorch-Lightning [21] and our aac-datasets<fnref n="4" /> package to download and manage audio captioning datasets.

## 4. SPIDEr RESULTS

Results on Clotho and AudioCaps of our model CNN10-Transformer are shown in Table 1. Standard deviations of our model are very small (0.001 and 0.004 for Clotho and AudioCaps, respectively). Cross-reference scores are computed by using one of the reference as a candidate and the four others as references five times.

<table><caption>Table 1: SPIDEr scores on Clotho v2.1 and AudioCaps with state-of-the-art results and cross-reference scores.</caption><thead><tr><th>System</th><th>Clotho</th><th>AudioCaps</th></tr></thead><tbody><tr><td>Best</td><td>0.320 [22]</td><td>0.465 [4]</td></tr><tr><td>Human</td><td>N/A</td><td>0.565 [14]</td></tr><tr><td>Cross-Referencing</td><td>0.573</td><td>0.564</td></tr><tr><td>CNN10-Transformer (ours)</td><td>0.247</td><td>0.401</td></tr></tbody></table>

Our model performs much better on AudioCaps than Clotho, with a SPIDEr score of 0.401 and 0.247, respectively. It is also closer to the cross-reference and human scores in AudioCaps. This is probably due to the fact that the CNN10 encoder has been pretrained on AudioSet, which is a superset of AudioCaps. In addition, the captions in AudioCaps are simpler than those in Clotho, with shorter sentences and a relatively smaller vocabulary. The current best score on AudioCaps is also much closer to the cross-reference top score (0.100 difference) than the one on Clotho (0.253 difference).

## 5. SPIDEr LIMITATIONS

### 5.1. The SPIDEr score varies greatly between beam search candidates

Tables 2 and 3 show examples of candidates and captions of reference, one from Clotho, one from AudioCaps. The probability *P* given by the model is also indicated. It used to select the best candidate among the beam search hypotheses. We also reported the SPIDEr score associated to each candidate.

In the Clotho example, the most likely caption candidate is also the one with the highest SPIDEr score, based on the fact that the rather rare word “tin” was found by the automatic system. Thus, in this example, the differences observed between the various hypotheses seem justified. On the contrary, in the second example, from AudioCaps, the most likely automatic caption is different from the one with the highest SPIDEr score.

The agreement accuracy between the best candidate according either to the likelihood and to the SPIDEr score is only of 26.5% on Clotho, and 22.6% on AudioCaps. The correlation coefficient on all the likelihoods and the SPIDEr scores is 0.224 on Clotho and 0.259 on AudioCaps. This shows that the maximum candidate likelihood *P* does not select the best caption according to the SPIDEr score.

<table><caption>Table 2: Captions for the Clotho development-testing file named “rain.wav”.</caption><thead><tr><th>Candidates</th><th>P</th><th>SPIDEr</th></tr></thead><tbody><tr><td>heavy rain is falling on a roof</td><td>0.361</td><td>0.562</td></tr><tr><td>heavy rain is falling on a tin roof</td><td><strong>0.408</strong></td><td><strong>0.930</strong></td></tr><tr><td>a heavy rain is falling on a roof</td><td>0.369</td><td>0.594</td></tr><tr><td>a heavy rain is falling on the ground</td><td>0.351</td><td>0.335</td></tr><tr><td>a heavy rain is falling on the roof</td><td>0.340</td><td>0.594</td></tr></tbody></table>

### References

heavy rain falls loudly onto a structure with a thin roof
heavy rainfall falling onto a thin structure with a thin roof
it is raining hard and the rain hits a tin roof
rain that is pouring down very hard outside
the hard rain is noisy as it hits a tin roof

<table><caption>Table 3: Captions for an AudioCaps testing file (id: ‘jid4t-FzUn0’).</caption><thead><tr><th>Candidates</th><th>P</th><th>SPIDEr</th></tr></thead><tbody><tr><td>a woman speaks and a sheep bleats</td><td>0.475</td><td>0.190</td></tr><tr><td>a woman speaks and a <strong>goat</strong> bleats</td><td>0.464</td><td><strong>1.259</strong></td></tr><tr><td>a man speaks and a sheep bleats</td><td>0.464</td><td>0.344</td></tr><tr><td>an adult male speaks and a sheep bleats</td><td>0.450</td><td>0.231</td></tr><tr><td>an adult male is speaking and a sheep bleats</td><td><strong>0.491</strong></td><td>0.189</td></tr></tbody></table>

### References

a man speaking and laughing followed by a goat bleat
a man is speaking in high tone while a goat is bleating one time
a man speaks followed by a goat bleat
a person speaks and a goat bleats
a man is talking and snickering followed by a goat bleating

### 5.2. Can we choose a better candidate automatically?

Selecting automatically the best candidate among the beam search hypotheses is a difficult problem: most candidates are very similar and usually describe the same events with different words. Figure 1 shows the histogram of the beam hypothesis indices that give the maximum SPIDEr score possible, for each candidate list when using a beam size of five. It reveals that no beam index seems better than another *a priori*. The same conclusion can be drawn with Clotho. We tried to automatically select the best candidate using several features: vocabulary size, sentence length, and even with a shallow neural network trained to rank the sentences, but all these approaches failed to significantly improve the global SPIDEr score.

<figure><img src="image_49.png" /><figcaption>Figure 1: SPIDEr best beam indexes on AudioCaps testing subset</figcaption></figure>

<footnote n="4">⁴https://pypi.org/project/aac-datasets</footnote>