<figure><img src="image_71.png" /><figcaption>Figure 5: Qualitative visualisations of *Gearbox* Source Section 1. Top: Histograms of normalised (cosine) *k*-nearest neighbour similarities. Bottom: t-SNE scatter plots. Applying a low-pass filter (left) decreases separability between inliers and anomalies while applying a high-pass filter increases separability and improves inlier generalisation, as evidenced through the change in scoring distribution.</figcaption></figure>

tion. The heatmaps show that occluding the low-frequency regions makes inliers appear more anomalous, leading to more incorrect decisions. Correspondingly, occluding the high-frequency regions makes anomalies appear more benign. We observe similar patterns across sections and machines, and on the STFT spectrograms. This behaviour appears to validate the high-frequency hypothesis.

## 2.5. Re-training the detectors with filtered audio

Our final step to empirically validate the high-frequency hypothesis is to apply low and high-pass filters to the input data, repeat fine-tuning on the ASTs and redo anomaly detection. If high frequencies are important, then we would expect low-pass filters to decrease ASD. We filtered the input spectrograms at both the fine-tuning and anomaly detection stages to prevent discrepancies between the training and test distributions [22]. This intervention removes the possibility that any changes in performance are caused by distribution shifts. In line with previous studies that looked at frequency artefacts in the speech domain [23], we vary the cutoff frequencies between 400 Hz and 8 kHz in increments of 400 Hz for both the low-pass and high-pass filters. We applied the filters directly on the input audio before converting to spectrograms.

Figure 3 depicts our Mel spectrogram results. The low-pass results indicate removing high-frequency information decreases separability between classes, causing fine-tuning classification accuracy and ASD performance across subsets to drop significantly. Separability between classes is still good after applying high-pass filters, and so downstream ASD performance is retained or even improves. We note that classification accuracy drops from the baseline fine-tuning accuracy of 94% (Section 2.2), which suggests separability between classes is an important but not necessary component for ASD. Overall, our results illustrate that high frequencies are helpful for ASD, except for *Fan*, where informative frequencies likely lie in sub-bands. Although *Fan* exhibits similar trends to the other ma-

chines, applying high-pass filters causes performance to drop significantly compared to the benchmark AST. We show Figure 3's results qualitatively by visualising the change in normalised *k*-NN (cosine) similarities and t-SNE [24] on the 768-dimensional learnt embeddings. Figure 5 illustrates an example on *Gearbox*. Aggressive low-pass filtering decreases separability between inliers and anomalies while high-pass filtering increases the dissimilarity. We also visualise a similar phenomenon for the other machines.

## 3. CONCLUSION AND FUTURE WORK

We test if higher frequencies aid ASD in three ways: (1) by varying the input spectrograms used to train the anomaly detector, (2) evaluating the heatmaps produced by SLIME, and (3) re-training the anomaly detectors after filtering frequencies of differing magnitudes. Our results corroborate the high-frequency hypothesis and suggest high-pass filtering to remove noisy information is a simple pre-processing step to boost performance. The scope of adapted SLIME in our experiments was isolated to frequency segmentations. Some avenues for future work include analysing changes in ASD after segmenting input spectrograms by time, investigating SLIME outputs on alternative anomaly detection architectures, or using more sophisticated thresholds in place of EER. Furthermore, the exceptional results on *Fan* suggest that some anomalies may manifest in sub-band frequencies. Future work could extend the re-training approach by using band-pass filters, which could provide a more fine-grained analysis of anomalous artefacts.

## 4. ACKNOWLEDGMENT

This project made use of time on Tier 2 HPC facility JADE2, funded by EPSRC under grant EP/T022205/1.
5. REFERENCES

[1] http://dcase.community/.

[2] Y. Kawaguchi, K. Imoto, Y. Koizumi, N. Harada, D. Niizumi, K. Dohi, R. Tanabe, H. Purohit, and T. Endo, "Description and discussion on dcase 2021 challenge task 2: Unsupervised anomalous detection for machine condition monitoring under domain shifted conditions," in *Proceedings of the 6th Detection and Classification of Acoustic Scenes and Events 2021 Workshop (DCASE2021)*, Barcelona, Spain, November 2021, pp. 186–190.

[3] J. Lopez, G. Stemmer, and P. Lopez-Meyer, "Ensemble of complementary anomaly detectors under domain shifted conditions," DCASE2021 Challenge, Tech. Rep., July 2021.

[4] K. Morita, T. Yano, and K. Tran, "Anomalous sound detection using cnn-based features by self supervised learning," DCASE2021 Challenge, Tech. Rep., July 2021.

[5] J. Frank, T. Eisenhofer, L. Schönherr, A. Fischer, D. Kolossa, and T. Holz, "Leveraging frequency analysis for deep fake image recognition," in *International Conference on Machine Learning (ICML)*, 2020, pp. 3247–3258.

[6] D. Hendrycks, S. Basart, N. Mu, S. Kadavath, F. Wang, E. Dorundo, R. Desai, T. Zhu, S. Parajuli, M. Guo, et al., "The many faces of robustness: A critical analysis of out-of-distribution generalization," in *Proceedings of the IEEE/CVF International Conference on Computer Vision*, 2021, pp. 8340–8349.

[7] D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le, "Specaugment: A simple data augmentation method for automatic speech recognition," *Proc. Interspeech 2019*, pp. 2613–2617, 2019.

[8] Y. Liu, J. Guan, Q. Zhu, and W. Wang, "Anomalous sound detection using spectral-temporal information fusion," in *ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, 2022, pp. 816–820.

[9] S. Mishra, B. L. Sturm, and S. Dixon, "Local interpretable model-agnostic explanations for music content analysis," in *Proceedings of the 18th International Society for Music Information Retrieval Conference*. ISMIR, Oct 2017, p. 537–543.

[10] N. Harada, D. Niizumi, D. Takeuchi, Y. Ohishi, M. Yasuda, and S. Saito, "ToyADMOS2: Another dataset of miniature-machine operating sounds for anomalous sound detection under domain shift conditions," *arXiv preprint arXiv:2106.02369*, 2021.

[11] R. Tanabe, H. Purohit, K. Dohi, T. Endo, Y. Nikaido, T. Nakamura, and Y. Kawaguchi, "MIMII DUE: Sound dataset for malfunctioning industrial machine investigation and inspection with domain shifts due to changes in operational and environmental conditions," *arXiv e-prints: 2006.05822*, 2021.

[12] Y. Gong, Y.-A. Chung, and J. Glass, "AST: Audio Spectrogram Transformer," in *Proceedings of Interspeech 2021*, 2021, pp. 571–575.

[13] D. Hendrycks, X. Liu, E. Wallace, A. Dziedzic, R. Krishnan, and D. Song, "Pretrained transformers improve out-of-distribution robustness," in *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*. Online: Association for Computational Linguistics, July 2020, pp. 2744–2751.

[14] S. Fort, J. Ren, and B. Lakshminarayanan, "Exploring the limits of out-of-distribution detection," in *Advances in Neural Information Processing Systems*, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, Eds., 2021.

[15] K. T. Mai, T. Davies, and L. D. Griffin, "Self-supervised losses for one-class textual anomaly detection," *arXiv preprint arXiv:2106.02369*, 2022.

[16] J. F. Gemmeke, D. P. W. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, "Audio set: An ontology and human-labeled dataset for audio events," in *2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, 2017, pp. 776–780.

[17] Y.-Y. Yang, M. Hira, Z. Ni, A. Astafurov, C. Chen, C. Puhrsch, D. Pollack, D. Genzel, D. Greenberg, E. Z. Yang, J. Lian, J. Hwang, J. Chen, P. Goldsborough, S. Narethiran, S. Watanabe, S. Chintala, and V. Quenneville-Bélair, "Torchaudio: Building blocks for audio and speech processing," in *ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, 2022, pp. 6982–6986.

[18] D. Hendrycks and K. Gimpel, "A baseline for detecting misclassified and out-of-distribution examples in neural networks," in *5th International Conference on Learning Representations (ICLR)*, 2017.

[19] F. T. Liu, K. M. Ting, and Z.-H. Zhou, "Isolation forest," in *2008 Eighth IEEE International Conference on Data Mining*, 2008, pp. 413–422.

[20] B. McFee, A. Metsai, M. McVicar, S. Balke, C. Thomé, C. Raffel, F. Zalkow, A. Malek, Dana, K. Lee, O. Nieto, D. Ellis, J. Mason, E. Battenberg, S. Seyfarth, R. Yamamoto, Viktorandreevichmorozov, K. Choi, J. Moore, R. Bittner, S. Hidaka, Z. Wei, nullmightybofo, D. Hereñú, F.-R. Stöter, P. Friesch, A. Weiss, M. Vollrath, T. Kim, and Thassilo, "Librosa/librosa: 0.8.1rc2," May 2021.

[21] S. Mishra, E. Benetos, B. L. Sturm, and S. Dixon, "Reliable local explanations for machine listening," in *International Joint Conference on Neural Networks (IJCNN)*. IEEE, 2020.

[22] S. Hooker, D. Erhan, P.-J. Kindermans, and B. Kim, "A benchmark for interpretability methods in deep neural networks," in *Advances in Neural Information Processing Systems 32*, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, Eds. Curran Associates, Inc., 2019, pp. 9737–9748.

[23] H. Tak, J. Patino, A. Nautsch, N. Evans, and M. Todisco, "An explainability study of the constant q cepstral coefficient spoofing countermeasure for automatic speaker verification," in *Proc. Odyssey 2020 The Speaker and Language Recognition Workshop*, 2020, pp. 333–340.

[24] L. van der Maaten and G. Hinton, "Visualizing data using t-sne," *Journal of Machine Learning Research*, vol. 9, no. 86, pp. 2579–2605, 2008.
# A DEVICE CLASSIFICATION-AIDED MULTI-TASK FRAMEWORK FOR LOW-COMPLEXITY ACOUSTIC SCENE CLASSIFICATION

Rohith Mars and Rohan Kumar Das

Fortemedia Singapore, Singapore

{rohithmars, rohankd}@fortemedia.com

## ABSTRACT

In this paper, we investigate the use of a multi-task learning framework to address the DCASE 2022 Task 1 on Low-complexity Acoustic Scene Classification (ASC). Specifically, we employ classification of the recording devices as an additional task to improve the performance of the ASC task. Both these tasks utilize our proposed convolutional neural network with a shared layer along with strided and separable convolution operations designed to comply with the model parameter and computational constraints imposed by Task 1 organizers. We also explore the use of various data augmentation techniques to improve the generalization of the model. Evaluations on the development dataset show that our proposed ASC system consisting of 127.2k parameters with 27.5 million multiply and accumulate operations provides a significant improvement on overall log-loss and accuracy over the baseline system.

**Index Terms—** acoustic scene classification, convolutional neural networks, data augmentation, multi-task learning, quantization aware training.

## 1. INTRODUCTION

The objective of an Acoustic Scene Classification (ASC) system is to classify a given audio recording of a certain temporal duration into one of the pre-defined acoustic scenes. As a sub-field of computational auditory scene analysis (CASA) [1,2], ASC has received much attention as an important and challenging research topic. It also finds several applications in consumer devices, which incorporates contextual-aware computing, such as smart-wearables, hearables as well as surveillance [3,4].

The Detection and Classification of Acoustic Scenes and Events (DCASE) challenge<fnref n="1" />, has played an important role in providing standard datasets for setting algorithmic benchmarks and to spearhead research on ASC. Over the years, the ASC task in DCASE challenge series has witnessed several modifications by introducing additional recording devices and constraints for model complexity. In DCASE 2021 Task 1, the model size of the proposed solutions were constrained to 128 kB for non-zero parameters, with no constraints set on the multiply and accumulate (MAC) operations. In addition, the length of audio clip for inference was set as 10 seconds for all the past editions.

A majority of the proposed systems for multi-device ASC task utilize a single task learning (STL) framework with log-Mel spectrogram as input feature to a convolutional neural network (CNN)-based model, which is trained to predict the acoustic scene [5, 6]. To improve the generalization of the model to unseen devices and

recordings from unseen locations, many techniques including spectrum correction and frequency instance normalization were proposed [6, 7]. In addition to these techniques, various data augmentation strategies have also been employed extensively in the past to address the same [8, 9]. To meet the model size requirements, approaches such as model compression with pruning and use of knowledge distillation have also been well-explored [10, 11].

The latest DCASE 2022 Task 1 [12] is an extension of the previous editions of DCASE ASC challenges, with an objective to design an ASC system with low computation complexity and sufficiently less number of parameters such that it could be deployed for inference on an edge-device. Specifically, the maximum number of the model parameters is set to 128k (including the zero-valued ones) and the variable type is fixed to 8-bit integer (INT8). Further, the maximum number of MAC operations for each inference is limited to 30 MMAC (million multiply-accumulate operations), while the length of the audio for inference is fixed to 1 second. In addition to these system complexity constraints, the proposed ASC system is expected to be robust across multiple recording devices and locations similar to the previous editions.

We address this ASC task by utilizing a multi-task learning (MTL) framework, where the classification of recording devices is used as an additional task to aid the ASC task. Our proposed CNN model architecture is designed such that it satisfies the system complexity requirements in terms of MAC operations and total number of parameters. To further improve generalization ability of the model to unseen devices, we employ multiple data augmentation techniques. The quantization-aware-training (QAT) and TFLite tools are then utilized to convert the proposed CNN model to INT8 representation for performing final inference and comparisons to other competing systems along with the challenge baseline.

In the following sections, we first describe the proposed MTL framework in Section 2. The details of the experimental setup and results with analysis are reported in Section 3 and Section 4, respectively. Finally, the conclusions are presented in Section 5.

## 2. DEVICE CLASSIFICATION-AIDED ASC

A majority of the systems developed previously to address ASC using multiple devices have employed an STL framework. In comparison, the use of MTL frameworks for ASC task is less explored. Generally, an MTL framework consists of an encoder block with shared layers, which are subsequently connected to multiple branches with task-specific layers. A solution based on MTL framework was recently proposed for the ASC task of DCASE. The authors of [13] used an additional task of classifying the audio clip to 3 broader acoustic scenes, in addition to the given task of classifying the audio clip to 10 acoustic scenes for joint training. The 3 broader

<footnote n="1">¹https://dcase.community/challenge2022/</footnote>
acoustic scenes are higher-level abstractions of the 10 scenes, which make these tasks highly related to each other.

In DCASE 2022 Task 1 on low-complexity ASC, the development dataset consists of audio samples from 3 real-devices (A, B, C) and 6 simulated devices (S1-S6). In our proposed MTL framework, we leverage recording device information by classifying recording devices corresponding to the given audio clip as a task, in addition to the ASC task. Specifically, we classify the given audio clip to be recorded using one among the seen real-devices (A, B, C) and simulated/unseen devices. All the simulated devices (S1-S6) and unseen devices are grouped to 1 device-class, thereby forming a total of 4 device-classes. By enabling the shared layers in the encoder block to learn complementary features to classify the recording device corresponding to a given audio clip, we envisage that these layers of the network can learn low-level features which could improve the performance of the model for multi-device ASC task.

One of the previous works, explored the use of device classification for ASC as well [14]. However, for joint training using MTL, the one-hot encoding representation of the predicted device is expanded and converted as a feature map to the multiple CNN layers in the ASC branch. Even though the use of such device-information conditioned MTL training shows improved performance compared to STL framework, such an approach would require the device-classification branch to be included during the inference stage as well. As such, it would increase the number of parameters and MAC operations of the inference model for the given ASC task. To ensure that the feature maps of the shared layers can leverage similar information without increasing the computation complexity, instead of utilizing the predicted device as a feature map, we combine the loss corresponding to the device classification task with the loss of the ASC task for joint training.

We use a combined loss function, which is the weighted loss function from the ASC branch and the device classification branch to perform the joint training of both these tasks. It can be expressed mathematically as

$$ \mathcal{L}_{\text{MTL}} = \beta \times \mathcal{L}_{\text{device}} + (1 - \beta) \times \mathcal{L}_{\text{ASC}}, \quad (1) $$

where $\beta$ is the trade-off parameter that controls the weighted loss. $\mathcal{L}_{\text{device}}$ and $\mathcal{L}_{\text{ASC}}$ corresponds to the 4-device classification loss and the 10-scene classification loss, respectively. Setting $\beta = 0$ is similar to the STL framework.

The use of MTL framework with joint training utilizing the weighted loss in Eq (1) for the given ASC task has the additional advantage that once the MTL-based model is trained, the device classification branch can be removed from the model architecture. During the inference stage, only the 10-scene classification branch is utilized. Therefore, the number of parameters and MAC operations remains the same as that of the STL framework.

## 2.1. CNN model architecture

Our proposed MTL framework based on CNN architecture is shown in Figure 1. It consists of two branches, i.e., a branch corresponding to the 10-scene classification task and another branch corresponding to the 4-device classification task. The 10-scene classification branch consists of a combination of separable and strided convolutions. We use $L = 5$ convolution layers with number of filters set as (150, 140, 160, 180, 220). It is noted that separable convolutions are used for all the layers except the first convolution layer. In addition, a stride of 2 is used for all convolutional layers except the last convolution layer. Strided convolutions are applied uniformly

<figure><img src="image_72.png" /><figcaption>Figure 1: The proposed MTL framework with separate branches for 10-scene classification and 4-device classification task. Conv2D and SeparableConv2D ($n$, $(p \times q)$, $s$) represents 2D convolution and separable convolution operation with $n$ filters of kernel size $p \times q$ with a stride of $s$.</figcaption></figure>

across both frequency and time dimensions. The kernel size for the first three convolution layers is chosen as $(3 \times 3)$, whereas we consider a kernel size of $(3 \times 5)$ and $(1 \times 5)$ for the fourth and fifth convolution layers, respectively. Finally, global pooling operation is performed to gather all the components from the last convolution layer. The output layer consists of a dense layer with 10 units corresponding to the number of acoustic scene classes which undergoes softmax(·) operation to obtain the scene prediction probabilities.

For the 4-device classification branch, the output of the first convolution layer undergoes a global pooling operation and is connected to two dense layers of 32 and 16 units each. The output layer consists of a dense layer with softmax(·) operation with 4 units corresponding to the device classes. Rectified linear unit (ReLU) is chosen as the activation function and a weight decay of $1e-5$ is applied for all convolution layers. We also use dropout of 0.5 after the $L = 2, 4, 5$ convolution layers and a dropout of 0.3 between the dense layers of the 4-device classification branch. The network parameters for both the branches are determined empirically.

## 2.2. INT8 Quantization

As part of the Task 1 constraints, it is required that the model and the input data used for inference should use INT8 variable type. To minimize the drop in performance after the quantization step, we first perform quantization-aware-training (QAT) utilizing TensorFlow framework with a subset of the training data [15]. After the model is fine-tuned using QAT, it is converted to INT8 TFLite model. The number of parameters and the MAC operations of the TFLite model is computed by using the scripts in Nessi<fnref n="2" /> toolkit, which is provided by the Task 1 challenge organizers. The proposed TFLite model has 127, 236 parameters with 27.53 MMAC operations, which satisfies the model size and complexity constraints.

## 2.3. Data augmentation

To ensure that the proposed model is generalized to recordings using an unseen device or a recording from an unseen location, multiple data augmentation techniques that are well-explored in the past are used [16, 17]. In addition, we also explore the use of AugMix [18]

<footnote n="2">²https://github.com/AlbertoAncilotto/NeSsi</footnote>
<table><caption>Table 1: Performance comparison of the STL and the proposed MTL framework for various $\beta$. The device-wise log loss is shown in columns A-S6. The best results are shown in bold.</caption><thead><tr><th>Framework</th><th>Average Log Loss</th><th>Average Accuracy (%)</th><th>A</th><th>B</th><th>C</th><th>S1</th><th>S2</th><th>S3</th><th>S4</th><th>S5</th><th>S6</th></tr></thead><tbody><tr><td>STL</td><td>1.333 ± 0.012</td><td>51.25 ± 0.33</td><td>0.98</td><td>1.24</td><td><b>1.11</b></td><td>1.37</td><td>1.35</td><td>1.35</td><td>1.50</td><td>1.45</td><td>1.61</td></tr><tr><td>MTL, β = 0.1</td><td>1.327 ± 0.009</td><td>51.65 ± 0.31</td><td>0.99</td><td>1.24</td><td><b>1.11</b></td><td>1.36</td><td>1.35</td><td><b>1.33</b></td><td>1.48</td><td>1.45</td><td>1.61</td></tr><tr><td>MTL, β = 0.2</td><td><b>1.319 ± 0.017</b></td><td><b>51.98 ± 0.55</b></td><td><b>0.97</b></td><td><b>1.22</b></td><td><b>1.11</b></td><td>1.36</td><td><b>1.34</b></td><td>1.34</td><td>1.48</td><td><b>1.43</b></td><td><b>1.58</b></td></tr><tr><td>MTL, β = 0.3</td><td>1.323 ± 0.014</td><td>51.31 ± 0.55</td><td><b>0.97</b></td><td><b>1.22</b></td><td><b>1.11</b></td><td><b>1.35</b></td><td><b>1.34</b></td><td>1.34</td><td><b>1.47</b></td><td>1.46</td><td>1.61</td></tr><tr><td>MTL, β = 0.4</td><td>1.347 ± 0.014</td><td>50.53 ± 0.54</td><td>0.99</td><td>1.25</td><td>1.13</td><td>1.39</td><td>1.37</td><td>1.37</td><td>1.49</td><td>1.47</td><td>1.63</td></tr><tr><td>MTL, β = 0.5</td><td>1.335 ± 0.010</td><td>51.31 ± 0.40</td><td>0.98</td><td>1.25</td><td>1.12</td><td>1.38</td><td>1.37</td><td>1.36</td><td><b>1.47</b></td><td>1.45</td><td>1.59</td></tr></tbody></table>

and RawBoost [19] technique that have not yet been extensively used for ASC task. Since device classification is utilized in the MTL framework, it is important to assign the device labels according to the augmentation technique being employed. The augmentation techniques utilized are summarised below.

* **Time stretch (TS)**: The audio sample is either slowed down or sped up without altering the pitch by a factor randomly chosen from the uniform distribution [0.8, 1.2].

* **Time shift (TSH)**: A random start time is chosen from the 1 second duration of the given audio clip to obtain two audio segments. The augmented audio clip is the appended audio segments shifted in time.

* **Block mixing (BM)**: For a given audio clip corresponding to a particular device and acoustic scene, an additional audio clip recorded with the same device and belonging to the same acoustic scene is selected randomly. The augmented audio clip is the weighted average of the given audio clip and the additional audio clip.

* **AugMix**: Similar to the AugMix strategy used for image data augmentation, a given audio clip undergoes multiple chains of transformations using the above data augmentation steps (time stretch, time shift, block mix). At the final stage, the original audio clip is combined with the transform-chain augmented audio clip to obtain the augmented audio sample. We set the default values of maximum number and depth of the transform chain to 3 with the $\alpha$ parameter of both Dirichlet and Beta distribution set to 1.0 as in [18].

* **RawBoost**: The recently proposed data augmentation technique RawBoost for anti-spoofing is considered for exploration in the given ASC task. In RawBoost, multiple notch filters with randomly chosen center frequencies and bandwidths are used to design an FIR filter. The use of FIR filter on the given audio clip, introduces variations on the audio spectrum, which could improve generalization of the model for unseen device. We set the similar filter design configurations as used in [19].

Among the above data augmentation techniques, time stretch, time shift, block mixing and AugMix does not change the frequency characteristics of the given audio clip. Therefore, they are device-label invariant techniques. However, the RawBoost technique applies FIR filtering on the given audio clip which modifies its frequency characteristics. As such, it is a device-label variant technique and these clips are labeled as belonging to the simulated and unseen device class. Next, we discuss the experimental setup for the studies conducted in this work.

### 3. EXPERIMENTAL SETUP

We use the TAU Urban Acoustic Scenes 2022 Mobile, development dataset provided as part of the challenge for the studies in this work [20]. This development set consists of an official training and validation split with audio clips of 1 second in duration sampled at 44.1 kHz. The clips correspond to audio scene recordings from 10 cities using 9 devices: 3 real devices (A, B, C) and 6 simulated devices (S1-S6). The simulated device recordings (S1-S6) are obtained by applying impulse responses of real devices and additional dynamic range compression on recordings from device A. There are 102, 150 clips from device A, while for devices (B, C, S1-S3) there are ≈ 7,500 clips each. It should be noted that the audio clips corresponding to devices S4-S6 does not appear in the training split and is only present in the validation split. The 10 audio scenes to be classified are namely {"Airport", "Indoor shopping mall", "Metro station", "Pedestrian street", "Public square", "Street with medium level of traffic", "Travelling by a tram", "Travelling by a bus", "Travelling by an underground metro" & "Urban park"}.

We do not utilize any external data or pre-trained models for training our proposed system. For each audio clip, we compute the short-time Fourier transform (STFT) of the signal by using a frame length of 2048 samples and hop length of 1024 samples. After computing the STFT, the corresponding log Mel-spectrogram is computed with 256 Mel-bins. We then obtain a time-frequency (TF) representation of each audio clip with dimension 44 × 256, which is provided as the input feature to the proposed MTL framework.

During training, the optimization is performed using the Adam optimizer [21], with an initial learning rate of 0.001 and a maximum epoch of 200 with a batch size of 32 samples. The learning rate is reduced by a factor of 0.1 if the validation loss does not decrease after 5 epochs. Early stopping method is used to stop the training if the validation loss does not decrease after 10 epochs. The categorical focal loss [22] is chosen as the loss function for both classification tasks. For all the evaluations, we utilize the validation split from the development set. The multi-class cross-entropy (log loss) is used as the primary metric for evaluating system performance along with average of the class-wise accuracy as a secondary measure. The performances are evaluated over 5 trials and the average results are reported.

### 4. RESULTS AND ANALYSIS

This section discusses the results and their analysis conducted for various studies in this work. The impact of proposed MTL framework and data augmentation on ASC and comparison to other systems are reported in the following subsections.
<table><caption>Table 2: Performance comparison for various data augmentation approaches and their combination. The best results are shown in bold.</caption><thead><tr><th>Method</th><th>Avg. Log Loss</th><th>Avg. Acc. (%)</th><th>A</th><th>B</th><th>C</th><th>S1</th><th>S2</th><th>S3</th><th>S4</th><th>S5</th><th>S6</th></tr></thead><tbody><tr><td>No Data Augmentation</td><td>1.319 ± 0.017</td><td>51.98 ± 0.55</td><td>0.97</td><td>1.22</td><td>1.11</td><td>1.36</td><td>1.34</td><td>1.34</td><td>1.48</td><td>1.43</td><td>1.58</td></tr><tr><td>TS + TSH + BM (1)</td><td>1.314 ± 0.007</td><td>52.08 ± 0.33</td><td>0.96</td><td>1.21</td><td>1.13</td><td>1.36</td><td>1.34</td><td>1.32</td><td>1.46</td><td>1.44</td><td>1.56</td></tr><tr><td>AugMix (2)</td><td>1.309 ± 0.014</td><td>52.45 ± 0.56</td><td>0.96</td><td>1.23</td><td>1.12</td><td>1.35</td><td>1.33</td><td>1.33</td><td>1.43</td><td>1.43</td><td>1.56</td></tr><tr><td>RawBoost (3)</td><td>1.298 ± 0.010</td><td>52.22 ± 0.71</td><td>0.99</td><td>1.18</td><td>1.10</td><td>1.33</td><td>1.33</td><td>1.31</td><td>1.41</td><td>1.41</td><td>1.58</td></tr><tr><td>Combined (1) + (2) + (3)</td><td><b>1.265 ± 0.009</b></td><td><b>53.59 ± 0.43</b></td><td><b>0.94</b></td><td><b>1.17</b></td><td><b>1.07</b></td><td><b>1.29</b></td><td><b>1.32</b></td><td><b>1.28</b></td><td><b>1.39</b></td><td><b>1.40</b></td><td><b>1.50</b></td></tr></tbody></table>

## 4.1. Performance evaluation of STL and MTL framework

We begin our experiments by evaluating the performance of the ASC system using the STL and MTL framework. For both these frameworks, the samples from the training split of the development dataset are used without applying any specific data augmentation technique. For MTL framework, the trade-off parameter, $\beta$ is varied from 0.1 to 0.5. Since our primary task is the 10-scene classification, we set the maximum value of $\beta$ as 0.5, which corresponds to equal weight to the loss of both tasks. The average log loss and the average accuracy obtained using the STL and MTL framework for various $\beta$ are shown in Table 1. It can be seen that the MTL framework with $\beta$ = 0.2 achieves the best performance with an average log loss of 1.319 and an average accuracy of 51.98%. In comparison, the STL framework achieves an average log loss of 1.333 and an average accuracy of 51.25%. From the results belonging to device-wise log loss, it can be seen that the use of MTL framework achieves comparatively lower log loss for most of the seen and unseen devices. In addition, we note that the MTL framework with $\beta$ = 0.2 achieves an average accuracy of ≈ 97% on the 4-device classification task. This indicates that enabling the shared layer to learn device-specific low-level features helps to improve the ASC task. Since the MTL framework with $\beta$ = 0.2 achieves the best ASC performance, we use the same for the rest of our experiments.

## 4.2. Performance evaluation of data augmentation techniques

In this set of experiments, we evaluate the performance of the proposed MTL framework using various data augmentation methods presented in Section 2.3. A given data augmentation technique is applied separately on all the samples in the training dataset for comparing its effect on the ASC task. We apply the time stretch, time shift and block mixing with equal probability and combine them as one data augmentation technique (TS + TSH + BM). The average log loss, average accuracy as well as the device-wise log loss obtained using none of the data augmentation techniques and applying each of them separately is shown in Table 3. It can be seen that each augmentation technique helps to improve the average log loss and accuracy. Since the objective of RawBoost technique is to generate unseen/simulated devices, its performance especially on devices S4 and S5 is improved, while performance on device A is degraded. Subsequently, all the data augmentation techniques are combined to train our final model. It achieves an average log loss and average accuracy of 1.265 and 53.59% respectively, which are better than those obtained with any of the individual data augmentation methods as well as those without any augmentation.

## 4.3. Comparison to other systems

We now compare the performance of the proposed MTL-based ASC system on the development set with the well performing systems

<table><caption>Table 3: Performance comparison of the proposed MTL-based system with other well performing systems on the development set of DCASE 2022 Task 1.</caption><thead><tr><th>System</th><th>Avg. Log Loss</th><th>Avg. Accuracy (%)</th></tr></thead><tbody><tr><td>Lee et al. [24]</td><td>0.835</td><td>70.1</td></tr><tr><td>Anastácio et al. [25]</td><td>1.103</td><td>60.5</td></tr><tr><td>Schmid et al. [26]</td><td>1.139</td><td>58.0</td></tr><tr><td>Sugahara et al. [23]</td><td>1.182</td><td>56.5</td></tr><tr><td>Kim et al. [27]</td><td>1.259</td><td>54.0</td></tr><tr><td>Proposed MTL (Ours)</td><td>1.273</td><td>53.5</td></tr><tr><td>Morocutti et al. [28]</td><td>1.288</td><td>52.7</td></tr><tr><td>Xin et al. [29]</td><td>1.295</td><td>60.3</td></tr><tr><td>Yu et al. [30]</td><td>1.305</td><td>51.7</td></tr><tr><td>Shao et al. [31]</td><td>1.360</td><td>54.1</td></tr><tr><td>Baseline [12]</td><td>1.575</td><td>42.9</td></tr></tbody></table>

submitted to the DCASE Task 1 challenge on low-complexity ASC as well as the challenge baseline. To this extent, we perform INT8 quantization on our best performing model as described in Section 2.2 for inference. The INT8 quantized model achieves an average log loss 1.273 and an average accuracy of 53.5%. The reported performances of other systems and the baseline, sorted with reference to average log loss are shown in Table 3. It can be seen that we achieve a comparable performance with other systems with a significant improvement over the baseline system. We note that most of the proposed systems are based on STL frameworks, which utilize knowledge distillation technique for model design. However, Sugahara et al. [23] uses an MTL framework with the 3 broader ASC as the additional task. It is also noted that the baseline system uses a 3-layer CNN using log Mel-spectrograms with 46, 512 parameters and 29.23 MMACS [12].

## 5. CONCLUSIONS

In this paper, we explore the use of an MTL framework with the device classification task as an additional task to address the DCASE 2022 Task 1 on low-complexity ASC. The proposed CNN model is designed such that it meets the Task 1 model complexity constraints. We also explore the use of various data augmentation techniques to improve the generalization of the model. Evaluations on the development set show that the use of MTL framework along with the various data augmentation techniques help to improve the performance of the ASC task.
## 6. REFERENCES

[1] D. Wang and G. J. Brown, Computational auditory scene analysis: Principles, algorithms, and applications. Wiley-IEEE press, 2006.

[2] T. Virtanen, M. D. Plumbley, and D. Ellis, Computational analysis of sound scenes and events. Springer, 2018.

[3] L. Sicong, Z. Zimu, D. Junzhao, S. Longfei, J. Han, and X. Wang, "UbiEar: Bringing location-independent sound awareness to the hard-of-hearing people with smartphones," in Proc. ACM Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT), vol. 1, no. 2, pp. 1-21, 2017.

[4] R. Radhakrishnan, A. Divakaran, and A. Smaragdis, "Audio analysis for surveillance applications," in Proc. IEEE Workshop Applicat. Signal Process. Audio Acoust. IEEE, 2005, pp. 158-161.

[5] S. Suh, S. Park, Y. Jeong, and T. Lee, "Designing acoustic scene classification models with CNN variants," DCASE2020 Challenge, Tech. Rep., June 2020.

[6] B. Kim, S. Yang, J. Kim, and S. Chang, "Domain generalization on efficient acoustic scene classification using residual normalization," in Proc. DCASE Workshop, 2021, pp. 21-25.

[7] M. Kośmider, "Spectrum correction: Acoustic scene classification with mismatched recording devices," in Proc. Inter-speech, pp. 4641-4645, 2020.

[8] C.-H. H. Yang, H. Hu, S. M. Siniscalchi, Q. Wang, W. Yuyang, X. Xia, Y. Zhao, Y. Wu, Y. Wang, J. Du, and C.-H. Lee, "A lottery ticket hypothesis framework for low-complexity device-robust neural acoustic scene classification," DCASE2021 Challenge, Tech. Rep., June 2021.

[9] H. Hee-Soo, J. Jee-weon, S. Hye-jin, and L. Bong-Jin, "Clova submission for the DCASE 2021 challenge: Acoustic scene classification using light architectures and device augmentation," DCASE2021 Challenge, Tech. Rep., June 2021.

[10] K. Koutini, S. Jan, and G. Widmer, "CPJKU submission to DCASE21: Cross-device audio scene classification with wide sparse frequency-damped CNNs," DCASE2021 Challenge, Tech. Rep., June 2021.

[11] N. Zhao, "Low-complexity acoustic scene classification using knowledge distillation and multiple classifiers," DCASE2021 Challenge, Tech. Rep., June 2021.

[12] I. Martín-Morató, F. Paissan, A. Ancilotto, T. Heittola, A. Mesaros, E. Farella, A. Brutti, and T. Virtanen, "Low-complexity acoustic scene classification in DCASE 2022 challenge," arXiv preprint arXiv:2206.03835, 2022.

[13] H.-j. Shim, J.-w. Jung, J.-h. Kim, and H.-J. Yu, "Attentive max feature map and joint training for acoustic scene classification," in Proc. ICASSP. IEEE, 2022, pp. 1036-1040.

[14] Z. Ren, Q. Kong, J. Han, M. D. Plumbley, and B. W. Schuller, "CAA-Net: Conditional atrous CNNs with attention for explainable device-robust acoustic scene classification," IEEE Transactions on Multimedia, vol. 23, pp. 4131-4142, 2020.

[15] R. Krishnamoorthi, "Quantizing deep convolutional networks for efficient inference: A whitepaper," arXiv preprint arXiv:1806.08342, 2018.

[16] J. Salamon and J. P. Bello, "Deep convolutional neural networks and data augmentation for environmental sound classification," IEEE Signal processing letters, vol. 24, no. 3, pp. 279-283, 2017.

[17] H. Hu, C.-H. H. Yang, X. Xia, X. Bai, X. Tang, Y. Wang, S. Niu, L. Chai, J. Li, H. Zhu, et al., "A two-stage approach to device-robust acoustic scene classification," in Proc. ICASSP. IEEE, 2021, pp. 845-849.

[18] D. Hendrycks, N. Mu, E. D. Cubuk, B. Zoph, J. Gilmer, and B. Lakshminarayanan, "AugMix: A simple method to improve robustness and uncertainty under data shift," in Proc. ICLR, 2020.

[19] H. Tak, M. Kamble, J. Patino, M. Todisco, and N. Evans, "RawBoost: A raw data boosting and augmentation method applied to automatic speaker verification anti-spoofing," in Proc. ICASSP. IEEE, 2022, pp. 6382-6386.

[20] T. Heittola, A. Mesaros, and T. Virtanen, "Acoustic scene classification in DCASE 2020 challenge: generalization across devices and low complexity solutions," in Proc. DCASE Workshop, 2020, pp. 56-60.

[21] D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization," arXiv preprint arXiv:1412.6980, 2014.

[22] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, "Focal loss for dense object detection," in Proc. ICCV, 2017, pp. 2980-2988.

[23] R. Sugahara, R. Sato, M. Osawa, Y. Yuno, and C. Haruta, "Self-ensemble with multi-task learning for low-complexity acoustic scene classification," DCASE2022 Challenge, Tech. Rep., June 2022.

[24] J.-H. Lee, J.-H. Choi, P. M. Byun, and J.-H. Chang, "HYU submission for the DCASE 2022: Efficient fine-tuning method using device-aware data-random-drop for device-imbalanced acoustic scene classification," DCASE2022 Challenge, Tech. Rep., June 2022.

[25] R. Anastácio, L. Ferreira, F. Mónica, and C. B. Luís, "Ai4edgept submission to DCASE 2022 low complexity acoustic scene classification task1," DCASE2022 Challenge, Tech. Rep., June 2022.

[26] F. Schmid, S. Masoudian, K. Koutini, and G. Widmer, "CPJKU submission to DCASE22: Distilling knowledge for low-complexity convolutional neural networks from a patchout audio transformer," DCASE2022 Challenge, Tech. Rep., June 2022.

[27] T. Kim, G. Lee, and J. Park, "KT submission for the DCASE 2022 challenge: Modernized convolutional neural networks for acoustic scene classification," DCASE2022 Challenge, Tech. Rep., June 2022.

[28] T. Morocutti and D. Shalaby, "Receptive field regularized CNNs with traditional audio augmentations," DCASE2022 Challenge, Tech. Rep., June 2022.

[29] Y. Xin, Y. Zou, F. Cui, and Y. Wang, "Low-complexity acoustic scene classification with mismatch-devices using separable convolutions and coordinate attention," DCASE2022 Challenge, Tech. Rep., June 2022.

[30] J. Yu, R. Shi, T. He, and K. Guo, "Acoustic scene classification based on feature fusion and dilated-convolution," DCASE2022 Challenge, Tech. Rep., June 2022.

[31] Y.-F. Shao, X. Zhang, G.-G. Bing, K.-M. Zhao, J.-J. Xu, Y. Ma, and W.-Q. Zhang, "Mini-segnet for low-complexity acoustic scene classification," DCASE2022 Challenge, Tech. Rep., June 2022.
# LOW-COMPLEXITY ACOUSTIC SCENE CLASSIFICATION IN DCASE 2022 CHALLENGE

Irene Martín-Morató¹, Francesco Paissan², Alberto Ancilotto², Toni Heittola¹,
Annamaria Mesaros¹, Elisabetta Farella², Alessio Brutti², Tuomas Virtanen¹

¹Computing Sciences, Tampere University, Finland
{irene.martinmorato, toni.heittola, annamaria.mesaros, tuomas.virtanen}@tuni.fi

² Fondazione Bruno Kessler, Italy
{fpaissan, aancilotto, efarella, brutti}@fbk.eu

## ABSTRACT

This paper presents an analysis of the Low-Complexity Acoustic Scene Classification task in DCASE 2022 Challenge. The task was a continuation from the previous years, but the low-complexity requirements were changed to the following: the maximum number of allowed parameters, including the zero-valued ones, was 128 K, with parameters being represented using INT8 numerical format; and the maximum number of multiply-accumulate operations at inference time was 30 million. Despite using the same previous year dataset, the audio samples have been shortened to 1 second instead of 10 second for this year challenge. The provided baseline system is a convolutional neural network which employs post-training quantization of parameters, resulting in 46.5 K parameters, and 29.23 million multiply-and-accumulate operations (MMACs). Its performance on the evaluation data is 44.2% accuracy and 1.532 log-loss. In comparison, the top system in the challenge obtained an accuracy of 59.6% and a log loss of 1.091, having 121 K parameters and 28 MMACs. The task received 48 submissions from 19 different teams, most of which outperformed the baseline system.

**Index Terms—** Acoustic scene classification, low-complexity, DCASE Challenge

## 1. INTRODUCTION

The task of acoustic scene classification is defined as classifying a short excerpt of audio into a class of a predefined set of classes, that indicates the context where the audio was recorded [1]. The task has been one of the main topics in the DCASE Challenge from its inception, and has developed from the original setup to include different additional problems, such as multiple devices and low-complexity conditions [2]. The current setup advances further towards real-world applicability by defining the low-complexity constraints, in terms of maximum number of parameters and maximum number of operations permitted at inference time, typical of current IoT devices (or microcontrollers).

For real-world applications, a classification method for acoustic scenes is expected to work in very diverse conditions, including audio captured with different devices and as short as possible inference time. The first task on low-complexity acoustic scene classification was defined in 2020 for only three classes and a single device [2], for which many submissions obtained very high performance. The solutions most commonly imposed restrictions on the

<figure><img src="image_73.png" /><figcaption>Figure 1: Acoustic scene classification for audio recordings.</figcaption></figure>

model architectures, using slim models and depth-wise separable CNNs. In addition, pruning and post-training quantization of the model weights were popular choices [3]. The data mismatch between training and testing when dealing with multiple devices has been first introduced as a separate task in 2019, and then repeated in 2020. The majority of the systems handled the mismatch through data augmentation [2], with the best performance in the 2020 task being 76.5% accuracy and 1.21 log loss [4].

The combination of multiple devices and low-complexity requirements was introduced in DCASE 2021 Challenge, with the model complexity limit being set to 128 K for the non-zero parameters. This created the situation in which most high-performing systems were very close to the allowed model size limit. Sparsity used in combination with quantization emerged as a popular and efficient way of reducing the model size, while the most popular system architectures submitted in 2021 were residual models. A few teams used modified versions of available residual models suitable for processing power-constrained devices, like MobileNet [5] and EfficientNet [6]. The best performing system had an accuracy of 76.1% and log loss of 0.724 [7], while 18 submitted systems had an accuracy above 70%.

The current edition formulates the problem of low-complexity acoustic scene classification by defining more concretely the low-complexity limitations by selecting a class of target devices for which the developed system should be suitable. This results in the number of allowed parameters being maximum 128 K, counting all parameters, in contrast from DCASE 2021 when only the non-zero ones were counted. In addition, a limit of 30 million multiply-accumulate operations (MMACs) is approximated based

<footnote>This work was partially funded by the EU H2020 project MARVEL (project number: 957337).</footnote>
on the computing power of the target device class.

This paper introduces the results and analysis of the DCASE 2022 Challenge Task 1: Low-Complexity Acoustic Scene Classification with Multiple Devices. The paper is organized as follows: Section 2 introduces the task setup, dataset, and baseline system. Sections 3 and 4 present the challenge participation statistics and analysis of the submitted systems, respectively, while Section 5 presents conclusions and ideas for future development of this task.

## 2. TASK SETUP

The particular aspect of this task is moving towards practical considerations that bring developed systems closer to possible target devices that impose constraints on computing power and capacity. Generalization across different devices is already a longer running feature of the acoustic scene classification task.

### 2.1. Dataset and performance evaluation

The task uses TAU Urban Acoustic Scenes 2022, a newly released version of the previous acoustic scene datasets. The data consists of recordings from ten acoustic scenes which represent the target classes [8]: airport, indoor shopping mall, metro station, pedestrian street, public square, street with medium level of traffic, travelling by a tram, travelling by a bus, travelling by an underground metro and urban park. Data was recorded in multiple European cities, with recordings from ten cities available in the training set and 12 in the evaluation set (two new cities compared to the training).

The audio files have been recorded simultaneously with four devices denoted A, B, C, and D, and another 11 devices denoted S1-S11 were simulated using the audio from device A. The development and evaluation sets consist of 64 and 22 hours of data, respectively. For complete details on the dataset creation and the exact amounts of data per device, we refer the reader to [2]. The difference from the previous datasets is that for this edition the audio data is presented in segments having a duration of 1 s, in order to comply with the inference time and computational limitations imposed by the considered target devices.

The submissions were evaluated using multi-class cross-entropy and accuracy. Accuracy was calculated as macro-average (average of the class-wise performance for each metric), but because the data is balanced, this corresponds to the overall accuracy. The systems were ranked based on the multi-class cross-entropy (log loss), for a ranking independent of the operating point.

As in each edition of the challenge, the audio material in the evaluation data was released two weeks prior to the challenge deadline. The participants were expected to provide class predictions for the provided audio material, and submit the system output for evaluation, together with additional information on the methods. The reference annotation of the evaluation data is only available to task organizers and was used for scoring the submissions.

### 2.2. System complexity requirements

The computational complexity is measured in terms of parameter count and MMACs (million multiply-accumulate operations) with the requirements modeled after Cortex-M4 devices (e.g. STM32L496@80MHz or Arduino Nano 33@64MHz).

The maximum number of parameters is 128 K, with variable type fixed into INT8, and counting all parameters. This is a major difference from DCASE 2021 in which the 128 K model size limit

<table><caption>Table 1: Baseline system size and performance on the development dataset. The value after ± is the standard deviation for 10 runs.</caption><thead><tr><td>System</td><td>Log loss</td><td>Accuracy</td><td>MMACs</td></tr></thead><tbody><tr><td>keras</td><td>1.575 (± 0.018)</td><td>42.9% (± 0.77)</td><td>29.23 M</td></tr></tbody></table>

was only for non-zero parameters, and there was no specific format imposed on the numerical representation. This change was made because in a real operational situation, even with a sparse model, the zero-valued parameters add to the number of MACs performed at inference, and produce additional computational overhead for handling sparsity.

The maximum number of MACS per inference is 30 MMACs, approximated based on the computing power of the target device class. This limit mimics the fitting of audio buffers into SRAM (fast access internal memory) on the target device for the analysis segment of 1 s, and allows some head space for feature calculation (e.g. FFT), assuming that the most commonly used features fit under this limit. In case learned features (embeddings) are used, e.g. VGGish [9], OpenL3 [10] or EdgeL3 [11], the network used to generate them contributes to the overall model size and complexity. Participants are required to provide full information about the model size and complexity in their technical report accompanying the submission. To facilitate model size calculation for the challenge participant, a script for calculating the number of parameters and the MMACs is provided for Keras, TFLite and PyTorch models<fnref n="1" />.

## 3. BASELINE SYSTEM

The baseline system has the same architecture as the 2021 one, being based on a convolutional neural network (CNN). The system consists of three CNN layers and one fully connected layer, followed by a softmax layer. The model is trained for 200 epochs with a batch size of 16. Complete details about the model and the parameters are provided with the code<fnref n="2" />. The feature extraction step follows a classical approach, where log mel-band energies are extracted every 40 ms with a 50% hop size. This results in an input shape of 40 × 51 for each 1 second audio file. Post-training quantization to 8 bits is used to reduce the model complexity. The quantization was done after training, using TFLite from TensorFlow 2.0, and setting the weights to INT8 type. The baseline system has a total number of parameters of 46512. The baseline system overall performance on the development data and system complexity information are provided in Table 1.

## 4. CHALLENGE RESULTS

The task received 48 submissions from a number of 19 teams. The number of participants in this edition is lower than in previous years, but similar to participation statistics of the other tasks. Only three of the 19 teams have lower performance than the baseline. The best system has a log loss of 1.091 and accuracy of 59.6%, with the four best spots belonging to team Schmid_CJJKU [12].

<footnote n="1">¹https://github.com/AlbertoAncilotto/NeSsi</footnote>

<footnote n="2">²https://github.com/marmoi/dcase2022_task1нострел</footnote>
<table><caption>Table 2: Performance on the evaluation set of the best systems for top 10 teams. The first column represents the overall rank of the system among 49 (including the baseline). The baseline is not officially ranked, but its log loss corresponds in order to place 26.</caption><thead><tr><th>Rank</th><th>Label</th><th>Log loss (95% CI)</th><th>Accuracy [%] (95% CI)</th></tr></thead><tbody><tr><td>1</td><td>Schmid_CPJKU_3</td><td>1.091 (1.040 - 1.141)</td><td>59.6 (59.4 - 59.9)</td></tr><tr><td>5</td><td>Chang_HYU_1</td><td>1.147 (1.081 - 1.214)</td><td>60.8 (60.6 - 61.1)</td></tr><tr><td>9</td><td>Morocutti_JKU_task1_4</td><td>1.311 (1.253 - 1.369)</td><td>54.5 (54.2 - 54.8)</td></tr><tr><td>11</td><td>AI4EDGE_IPL_4</td><td>1.330 (1.281 - 1.378)</td><td>51.6 (51.3 - 51.9)</td></tr><tr><td>14</td><td>Sugahara_RION_3</td><td>1.366 (1.305 - 1.426)</td><td>51.7 (51.4 - 51.9)</td></tr><tr><td>19</td><td>Park_KT_2</td><td>1.431 (1.364 - 1.498)</td><td>52.7 (52.4 - 53.0)</td></tr><tr><td>20</td><td>Zou_PKU_1</td><td>1.442 (1.362 - 1.521)</td><td>56.3 (56.0 - 56.6)</td></tr><tr><td>21</td><td>Yu_XIAOMI_1</td><td>1.456 (1.409 - 1.504)</td><td>46.2 (46.0 - 46.5)</td></tr><tr><td>22</td><td>Houyb_XDU_1</td><td>1.481 (1.416 - 1.547)</td><td>49.3 (49.0 - 49.5)</td></tr><tr><td>23</td><td>Singh_Surrey_3</td><td>1.492 (1.441 - 1.544)</td><td>45.9 (45.6 - 46.2)</td></tr><tr><td>(26)</td><td>DCASE2022 baseline</td><td>1.532 (1.490 - 1.574)</td><td>44.2 (44.0 - 44.5)</td></tr></tbody></table>

## 4.1. Performance analysis

The performance (log loss and accuracy) obtained by the top 10 teams, best system of each team, are presented in Table 2 and depicted in Figure 2. The submission label was simplified to remove redundant information; submission number was kept for correspondence with the results on the website<fnref n="3" />. The 95% confidence intervals for log loss were calculated using the jackknife procedure [13].

The ranking of the systems is based on log loss, where the top ranked one is the system of Schmid_CPJKU [12] with a log loss of 1.091. Its accuracy of 59.6% is second-best accuracy among the 48 submissions. Team Chang_HYU [14] is ranked second by log loss, but has the overall best accuracy among all submissions. Their accuracy is 60.8%, which appears to be significantly higher than Schmid_CPJKU according to the 95% confidence interval, given the amount of data in the evaluation set. Compared to last edition, the top accuracy has decreased by 16%, and the log loss of the top systems is much higher. While in 2021 there were 21 systems with a log loss under 1, this year there is none. Top 10 systems have a log loss under 1.5, and an accuracy between 45.9%-60.8%. The decrease in performance is mostly a consequence of the data segment size being reduced from 10 to 1 second.

Considering all submissions, the difference in performance between data belonging to devices seen or not in training is generally 10% in accuracy. However, the simulated unseen devices still have a better recognition rate than data from the real device D, which is the GoPro - it appears that its characteristics are very different from those of handheld devices developed for audio (mobile phones and tablets). Among the seen devices, the mobile devices have similar recognition rate, whether real (B, C) or simulated (S); systems have slightly better performance on device A. Given that most data in the development set belongs to device A, the relatively small difference in performance among devices shows that the systems have very strong generalization properties which cover the device mismatch. We also observe good generalization between seen and unseen cities, with almost no difference in classification performance between them. Class-wise performance indicates that some acoustic scenes are more difficult overall: while scenes like bus or park obtain accuracies over 70-80% for many systems, the large majority of systems classify scenes from pedestrian street and public square with around 30% accuracy only.

## 4.2. Machine learning characteristics

Regarding feature extraction, all the systems make use of log-mel energies or mel spectogram, sometimes in combination with other features like deltas, spectral entropy/flatness, CQT or Gammatone. Augmentation techniques are used by most of the systems, only 5 teams do not report use of augmentation<fnref n="4" />. The most popular technique is mixup (used by 33 systems), followed by SpecAugment and pitch shifting (used by 16 systems). Only one team, Zou_PKU [15] uses SpecAugment++, which is applied not only at the input but also at the hidden space of the neural network, to enhance also the intermediate feature representations. The system is ranked 7th based on the accuracy.

The most popular architectures are CNNs (used by 34 systems); some report use of MobileNet [5] (still convolutions, but depthwise separable) or BC-ResNet [7]. The use of residual models is reported by five teams, a significant reduction compared to the 2021 edition when residual networks were the most popular architecture. The top team Schmid_CPJKU uses a teacher-student setup, where the PaSST models pretrained on AudioSet are used as teacher, and the student model is a RF-regularized CNN [16]. The system is based on their previous submission's system reducing its complexity to fit the current constraints. For data augmentation they use Frequency MixStyle, mixing frequency-wise statistics to enhance device generalization.

## 4.3. System complexity analysis

Almost all submissions are based on inverted residual blocks, or a slight variation of this convolutional block. This is mainly because the common pattern for all participants was to adapt state-of-the-art convolutional networks to meet the computational requirements. Among the adapted networks there are MobileNets and BC-ResNets, and one submission with ShuffleNet. Other notable solutions include the use of very involved feature extraction solutions coupled with very simple neural architectures. While the networks where only slightly modified or carefully designed to meet the computational requirements (without any particular trick), a lot of focus was put on the training and data augmentation strategies. In particular, to boost inference performance, quantization-aware training (QAT) was applied by most of the participants. Another common

<footnote n="3">³https://dcase.community/challenge2022/task-low-complexity-acoustic-scene-classification-results</footnote>

<footnote n="4">⁴The analysis of machine learning characteristics is based on self-reported information provided by the authors with the submission of the system outputs.</footnote>
<figure><img src="image_74.png" /><figcaption>Figure 2: Classification log loss and accuracy for overall and for seen/unseen devices performance of top 10 teams, best system per team.</figcaption></figure>

<figure><img src="image_75.png" /><figcaption>Figure 3: Analysis of performance/complexity tradeoff for different submissions.</figcaption></figure>

alternative was knowledge distillation with pretrained bigger networks fine-tuned on the proposed task. Given the homogeneity in network topology of the submissions, the models proposed perform similarly in acoustic sound classification without being to diverse in computational requirements.

Three out of the top four performing models are based on architectures characterised by large receptive fields employing, respectively, a transformer architecture, coordinate attention and an encoder-decoder architecture. This proved to optimize the performance given the limited resources available, cleverly maximizing the working memory usage of the network, as this parameter was not limited in the task description. Another notable architecture is that proposed in AIT_Essex [17], providing almost optimal performance but very limited MMACs and/or parameter usage. This is possible thanks to their optimized convolutional block, which resembles a grouped convolution whose inputs are a combination of the original input sequence. More standard approaches, based on BC-ResNet, inverted residual blocks or standard bi-dimensional convolutions proved less effective at solving the task with the very limited resources available. This highlights the necessity to develop and optimize neural networks specifically for different hardware platforms. In JH_PM_HYU [14], the authors used clever regularization techniques in order to improve the generalization capabilities of the network. In conclusion, it is clear that, despite clever architectural designs, neural networks trained with optimized pre-processing and training strategies outperform the other approaches. In the future, it would be nice to see such techniques applied to the less computationally expensive models. A performance versus computational cost plot containing the best performing system of each participating team is presented in Figure 3.

## 5. CONCLUSIONS

This paper presented an analysis of the Low-Complexity Acoustic Scene Classification task in DCASE 2022 Challenge. The task was modeled after devices to bring the research problem closer to real-world applications. The number of MMAC set to 30 M and the total maximum number of parameters set to 128 K have been a sufficient constraint to receive a variety of interesting techniques, even though most systems were close to the imposed limits. The top systems employed large receptive fields, coordinate attention and transformer architectures to optimize performance, while quantization-aware training was the most used technique among participants to fulfill the complexity constraints. The number of submissions has decreased slightly from previous years, which may reflect the increased complexity of the task. However, the use of different devices for context-awareness is a sought after direction for applicability, therefore solutions suitable for limited computational power are needed. Moreover, the task could consider steering development towards solutions where room for improvement is still needed, like minimizing the working memory usage.
## 6. REFERENCES

[1] E. Benetos, D. Stowell, and M. D. Plumbley, *Approaches to Complex Sound Scene Analysis*. Cham: Springer International Publishing, 2018, pp. 215–242.

[2] T. Heittola, A. Mesaros, and T. Virtanen, “Acoustic scene classification in DCASE 2020 challenge: generalization across devices and low complexity solutions,” in *Proceedings of the Detection and Classification of Acoustic Scenes and Events 2020 Workshop*, Tokyo, Japan, November 2020, pp. 56–60.

[3] K. Koutini, F. Henkel, H. Eghbal-zadeh, and G. Widmer, “CP-JKU submissions to DCASE’20: Low-complexity cross-device acoustic scene classification with rf-regularized CNNs,” DCASE2020 Challenge, Tech. Rep., June 2020.

[4] S. Suh, S. Park, Y. Jeong, and T. Lee, “Designing acoustic scene classification models with CNN variants,” DCASE2020 Challenge, Tech. Rep., June 2020.

[5] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, “Mobilenetv2: Inverted residuals and linear bottlenecks,” in *2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition*, 2018, pp. 4510–4520.

[6] M. Tan and Q. V. Le, “Efficientnet: Rethinking model scaling for convolutional neural networks,” in *36th Int. Conf. on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA*, ser. Proceedings of Machine Learning Research, K. Chaudhuri and R. Salakhutdinov, Eds., vol. 97. PMLR, 2019, pp. 6105–6114.

[7] B. Kim, Y. Seunghan, K. Jangho, and C. Simyung, “QTI submission to DCASE 2021: Residual normalization for device-imbalanced acoustic scene classification with efficient design,” DCASE2021 Challenge, Tech. Rep., June 2021.

[8] A. Mesaros, T. Heittola, and T. Virtanen, “Acoustic scene classification in DCASE 2019 challenge: closed and open set classification and data mismatch setups,” in *Proc. of the DCASE 2019 Workshop*, New York, Nov 2019.

[9] S. Hershey, S. Chaudhuri, D. P. W. Ellis, J. F. Gemmeke, A. Jansen, R. C. Moore, M. Plakal, D. Platt, R. A. Saurous, B. Seybold, M. Slaney, R. J. Weiss, and K. Wilson, “CNN architectures for large-scale audio classification,” in *2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, 2017, pp. 131–135.

[10] J. Cramer, H.-H. Wu, J. Salamon, and J. P. Bello, “Look, listen and learn more: Design choices for deep audio embeddings,” in *IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP)*, Brighton, UK, May 2019, pp. 3852–3856.

[11] S. Kumari, D. Roy, M. Cartwright, J. P. Bello, and A. Arora, “EdgeL3: Compressing L3-net for mote scale urban noise monitoring,” in *2019 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW)*, May 2019, pp. 877–884.

[12] F. Schmid, S. Masoudian, K. Koutini, and G. Widmer, “CP-JKU submission to dcase22: Distilling knowledge for low-complexity convolutional neural networks from a patchout audio transformer,” DCASE2022 Challenge, Tech. Rep., June 2022.

[13] B. Efron, “Jackknife-after-bootstrap standard errors and influence functions,” *Journal of the Royal Statistical Society.*

*Series B (Methodological)*, vol. 54, no. 1, pp. 83–127, 1992.
[Online]. Available: http://www.jstor.org/stable/2345949

[14] J.-H. Lee, J.-H. Choi, P. M. Byun, and J.-H. Chang, “Hyu submission for the DCASE 2022: Efficient fine-tuning method using device-aware data-random-drop for device-imbalanced acoustic scene classification,” DCASE2022 Challenge, Tech. Rep., June 2022.

[15] Y. Xin, Y. Zou, F. Cui, and Y. Wang, “Low-complexity acoustic scene classification with mismatch-devices using separable convolutions and coordinate attention,” DCASE2022 Challenge, Tech. Rep., June 2022.

[16] K. Koutini, H. Eghbal-zadeh, and G. Widmer, “Receptive field regularization techniques for audio classification and tagging with deep convolutional neural networks,” *IEEE/ACM Transactions on Audio, Speech, and Language Processing*, vol. 29, pp. 1987–2000, 2021.

[17] L. Pham, A. Schindler, H. Tang, and T. Hoang, “DCASE 2021 task 1A: Technique report,” DCASE2021 Challenge, Tech. Rep., June 2021.
# A SUMMARIZATION APPROACH TO EVALUATING AUDIO CAPTIONING

Irene Martín-Morató, Manu Harju, Annamaria Mesaros

Computing Sciences Tampere University, Finland
{irene.martinmorato, manu.harju, annamaria.mesaros}@tuni.fi

## ABSTRACT

Audio captioning is currently evaluated with metrics originating from machine translation and image captioning, but their suitability for audio has recently been questioned. This work proposes content-based scoring of audio captions, an approach that considers the specific sound events content of the captions. Inspired from text summarization, the proposed measure gives relevance scores to the sound events present in the reference, and scores candidates based on the relevance of the retrieved sounds. In this work we use a simple, consensus-based definition of relevance, but different weighing schemes can be easily incorporated to change the importance of terms accordingly. Our experiments use two datasets and three different audio captioning systems and show that the proposed measure behaves consistently with the data: captions that correctly capture the most relevant sounds obtain a score of 1, while the ones containing less relevant sounds score lower. While the proposed content-based score is not concerned with the fluency or semantic content of the captions, it can be incorporated into a compound metric, similar to SPIDEr being a linear combination of a semantic and a syntactic fluency score.

**Index Terms**— audio captioning, evaluation, content-based retrieval

## 1. INTRODUCTION

Automated captioning, the description of images, audio, or video content using unrestrained natural language, is an active research topic in all these fields. The first works in image captioning defined it as a machine translation task, and evaluated it using metrics from machine translation such as BLEU [1], METEOR [2] and ROUGE [3], which are primarily based on n-gram overlap between the reference and candidate caption. Subsequently, it was observed that these metrics do not correlate well with human opinion [4], resulting in development of new metrics optimized for image captioning such as SPICE [5] and CIDEr [6]. SPICE measures performance using a graph-based semantic representation that explicitly encodes the objects, attributes and relationships found in image captions, while CIDEr measures how well a candidate sentence matches the consensus of a set of image descriptions, using n-grams weighted using Term Frequency Inverse Document Frequency (TF-IDF) weighting, combining n-grams of varying lengths (typically up to 4-grams). Further optimization in image captioning resulted in SPIDEr [7], a linear combination of SPICE and CIDEr.

Audio captioning was defined and evaluated the same way as image captioning [8]. In the last few years, the DCASE Challenge has accelerated development of audio captioning methods, seen

clearly in the significant improvement of the evaluation results. Approaches are based on encoder-decoder systems, with the decoder usually a recurrent network with sequence-to-sequence modeling. Recently, the use of transformers has become very popular [9], with pretrained models such as BERT [10] and BART [11] consistently ranked as state of the art. The evaluation scheme has not changed, and consists of the same set of metrics, from BLEU and ROUGE to CIDEr, SPICE and SPIDEr.

The recent development of large language processing models, notably BERT [10], brought a new approach to measuring similarity of text, initiating research into metrics more suitable for measuring audio captioning outputs. For example the work in [12] proposed FENSE, a new metric which combines Sentence-BERT for semantic similarity [13] with an error detector to penalize erroneously formed sentences. FENSE was shown to correlate with human judgments, in experiments that evaluated the output of four different captioning systems on the Clotho dataset [14], and showed that FENSE ranked the best systems the same way as humans did.

We introduce a novel perspective to audio captioning, namely summarization. Instead of cross-modal machine translation, we regard audio captioning as *cross-modal summarization*. A detailed description of a complex acoustic scene using natural language would include information on all the different sounds present at the scene. However, humans in fact do not care or pay attention to everything, and may consider some content irrelevant. We assume that annotators required to describe audio content would include the most relevant content, subject to their own judgement. In this respect, the textual description can be viewed as a summary.

In this paper we propose a novel measurement for audio captioning, which considers the captions content in terms of sound events. The different sounds mentioned in the captions are given relevance scores based on the annotators' consensus in producing the reference captions. A candidate caption is then evaluated based on the relevance of its content with respect to the reference information, given that the optimal caption for an audio will contain the topmost relevant sounds. In effect, this is a content-based scoring scheme that rewards the captions which retrieve the most relevant sounds in the captioned audio. The contributions of this work are the following: (i) we formulate audio captioning as audio-to-text summarization; (ii) we propose a method to estimate relevance of sound events based on multiple reference captions; (iii) we propose a relevance-based score for evaluation of audio captions content.

The paper is organized as follows: Section 2 introduces the proposed sound relevance estimation scheme and the proposed metric for evaluating candidate captions. Section 3 presents evaluation results using different captioning systems and datasets and introduces ideas for possible further development. Finally, Section 4 presents the conclusions.

<footnote>This paper has received funding from Jane and Aatos Erkko Foundation, Finland.</footnote>
## 2. CONTENT-BASED EVALUATION OF CAPTIONS

This work is inspired by the Pyramid method for evaluating content selection of textual summaries [15]. It was observed that among textual summaries produced by humans, many seem equally good without having identical content. This is valid for human-produced audio captions too, in particular for complex scenes. The Pyramid method starts by annotating “summarization content units” (SCU), then uses these SCUs to produce an optimal summary. The Pyramid score is defined as the ratio of a candidate summary to the ideal summary having the same number of SCUs.

Considering the caption as a summary, we define our content units as sound events. In consequence, we will evaluate a caption based on the sound events that are mentioned in this caption. The sound events are extracted from the captions using the AudioSet ontology [16], on the grounds that while it may be an incomplete list of possible sounds, the ontology provides a large set of the most common sounds. The human-produced reference captions are processed to extract the sound events from each caption, which are then assigned a relevance as explained in the following.

### 2.1. Estimating relevance of sound events

In its simplest way, the relevance of a sound event to an audio clip can be defined based on how many annotators have referred to it in the caption they provided. Consider the example from Table 1: given ten captions for one audio clip, four of them mention birds singing, two mention car passing by, all ten mention children laughing, and nine mention children talking.

We define relevance of sound event *i* as its consensus-based weight, calculated as:

$$ rel_i = \frac{N_i}{\sum_{j=1}^{M} (N_j)} \qquad (1) $$

where $N_i$ is the number of times sound event *i* is mentioned in the captions assigned to a clip, with *M* being the total number of sound events mentioned in all captions. The effect of this definition is that a relevance of 1 is distributed among all the mentioned sound event classes *M* based on how frequently they appear in the captions:

$$ \sum_{i=1}^{M} rel_i = 1. \qquad (2) $$

For the example in Table 1, bird singing has a relevance of 0.16, while children laughing has a relevance of 0.40.

One may argue that in certain cases the rare sound events may be more relevant to a clip, instead of the most commonly mentioned ones. Depending on the application and the desired output, the relevance of individual sound events to a clip can be estimated based on direct frequency as in the example above, or using TF-IDF weighing to give more weight to sounds that are very specific to a clip. For simplicity, we only use the former approach in this work, and leave other weighing schemes for future development.

### 2.2. Evaluation of candidate captions

The content-based score (CB-score) of a candidate caption *C* containing *K* sound events is defined as the ratio between the relevance

<table><caption>Table 1: Consensus-based relevance of sound events to a clip</caption><thead><tr><td>Label</td><td>Freq</td><td>Relevance</td></tr></thead><tbody><tr><td>Bird singing</td><td>4</td><td>0.16</td></tr><tr><td>Car passing by</td><td>2</td><td>0.08</td></tr><tr><td>Children laughing</td><td>10</td><td>0.40</td></tr><tr><td>Children talking</td><td>9</td><td>0.36</td></tr></tbody></table>

of its content and the relevance of the optimal caption $C_K$ containing the same number *K* of sound events:

$$ CB\text{-score} = \frac{\sum_{j=1}^{K} rel_j}{\sum_{k=1}^{K} rel_k} \qquad (3) $$

where sound events *j* belong to the candidate caption *C*, and events *k* belong to the optimal caption $C_K$. The optimal caption $C_K$ is defined as containing the most relevant *K* of the *M* sound events mentioned in the reference captions. This means that for the example in Table 1 the optimal caption containing only one sound event would contain “children laughing”, while the optimal caption containing two sound events would contain “children laughing” and “children talking”. Table 2 gives examples of CB-score calculation for different captions, using the relevance scores from Table 1.

## 3. EXPERIMENTS

For each audio clip, the reference captions are processed to extract sound events and estimate their consensus-based relevance. Then, the candidate caption is evaluated against the optimal caption as explained in the previous section. Experiments were performed using three different systems and multiple datasets, in order to verify the behavior of the metric. For comparison, SPIDEr scores (as used in DCASE Challenge 2022) and FENSE scores were calculated.

### 3.1. Captioning datasets and systems

The datasets used for evaluating the behaviour of the proposed metric are Clotho [14] which has five captions per clip, for audio clips 15 to 30 seconds long that were collected from Freesound [17] and AudioCaps [18], consisting of 10-second long audio clips from AudioSet [16], for which only the test split has five captions per clip, the rest has only one caption.

We use three different systems to generate automatic captions. The first system is the DCASE task6 subtask A baseline system<fnref n="1" />, which is a sequence-to-sequence transformer based on BART model<fnref n="2" />. The second system (ED-RNN) consists of an Encoder-Decoder architecture with an attention layer in the decoder and bi-directional RNN in the encoder. The third model (AACTrans) is a sequence-to-sequence transformer similar to the baseline system, but having smaller number of parameters and using greedy token generation. All three models use VGGish [19] features as inputs. For brevity, we do not present more details about the systems, since they are irrelevant for the objective of this study. The systems are separately trained and tested using Clotho and AudioCaps, respectively, using the training/test splits provided with the datasets.

<footnote n="1">¹https://github.com/felixgontier/dcase-2022-baseline</footnote>

<footnote n="2">²https://huggingface.co/docs/transformers/model_doc/bart</footnote>
Candidate captions:

C1. Children are talking outside.

C2. A dog is barking at a car passing by.

C3. A car is passing by a group of children that are playing and laughing.

<table><caption>Table 2: Example of CB-scores for candidate captions, based on relevance scores from Table 1.</caption><thead><tr><th></th><th>Sound events</th><th>Relevance</th><th>Ideal caption content</th><th>CB-score</th></tr></thead><tbody><tr><td>C1</td><td>Children talking</td><td>0.36</td><td>Children laughing</td><td>0.36/0.40=0.90</td></tr><tr><td>C2</td><td>Dog barking, Car passing by</td><td>0.08</td><td>Children laughing, Children talking</td><td>0.08/(0.40+0.36)=0.11</td></tr><tr><td>C3</td><td>Car passing by, Children laughing</td><td>0.08+0.40</td><td>Children laughing, Children talking</td><td>0.48/0.76=0.63</td></tr></tbody></table>

<table><caption>Table 3: Performance of different systems on Clotho and AudioCaps datasets.</caption><thead><tr><th rowspan="2">System</th><th colspan="3">CLOTHO</th><th colspan="3">AudioCaps</th></tr><tr><th>SPIDEr</th><th>FENSE</th><th>CB-score</th><th>SPIDEr</th><th>FENSE</th><th>CB-score</th></tr></thead><tbody><tr><td>Baseline</td><td>0.22 (0.21, 0.24)</td><td>0.46 (0.45, 0.47)</td><td>0.49 (0.46, 0.51)</td><td>0.34 (0.32, 0.37)</td><td>0.57 (0.56, 0.58)</td><td>0.63 (0.60, 0.65)</td></tr><tr><td>ED-RNN</td><td>0.15 (0.14, 0.16)</td><td>0.41 (0.40, 0.43)</td><td>0.40 (0.38, 0.43)</td><td>0.30 (0.28, 0.33)</td><td>0.54 (0.53, 0.55)</td><td>0.62 (0.59, 0.64)</td></tr><tr><td>AACTransformer</td><td>0.19 (0.18, 0.21)</td><td>0.40 (0.39, 0.41)</td><td>0.47 (0.45, 0.50)</td><td>0.35 (0.32, 0.37)</td><td>0.54 (0.53, 0.55)</td><td>0.65 (0.63, 0.68)</td></tr><tr><td>Reference caption</td><td>0.58 (0.55, 0.61)</td><td>0.58 (0.57, 0.59)</td><td>0.64 (0.62, 0.66)</td><td>0.56 (0.52, 0.59)</td><td>0.68 (0.68, 0.69)</td><td>0.76 (0.75, 0.78)</td></tr></tbody></table>

## 3.2. Practical implementation details

The extraction of sound events from the textual captions is not possible using simple word matching, due to the unconstrained nature of the annotation process. We therefore match individual terms and their synonyms in a controlled manner, to extract what we call sound event tokens as well as possible. The processing steps are the following: First, AudioSet vocabulary and the reference captions are tokenized and lemmatized to obtain the root form for each word. Then, for each token in the caption, its first order synonym and first order hypernym are extracted from WordNet [20], based on its POS (part of speech) obtained using spaCy [21] and nltk [22]. This allows identifying words that are related to sounds, when they do not match the exact vocabulary of AudioSet. AudioSet vocabulary is used as a two-level hierarchy to standardize the depth of the vocabulary. If the child node in AudioSet does not match the extracted sound event from the caption but its synonym does, the extracted token is matched to the parent node. For example “people talking” is processed to “group, people, citizenry, speak, talk, communicate”, where speak is a child of speech in AudioSet, therefore talking is matched to speech.

We expect that this matching process will result in some amount of errors, in some cases matching wrong terms. However, we consider that the vocabulary used in the captions and in general for describing sounds are a small subset of WordNet, and words are mostly used with their most common meaning, which means that wrongly matched synonyms should not affect the scores very much. There are also cases where a correspondence to the AudioSet vocabulary is not found, and therefore the process fails to identify sounds.

## 3.3. Numerical results and analysis

The evaluation results are presented in Table 3 for the three systems, comparing SPIDEr, FENSE and CB-score. Confidence intervals for the metrics were calculated using the jackknife resampling procedure on the system output. To understand the dynamics of the metric values and to have better insight on the expected upper bound for each metric, we selected randomly one of the five reference captions and compared it to the other four using the three metrics. The

results of this evaluation are presented in the last row of Table 3.

Among the captioning systems, we observe that the baseline has the best performance on Clotho, a result which is consistent for all three metrics. On AudioCaps, the baseline and AACTransformer have similar performance, with confidence intervals of the metrics overlapping. In comparison, the evaluation of one reference caption against the others produces similar SPIDEr score on both datasets, while FENSE and CB-score are significantly higher on AudioCaps. It may be noteworthy that the datasets are of different size, with 46k and 29k captions available for training in AudioCaps and Clotho, respectively. Since usually larger datasets lead to more robust models, the size difference may explain the AudioCaps better scores.

To understand the meaning of these average values, the distribution of the scores calculated for the baseline system are illustrated in Fig. 1. We observe that SPIDEr is very concentrated close to 0, while FENSE looks normally distributed between 0 and 1. CB-score has many values at the extremes, with 0 corresponding to the case when no sounds were matched between the evaluated caption and the reference ones, and 1 for the case when the evaluated caption contained the most relevant sound(s). For Clotho 33% of the captions produced for the test split have CB-score 0, while 27% have a score of 1. On the other hand the FENSE score is more moderate, with most captions scoring between 0.4–0.6, only 13 files having a FENSE score of 0. SPIDEr scores are under 0.25 for 66% of the captions. AudioCaps data obtains more balanced scores, with only around 13% of the captions in the test split getting a CB-score of 0, while the maximal score of 1 is obtained by around 30% of the files, similar to Clotho.

For comparison, Fig. 2 shows the CB-score and FENSE distributions for evaluating one reference caption (selected randomly) against the others. Interestingly, 17% of the Clotho test split gets CB-score 0, which means that the captions produced by different people do not contain the same sound events. This information is not indicated by FENSE, which measures the general semantic similarity of the sentences. For both Clotho and AudioCaps, a CB-score of 1 is obtained by around 40% of the human-produced captions.

To analyze the content evaluated by the CB-score, we investigate the number of sound tokens extracted from the caption. The statistics are shown in Fig. 3; the prediction uses the output of the
<figure><img src="image_76.png" /><figcaption>Figure 1: Distribution of SPIDEr, FENSE and CB-score on the test splits of Clotho and AudioCaps for the baseline system.</figcaption></figure>

<figure><img src="image_77.png" /><figcaption>Figure 2: Distribution of FENSE and CB-score for evaluating one reference caption against the others for each clip in the test set.</figcaption></figure>

<figure><img src="image_78.png" /><figcaption>Figure 3: Statistics of the extracted tokens that refer to sound events in the reference captions and in the automatically generated ones.</figcaption></figure>

baseline system. For both Clotho and AudioCaps, our token extraction process results in a large number of terms per clip. On the other hand, 45% of the captions produced for Clotho result in only one token, while 6% do not match any sounds. For AudioCaps, the same system produces captions which are richer in sound tokens, and our token extraction method results in 2 or 3 tokens for 54% of them.

the CB-score itself, therefore the CB-score is flexible in allowing sound relevance score formulations depending on the target application. Additionally, given the relatively short sentences available as captions, we hypothesize that a higher number of reference captions would provide more reliable relevance estimates.

## 3.4. Discussion and further development

Based on its formulation in Section 2, it can be noticed that this is a precision-type of metric, reflecting how many of the sound events that appear in a caption are as highly weighted as possible. This scoring penalizes the presence of sounds if there are others more highly weighted but not included in the caption. A recall-oriented score can be formulated by defining the optimal caption as containing the *average number* of sound events in the reference annotations (instead of the same number *K* as the candidate). It can also be observed that the formulation in eq.3 is not sensitive to extra information, which in effect means that inserted sounds are not penalized, even if they were not present in the reference captions at all.

The idea of scoring the caption using consensus is not new: CIDEr also uses consensus, but is based on n-grams, while our proposal is based on sound events. We find the use of sound events as units of information more relevant to the audio captioning than the use of n-grams, even though this ignores the lexical structure. Because the definition of the sound relevance is independent from

## 4. CONCLUSIONS

We introduced CB-score, a metric to evaluate how well the sound events mentioned in automatically produced captions correspond to the sounds mentioned in the reference captions. We also introduced a simple method to estimate relevance of reference sounds based on multiple captions, which can be extended depending on the target application to give more importance to the most commonly mentioned sounds or to rare sounds that are highly specific to a clip. The proposed metric lacks the ability to measure lexical or grammar structure of the caption, therefore it is not a sufficient metric for evaluating audio captions, but it successfully summarizes the content of the acoustic description, which means that it can be used as such for evaluating captioning tasks focused on the correctness rather than syntactic richness of the produced sentences. Similar to SPIDEr being a linear combination of a semantic and a syntactic fluency score, the CB-score can further evolve into more complex compound measurements, with additional components measuring for example syntactic and semantic aspects of the produced caption.
## 5. REFERENCES

[1] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, "Bleu: A method for automatic evaluation of machine translation," in *Proceedings of the 40th Annual Meeting on Association for Computational Linguistics*, ser. ACL '02. USA: Association for Computational Linguistics, 2002, p. 311–318.

[2] A. Lavie and A. Agarwal, "METEOR: An automatic metric for MT evaluation with high levels of correlation with human judgments," in *Proceedings of the Second Workshop on Statistical Machine Translation*. Prague, Czech Republic: Association for Computational Linguistics, Jun. 2007, pp. 228–231.

[3] C.-Y. Lin, "ROUGE: A package for automatic evaluation of summaries," in *Text Summarization Branches Out*. Barcelona, Spain: Association for Computational Linguistics, Jul. 2004, pp. 74–81.

[4] M. Hodosh, P. Young, and J. Hockenmaier, "Framing image description as a ranking task: Data, models and evaluation metrics," *J. Artif. Int. Res.*, vol. 47, no. 1, p. 853–899, May 2013.

[5] P. Anderson, B. Fernando, M. Johnson, and S. Gould, "Spice: Semantic propositional image caption evaluation," in *Computer Vision – ECCV 2016*, B. Leibe, J. Matas, N. Sebe, and M. Welling, Eds. Cham: Springer International Publishing, 2016, pp. 382–398.

[6] R. Vedantam, C. L. Zitnick, and D. Parikh, "Cider: Consensus-based image description evaluation." in *CVPR*. IEEE Computer Society, 2015, pp. 4566–4575.

[7] S. Liu, Z. Zhu, N. Ye, S. Guadarrama, and K. Murphy, "Improved image captioning via policy gradient optimization of spider," in *2017 IEEE International Conference on Computer Vision (ICCV)*, 2017, pp. 873–881.

[8] K. Drossos, S. Adavanne, and T. Virtanen, "Automated audio captioning with recurrent neural networks," in *2017 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA)*, 2017, pp. 374–378.

[9] X. Mei, X. Liu, Q. Huang, M. D. Plumbley, and W. Wang, "Audio captioning transformer," in *Proceedings of the Detection and Classification of Acoustic Scenes and Events 2021 Workshop (DCASE2021)*, Barcelona, Spain, November 2021, pp. 211–215.

[10] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, "BERT: Pre-training of deep bidirectional transformers for language understanding," in *Proceedings of the 2019 Conference of the NAACL HLT, Vol. 1*. Minneapolis, Minnesota: Association for Computational Linguistics, Jun. 2019, pp. 4171–4186.

[11] M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer, "BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension," in *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, Jul. 2020, pp. 7871–7880.

[12] Z. Zhou, Z. Zhang, X. Xu, Z. Xie, M. Wu, and K. Q. Zhu, "Can audio captions be evaluated with image caption metrics?" in *ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, 2022, pp. 981–985.

[13] N. Reimers and I. Gurevych, "Sentence-BERT: Sentence embeddings using siamese bert-networks," in *EMNLP/IJCNLP*, K. Inui, J. Jiang, V. Ng, and X. Wan, Eds. Association for Computational Linguistics, 2019, pp. 3980–3990.

[14] K. Drossos, S. Lipping, and T. Virtanen, "Clotho: an audio captioning dataset," in *IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, 2020, pp. 736–740.

[15] A. Nenkova and R. Passonneau, "Evaluating content selection in summarization: The pyramid method," in *Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004*. Boston, Massachusetts, USA: Association for Computational Linguistics, May 2004, pp. 145–152.

[16] J. F. Gemmeke, D. P. W. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, "Audio set: An ontology and human-labeled dataset for audio events," in *IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, 2017, pp. 776–780.

[17] F. Font, G. Roma, and X. Serra, "Freesound technical demo," in *ACM International Conference on Multimedia (MM'13)*, ACM. Barcelona, Spain: ACM, Oct. 2013, pp. 411–412.

[18] C. D. Kim, B. Kim, H. Lee, and G. Kim, "AudioCaps: Generating captions for audios in the wild," in *Proceedings of the 2019 Conference of the NAACL HLT, Vol. 1*. Minneapolis, Minnesota: Association for Computational Linguistics, Jun. 2019, pp. 119–132.

[19] S. Hershey, S. Chaudhuri, D. P. W. Ellis, J. F. Gemmeke, A. Jansen, C. Moore, M. Plakal, D. Platt, R. A. Saurous, B. Seybold, M. Slaney, R. Weiss, and K. Wilson, "Cnn architectures for large-scale audio classification," in *IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, 2017, pp. 131–135. [Online]. Available: https://arxiv.org/abs/1609.09430

[20] C. Fellbaum, *WordNet: An Electronic Lexical Database*. The MIT Press, 05 1998. [Online]. Available: https://doi.org/10.7551/mitpress/7287.001.0001

[21] I. Montani, M. Honnibal, M. Honnibal, S. V. Landeghem, A. Boyd, H. Peters, P. O. McCann, M. Samsonov, J. Geovedi, J. O'Regan, D. Altinok, G. Orosz, S. L. Kristiansen, L. Miranda, D. de Kok, Roman, E. Bot, L. Fiedler, G. Howard, Edward, W. Phatthiyaphaibun, Y. Tamura, S. Bozek, murat, R. Daniels, M. Amery, B. Böing, B. Vanroy, and P. K. Tippa, "explosion/spaCy: v3.3.0: Improved speed, new trainable lemmatizer, and pipelines for Finnish, Korean and Swedish," Apr. 2022. [Online]. Available: https://doi.org/10.5281/zenodo.6504092

[22] S. Bird, E. Klein, and E. Loper, *Natural language processing with Python: analyzing text with the natural language toolkit*. "O'Reilly Media, Inc.", 2009.
# FEW-SHOT BIOACOUSTIC EVENT DETECTION USING AN EVENT-LENGTH ADAPTED ENSEMBLE OF PROTOTYPICAL NETWORKS

John Martinsson¹,²*, Martin Willbo¹, Aleksis Pirinen¹, Olof Mogren¹, Maria Sandsten²

¹Computer Science, RISE Research Institutes of Sweden, Sweden
{john.martinsson, martin.willbo, aleksis.pirinen, olof.mogren} @ri.se
² Centre for Mathematical Sciences, Lund University, Sweden
maria.sandsten@matstat.lu.se

## ABSTRACT

In this paper we study two major challenges in few-shot bioacoustic event detection: variable event lengths and false-positives. We use prototypical networks where the embedding function is trained using a multi-label sound event detection model instead of using episodic training as the proxy task on the provided training dataset. This is motivated by polyphonic sound events being present in the base training data. We propose a method to choose the embedding function based on the average event length of the few-shot examples and show that this makes the method more robust towards variable event lengths. Further, we show that an ensemble of prototypical neural networks trained on different training and validation splits of time-frequency images with different loudness normalizations reduces false-positives. In addition, we present an analysis on the effect that the studied loudness normalization techniques have on the performance of the prototypical network ensemble. Overall, per-channel energy normalization (PCEN) outperforms the standard log transform for this task. The method uses no data augmentation and no external data. The proposed approach achieves a F-score of 48.0% when evaluated on the hidden test set of the Detection and Classification of Acoustic Scenes and Events (DCASE) task 5.

**Index Terms—** Machine listening, bioacoustics, few-shot learning, ensemble

## 1. INTRODUCTION

The human-induced accelerated loss in biodiversity [1] has led to a need for automated and low-cost wildlife monitoring where machine learning is a promising way forward [2]. Passive acoustic monitoring (PAM) is becoming an important tool in ecology for monitoring animal populations through their vocalizations [3]. Annotating PAM data is costly and requires specific domain expertise which motivates research on few-shot learning for bioacoustic event detection [4]. The goal of few-shot bioacoustic event detection is to detect the onset and offset of animal vocalizations in sound recordings using only a few annotated examples.

Recent work has demonstrated that prototypical networks are a promising approach for few-shot sound event detection [5, 6, 7], but a remaining challenge is high variance in classification accuracy between models because of the small amount of training data. Recent work on audio classification and sound event detection has demonstrated promising results using ensembles [8, 9, 10]. Ensembles

may be especially useful for the few-shot task due to the high variance in classification accuracy between models [11]. To the best of our knowledge, prior work on ensemble methods for few-shot sound event detection remains understudied and motivated by this we study the effect of using an ensemble of prototypical networks for few-shot bioacoustic event detection.

Another challenge in few-shot sound event detection is the high variability in event lengths for the different event classes [6]. The event lengths can range from milliseconds to multiple seconds, which necessitates methods capable of adapting to the task specific event lengths. Wang et al. [6] suggest that future work should look into adapting the context window to the few-shot task. A common approach is to use a model which can handle variable context windows, train using a fixed context window, and at test time adapt the context window to the few-shot task. In this work we propose choosing the embedding function as well as the context window based on the few-shot examples. The embedding function is chosen from a set of embedding functions trained on different context windows, thus acting as experts on certain event lengths. Another way to approach this problem is by using a proposal based method [12].

## 2. METHOD

In this section we present our method which is based on prototypical networks [13] and extended with an event-length adapted ensemble. We describe how each embedding function for the prototypical networks is trained and how the embedding functions are selected based on the few-shot examples to produce an ensemble prediction at test time. The full source code and instructions on how to reproduce the results can be found at: https://github.com/johnmartinsson/few-shot-learning-bioacoustics.

### 2.1. Training the embedding function

The goal is to learn an embedding function from the base training data, acting as a proxy task for the few-shot task. The base training data set consists of annotated sound recordings for 47 known event classes and one “unknown” event class. The set of sound event classes are disjoint between the base training data and the few-shot task. We are given the start and end times $A_k = \{(s_i^k, e_i^k)\}_{i=1}^N$ of these classes, where $(s_i^k, e_i^k)$ denotes the start and end time of sound event class $k$ for annotation $i$. There is overlap in the annotations, i.e. two different sound events can occur (partially) simultaneously, and we therefore treat this as a multi-label problem. We model the

<footnote>*Thanks to the Swedish foundation for strategic research for funding.</footnote>
<figure><img src="image_79.png" /><figcaption>Figure 1: A log Mel spectrogram of part of a sound recording (top) and examples of predictions (bottom) from an ensemble prototypical network (solid blue line) and a prototypical network (dashed blue line) as well as the given few-shot examples (purple line) and remaining ground truth events (green line). The decision threshold τ is 0.5 (red line).</figcaption></figure>

47 known sound event classes and the “unknown” sound event class identically, yielding a total of K = 48 classes.

We assume that a fixed length audio segment $x \in \mathbb{R}^T$, consisting of $T$ consecutive audio samples, is fed to the embedding function $f_{\theta}^T: \mathbb{R}^T \rightarrow \mathbb{R}^M$ (see section 2.4 for further details), where $M \ll T$. We split the audio recordings into audio segments $x_i \in \mathbb{R}^T$ by sliding a window of size $T$ with a hop size of $T/2$ over each recording. For each audio segment $x_i$, a target vector $y_i \in \{0, 1\}^{K \times n}$ is derived. If $n = T$ it means that the target contains one label per audio sample. Choosing $n < T$ means that the temporal resolution for the target is reduced. The resulting dataset $\mathcal{D}_b = \{(x_i, y_i)\}_{i=1}^N$ defines the sound event detection task used to train the embedding function.

A prediction of the target classes for a given audio segment $x_i$ is derived by $\hat{y}_i = h_{\phi}(f_{\theta}^T(x_i))$, where $h_{\phi}(\cdot)$ is a linear layer followed by an element-wise sigmoid activation function, and $f_{\theta}^T(\cdot)$ is a convolutional neural network where the first layer is a (non-learnable) time-frequency transform.

The loss function is the mean element-wise binary cross-entropy between the target $y_i$ and the prediction $\hat{y}_i$, where the mean is taken over the class dimension K and the temporal dimension n.

For a fixed $T$, we train a set of C different embedding functions, parameterized as $\Theta = \{\theta_1, \dots, \theta_C\}$, each with different randomly initialized weights of the neural network, different training and validation splits of the base training data, and different time-frequency transforms in the first layer of the embedding function.

## 2.2. Prototypical network at test time

At test time we are given a sound recording and the $M = 5$ first event examples of the class of interest. We denote these $A_p = \{(s_i, e_i)\}_{i=1}^M$ and call them the positive sound events. We assume that the gaps between the positive event annotations are background noise and let $A_n = \{(e_i, s_{i+1})\}_{i=1}^{M-1}$ denote the start and end time of the $M-1$ first negative sound events. We assume the likelihood of an annotator missing events to be low.

Let $l_i = e_i - s_i$ be the length of annotation i. If $l_i < T$ we

“expand” the annotation with the $(T - l_i)/2$ preceding and subsequent audio samples to get an audio segment of length $T$, and if $l_i \ge T$ we do not expand. We then split this into segments of length $T$ by sliding a window of size $T$ over the signal with a hop size of $T/16$ (if expanded this will only result in one segment). Let $S_p$ denote the set of positive audio segments derived from these annotated start and end times, and let $S_n$ denote the set of negative audio segments. We use the embedding function $f_{\theta}^T$ and define the prototypes as

$$c_k = \frac{1}{|S_k|} \sum_{x \in S_k} f_{\theta}^T(x) \quad (1)$$

and derive a pseudo-probability of audio segment $x$ belonging to sound class $k$ from the prototypical network by

$$p_{\theta}(y = k | x) = \frac{\exp(-d(f_{\theta}^{T}(x), c_k))}{\sum_{k'} \exp(-d(f_{\theta}^{T}(x), c_{k'}))}, \quad (2)$$

where $k \in \{n, p\}$ and $d(f_{\theta}^T(x), c_k)$ denotes the Euclidean distance between the query $f_{\theta}^T(x)$ and the prototype $c_k$.

The query set $S_q$ is derived by sliding a window of size $T$ over the signal with a hop size of $T/2$. The reason for setting the hop size relative to $T$ is that this means that we do equally many predictions for each audio sample in the validation recordings.

## 2.3. Our contributions

We now present the two main contributions of this paper: i) an event-length adapted embedding function for the few-shot task, and ii) using an ensemble of predictions.

**Adapting the embedding function.** We use the annotated positive events $A_p = \{(s_i, e_i)\}_{i=1}^M$ and compute the set of event lengths $L = \{e_i - s_i\}_{i=1}^M$. We choose $T^* \in \{T_1, 2^1 T_1, 2^2 T_1, 2^3 T_1\}$ such that $\sqrt{(T - l_{min}/2)^2}$ is minimized, where $l_{min}$ is the shortest event length in $L$.

We choose $T_1 = 2048$ which is 0.09 seconds at a sampling rate of 22050 Hz so that we can plausibly detect the shortest
events in the few-shot validation set. We limit the amount of extra computation needed during training and the extra memory needed during inference by setting the maximum *T* to 2³*T₁*.

**Ensemble.** Let $\Theta = \{\theta_i^*\}_{i=1}^C$ denote the set of parameters of *C* different prototypical network models adapted to the average event length of the few-shot task. Then we define

$$p_{\Theta}(y = k|x) = \frac{1}{C} \sum_{\theta \in \Theta} p_{\theta}(y = k|x) \quad (3)$$

as in [14], which can be viewed as a uniformly-weighted mixture of experts. We say that *x* belongs to the positive event class if $p_{\Theta}(y = p|x) > \tau$ and otherwise *x* belongs to the negative event class. This is done for every $x \in S_q$. Finally, if the query is classified as a positive event then the start and end time associated with that query is used as the predicted positive event timings.

## 2.4. Details of the embedding function

The embedding function consists of a time-frequency transform followed by a convolutional neural network, both of which are briefly described below.

**Time-frequency transform.** The first layer of the embedding function is a time-frequency transform. We use the Mel transform where the number of Mel bins is 128, the window size is roughly 25ms, and the hop size is half the window size. We either use the log transform as a loudness normalization or we use PCEN [15] with fixed parameters developed for speech audio or for bioacoustics as suggested in [16].

**Convolutional neural network.** The convolutional neural network used is an adapted version of the 10-layer residual neural network [17] used in the baseline for the challenge. Specifically, we i) add the classification head $h_{\phi}(\cdot)$ so that we can model the defined multi-label task, ii) use the same number of filters in every convolutional layer, and iii) reduce the max pooling along the time-dimension when audio segments are too short.

## 2.5. Evaluation metric

The method is evaluated by taking the harmonic mean over the F-scores for the different subsets in the evaluation sets. The F-score is computed by a bi-partite matching between the predicted and ground truth events, where the requirement for a match is an intersection-over-union (IoU) of at least 0.3 [4].

## 2.6. Post-processing

Since we get one prediction for each query audio segment, this limits the possible length of the prediction with this model. To solve this, we simply merge all overlapping predicted positive events into one detected event with a single start and end time.

A predicted positive event will only be considered to be a match with a true positive event during evaluation if they have an intersection-over-union (IoU) of at least 0.3. We therefore remove predictions which are shorter than 0.3 * $l_{avg}$ or longer than (1/0.3) * $l_{avg}$, where $l_{avg}$ is the average event length of the given five annotations. Since predictions of these lengths can on average not be matched with true events as the evaluation is defined.

<table><caption>Table 1: Few-shot validation data statistics.</caption><thead><tr><th>Subset</th><th>Mean event length</th><th>Mean gap length</th><th>Mean density</th></tr></thead><tbody><tr><td>HB</td><td>11.25 ± 3.11</td><td>6.12 ± 5.39</td><td>0.73 ± 0.12</td></tr><tr><td>ME</td><td>0.22 ± 0.03</td><td>1.40 ± 0.04</td><td>0.17 ± 0.02</td></tr><tr><td>PB</td><td>0.12 ± 0.08</td><td>59.89 ± 55.55</td><td>0.01 ± 0.02</td></tr></tbody></table>

## 3. DATA

We use the few-shot examples to compute the mean event length, the mean gap length, and the density of annotated sound events – see table 1. The few-shot validation set consists of three different subsets: HB, ME, and PB. The HB subset contains long events with low noise. The ME subset contains short events with low noise. The PB subset contains very short events with very high noise. The mean event length is defined as the mean length of the five annotated events; the mean gap length is defined as the mean length of the *unannotated* gaps between the five annotated events; and the density is the sum of the time of the five annotated events divided by the total time. A full description of the dataset can be found in [18].

## 4. EXPERIMENTS AND RESULTS

We have trained each embedding function on the described multi-label task on the base training data using the Adam [19] optimizer with a learning rate of 1e-3. The network is trained on a random split with 80% training data and validated on the remaining 20%. Each network in the ensemble is trained on a different random split. The training proceeds until we have observed no reduction in validation loss for the last 10 epochs and the model with the lowest validation loss is chosen as the final model. The temporal resolution of the targets have been fixed to *n* = 16, meaning that we have 16 targets for any given audio segment.

In figure 2a we compare the F-score achieved on the few-shot validation set when using an ensemble of five predictions with using each of these predictions by themselves. The time-frequency transform used is PCEN (bioacoustics). The achieved F-score by the ensemble is higher than the best of these individual predictions for 0.4 ≤ τ ≤ 0.6, and outperforms or matches the mean of them for other τ. We also note that the optimal τ is around 0.7 for the single predictions, and moves to 0.6 for the ensemble.

In figure 2b we compare the F-score of a five prediction ensemble for each time-frequency transform and compare this to an ensemble over all three time-frequency transform ensembles. We do not observe a significant increase in F-score when comparing the time-frequency ensemble to the ensemble using the PCEN (bioacoustic) time-frequency transform, but the time-frequency ensemble outperforms the ensemble using PCEN (speech) and log Mel transform. The optimal threshold τ varies around 0.6 to 0.7 for the ensembles using a single transform, and is at 0.6 for the time-frequency ensemble.

In figure 2c we compare the F-score achieved on the few-shot validation set when using the event-length adapted embedding functions in the ensemble with using any of the fixed *T* ∈ {T₁, 2¹T₁, 2²T₁, 2³T₁}. Adapting the embedding function increases performance from 53.0% (using best *T* = 4096) to 60.0% F-score for τ = 0.6.

In table 2 we show an ablation study. Adapting the embedding function increases the F-score on average with 8.3 percentage points, and adding the ensemble increases the F-score an additional
<figure><img src="image_80.png" /><figcaption>Figure 2: A comparison between: (a) an ensemble of five predictions using embedding functions trained on PCEN (bioacoustics) features with each of the individual predictions, (b) an ensemble of embedding functions trained and tested on log Mel, PCEN (bioacoustics), or PCEN (speech), with an ensemble of predictions over all three, and (c) the adaptive embedding function with using each of the fixed size embedding functions respectively (sr denotes the sample rate). All results in the figure are derived on the few-shot validation set.</figcaption></figure>

<table><caption>Table 2: An ablation study of our system on the few-shot validation set where we add adaptive embedding functions and ensemble.</caption><thead><tr><th>Method</th><th>Ensemble</th><th>Adaptive</th><th>F-score</th></tr></thead><tbody><tr><td>Ours</td><td>No</td><td>No</td><td>41.3 ± 3.8</td></tr><tr><td>Ours</td><td>No</td><td>Yes</td><td>49.6 ± 5.3</td></tr><tr><td>Ours</td><td>Yes</td><td>Yes</td><td>60.0</td></tr></tbody></table>

<table><caption>Table 3: The final F-score evaluation on the hidden test set for the baselines provided by the challenge organizers: template matching (TM) and prototypical networks (PN), and the top five submissions for the challenge.</caption><thead><tr><th>System</th><th>External data</th><th>Augmentation</th><th>F-score</th></tr></thead><tbody><tr><td>Baseline (TM)</td><td>No</td><td>No</td><td>12.3</td></tr><tr><td>Baseline (PN)</td><td>No</td><td>No</td><td>5.3</td></tr><tr><td>Ours [20]</td><td>No</td><td>No</td><td>48.0</td></tr><tr><td>Tang et al., [21]</td><td>No</td><td>No</td><td>62.1</td></tr><tr><td>Liu et al., [22]</td><td>Yes</td><td>Yes</td><td>48.2</td></tr><tr><td>Hertkorn [23]</td><td>No</td><td>No</td><td>44.4</td></tr><tr><td>Liu et al., [24]</td><td>Yes</td><td>Yes</td><td>44.3</td></tr></tbody></table>

11.4 percentage points. We compare against a prototypical network using an embedding function (no ensemble) which has been trained on PCEN (speech) and a best performing fixed segment length of 4096. The F-score when no ensemble is performed is the average (and standard deviation) over each single network in the ensemble.

In Table 3 we present the F-score from the final evaluation on the hidden test set from the challenge. We include information on whether or not the system uses data augmentation techniques or external datasets during training.

## 5. DISCUSSION AND CONCLUSIONS

In this section we will discuss our results and relate them to the baselines which were all developed concurrently to our work.

During development of this method we observed that random sampling of $S_n$, the set of negative examples does not work well for validation files with high event densities, which is why we chose to

use the gaps between the first five annotated events instead. This observation was also made in concurrent work submitted to the challenge [21, 22, 24].

We further observed that a fixed audio segment size $T$ resulted in poor predictive performance on the few-shot validation set in cases where event-lengths deviated from size $T$. Indicating the importance of adapting the embedding function.

We observed that the optimal threshold was different for the few-shot validation tasks and choosing a default value of $\tau = 0.5$ to be detrimental. However, finding an optimal threshold for the few-shot tasks is a difficult problem. Using an ensemble alleviates this issue by moving the optimal threshold closer to the default value.

The ensemble improves performance by correctly predicting most true positives, while reducing the number of false positives. This could intuitively be thought of as the ensemble being in agreement for true positive predictions, the average of which still yields a high pseudo-probability, while being in disagreement when predicting false positives, the average of which would be closer to 0.5. This effect can be seen in Figure 1, where some of the false positives predicted when not using an ensemble (dashed blue line) are removed by using an ensemble of the predictions (solid blue line), leading to a reduction in false-positives.

The baselines in this study were all developed concurrently to our work. Tang et al., [21] propose using a frame-level cross-entropy loss function for training instead of episodic training as the proxy task. Our approach is similar when setting the temporal resolution $n$ of the target vector to the number of frames in the time-frequency image. The effect of varying temporal resolutions $n$ for the proxy task would be interesting to study in future work. Tang et al. [21] further propose an iterative training scheme to adapt their method to the few-shot task [21] where the unlabeled audio in the test files is iteratively classified and then used for training. Liu et al. [22] and Liu et al. [24] use transductive inference to better adapt to the evaluation set, and Hertkorn [23] studies the importance of choosing appropriate parameters for the used time-frequency transform.

In conclusion, we have shown that choosing the embedding function based on the event lengths will increase performance, and that false-positives can be reduced by an ensemble of predictions. We have also shown that out of the three time-frequency transforms we have studied, PCEN (bioacoustics) performs best, followed by PCEN (speech) and log Mel.
## 6. REFERENCES

[1] S. Díaz, J. Settele, E. S. Brondízio, H. T. Ngo, J. Agard, A. Ar-neth, P. Balvanera, K. A. Brauman, S. H. Butchart, K. M. Chan, A. G. Lucas, K. Ichii, J. Liu, S. M. Subramanian, G. F. Midgley, P. Miloslavich, Z. Molnár, D. Obura, A. Pfaff, S. Po-lasky, A. Purvis, J. Razzaque, B. Reyers, R. R. Chowdhury, Y. J. Shin, I. Visseren-Hamakers, K. J. Willis, and C. N. Za-yas, "Pervasive human-driven decline of life on Earth points to the need for transformative change," *Science*, vol. 366, no. 6471, 2019.

[2] D. Tuia, B. Kellenberger, S. Beery, B. R. Costelloe, S. Zuffi, B. Risse, A. Mathis, M. W. Mathis, F. van Langevelde, T. Burghardt, R. Kays, H. Klinck, M. Wikelski, I. D. Couzin, G. van Horn, M. C. Crofoot, C. V. Stewart, and T. Berger-Wolf, "Perspectives in machine learning for wildlife conserva-tion," *Nature Communications*, vol. 13, no. 1, pp. 1–15, 2022.

[3] R. Gibb, E. Browning, P. Glover-Kapfer, and K. E. Jones, "Emerging opportunities and challenges for passive acoustics in ecological assessment and monitoring," *Methods in Ecol-ogy and Evolution*, vol. 10, no. 2, pp. 169–185, 2019.

[4] V. Morfi, I. Nolasco, V. Lostanlen, S. Singh, A. Strandburg-Peshkin, L. Gill, H. Pamula, D. Benvent, and D. Stowell, "Few-Shot Bioacoustic Event Detection: A New Task at the DCASE 2021 Challenge," *Proceedings of the Detection and Classification of Acoustic Scenes and Events 2021 Workshop (DCASE2021)*, no. November, pp. 145–149, 2021.

[5] B. Shi, M. Sun, K. C. Puvvada, C. C. Kao, S. Matsoukas, and C. Wang, "Few-Shot Acoustic Event Detection Via Meta Learning," *ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings*, vol. 2020-May, pp. 76–80, 2020.

[6] Y. Wang, J. Salamon, N. J. Bryan, and J. P. Bello, "Few-shot sound event detection," *ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings*, vol. 2020-May, pp. 81–85, 2020.

[7] J. Pons, J. Serra, and X. Serra, "Training Neural Audio Classifiers with Few Data," *ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings*, vol. 2019-May, pp. 16–20, 2019.

[8] D. Lee, S. Lee, Y. Han, and K. Lee, "Ensemble of Convolutional Neural Networks for Weakly-Supervised Sound Event Detection Using Multiple Scale Input," *DCASE 2017*, vol. 1, no. November, pp. 14–18, 2017. [Online]. Available: http://www.cs.tut.fi/sgn/arg/dcase2017/documents/challenge_technical_reports/DCASE2017_Lee_199.pdf

[9] L. Nanni, Y. M. Costa, R. L. Aguiar, R. B. Mangolin, S. Brah-nam, and C. N. Silla, "Ensemble of convolutional neural net-works to improve animal audio classification," *Eurasip Journal on Audio, Speech, and Music Processing*, vol. 2020, no. 1, 2020.

[10] L. Nanni, G. Maguolo, S. Brahnam, and M. Paci, "An ensemble of convolutional neural networks for audio classification," *Applied Sciences (Switzerland)*, vol. 11, no. 13, pp. 1–27, 2021.

[11] N. Dvornik, J. Mairal, and C. Schmid, "Diversity with cooperation: Ensemble methods for few-shot classification," *Proceedings of the IEEE International Conference on Computer Vision*, vol. 2019-October, pp. 3722–3730, 2019.

[12] P. Wolters, C. Daw, B. Hutchinson, and L. Phillips, "Proposal-based Few-shot Sound Event Detection for Speech and Environmental Sounds with Perceivers," pp. 1–7, 2021. [Online]. Available: http://arxiv.org/abs/2107.13616

[13] J. Snell, K. Swersky, and R. Zemel, "Prototypical networks for few-shot learning," *Advances in Neural Information Processing Systems*, vol. 2017-Decem, pp. 4078–4088, 2017.

[14] B. Lakshminarayanan, A. Pritzel, and C. Blundell, "Simple and scalable predictive uncertainty estimation using deep ensembles," *Advances in Neural Information Processing Systems*, vol. 2017-December, pp. 6403–6414, 2017.

[15] Y. Wang, P. Getreuer, T. Hughes, R. F. Lyon, and R. A. Saurous, "Trainable frontend for robust and far-field keyword spotting," *ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings*, no. 1, pp. 5670–5674, 2017.

[16] V. Lostanlen, J. Salamon, M. Cartwright, B. Mcfee, A. Farnsworth, S. Kelling, and J. P. Bello, "Per-Channel Energy Normalization: Why and How," *IEEE SIGNAL PROCESSING LETTERS*, no. September, pp. 1–6, 2018. [Online]. Available: http://www.justinsalamon.com/uploads/4/3/9/4/4394963/lostanlen_pcen_spl2018.pdf

[17] K. He, X. Zhang, S. Ren, and J. Sun, "Deep Residual Learning for Image Recognition," *Arxiv.Org*, vol. 7, no. 3, pp. 171–180, 2015. [Online]. Available: http://arxiv.org/pdf/1512.03385v1.pdf

[18] I. Nolasco, S. Singh, E. Vidana-Villa, E. Grout, J. Morford, M. Emmerson, F. Jensens, H. Whitehead, I. Kiskin, A. Strandburg-Peshkin, L. Gill, H. Pamula, V. Lostanlen, V. Morfi, and D. Stowell, "Few-shot bioacoustic event detection at the dcase 2022 challenge," 2022. [Online]. Available: https://arxiv.org/abs/2207.07911

[19] D. P. Kingma and J. Ba, "Adam: A Method for Stochastic Optimization," pp. 1–15, 2014. [Online]. Available: http://arxiv.org/abs/1412.6980

[20] J. Martinsson, M. Willbo, A. Pirinen, O. Mogren, and M. Sandsten, "Few-shot bioacoustic event detection using a prototypical network ensemble with adaptive embedding functions," Tech. Rep., 2022.

[21] J. Tang, X. Zhang, T. Gao, D. Liu, X. Fang, J. Pan, Q. Wang, J. Du, K. Xu, and Q. Pan, "Few-shot embedding learning and event filtering for bioacoustic event detection," iFLYTEK Research Institute, Hefei, China, Tech. Rep., 2022.

[22] H. Liu, X. Liu, X. Mei, Q. Kong, W. Wang, and M. D. Plumbley, "Surrey system for DCASE 2022 task 5 : few-shot bioacoustic event detection with segment-level metric learning," University of Surrey, Surrey, United Kingdom, Tech. Rep., 2022.

[23] M. Hertkorn, "Few-shot bioacoustic event detection : don't waste information," ZF Friedrichshafen AG, Friedrichshafen, Germany, Tech. Rep., 2022.

[24] M. Liu, J. Zhang, L. Wang, J. Peng, and C. Hu, "Bit SRCB teams 's submission for DCASE2022 task5 - few-shot bioacoustic event detection," Beijing Institute of Technology, Beijing, China, Tech. Rep., 2022.
# QUANTITY OVER QUALITY? INVESTIGATING THE EFFECTS OF VOLUME AND STRENGTH OF TRAINING DATA IN MARINE BIOACOUSTICS

Andrea Napoli*, Paul R. White, Thomas Blumensath

Institute of Sound and Vibration Research

University of Southampton, UK

{an1g18, P.R.White, Thomas.Blumensath}@soton.ac.uk

## ABSTRACT

The trade-off between the quality and quantity of training data is considered for the detection of minke whale (Balaenoptera acutorostrata) vocalisations. The performance of two different detectors is measured across a range of label strengths using training sets of different sizes. A detector based on spectrogram correlation and a convolutional neural network (CNN) are considered. The results show that increasing label strength does not benefit either detector past a certain point, corresponding here to a label density of 60 to 70%. Performance is found to be good even when labels are extremely weak (4% label density). Additionally, it is noted that performance of the spectrogram correlation plateaus beyond the use of 5 training calls, whereas the CNN's performance continues to increase up to the maximum training set size tested. Finally, interaction effects are observed between label strength and quantity, indicating that larger training sets are more robust to weaker labels. Overall, these findings suggest that there is indeed a benefit to collecting more, lower quality data when training a CNN, but that for a correlation-based detector this is not the case.

**Index Terms**— Weak labels, marine bioacoustics, CNN, spectrogram correlation, sound event detection

## 1. INTRODUCTION

Passive acoustic monitoring (PAM) forms a major part of marine mammal conservation. Acoustic surveys are an effective and non-invasive means to further our understanding of species-wise geographic distributions, migration patterns and feeding grounds, monitor ecosystem health, and help to mitigate the impacts of human activity. Automated analysis of survey data can improve our ability to achieve these goals, whilst substantially reducing the manual effort required [1].

It is generally accepted that larger training sets allow deep neural networks to build richer representations and improve performance across many classification tasks [2]–[4]. However, labelling large-scale datasets is expensive and time-consuming, so weak labels are commonly used to allow more training data to be collected [4]–[7]. This has resulted in a “quantity over quality” mantra which is at risk of being rashly applied without giving due consideration to the data distributions, labelling techniques, and algorithms pertinent to a particular application. So, work is needed to empirically determine whether “quantity over quality” retains its relevance in marine bioacoustics.

Weak data in PAM can take many forms [1], but this study will consider the effects of label noise – when some training instances do not represent the label they are assigned. This is common when audio is labelled without exact temporal localisation of the signal of interest, which is the scenario presented here. The longer the label, the more irrelevant or even confounding information is likely to be present. The question is then how long the labels need to be in order to make best use of the analysts’ time – accepting some label noise allows the analyst to work faster and thus collect more samples.

Label noise can also be introduced through computer-assisted labelling. In PAM, manual labelling is often combined with generic energy detection and unsupervised clustering to reduce annotation effort, at the cost of increased label noise [8]–[10]. Here, the relevant parameter is the sensitivity threshold for the energy detectors. A lower threshold yields more training samples, but also results in more erroneous labels.

Regardless of the origin of the label noise, the recurring theme is that a direct trade-off exists between the strength and quantity of the training data created. Determining the best labelling strategy therefore requires a quantification of exactly how classification performance is affected by these two opposing variables. The impact of label noise has been already been investigated for a range of audio tasks [11]–[13]. Additionally, the effect of training set size is often reported when new models are developed [4]. However, varying both strength and quantity concurrently, and in a controlled manner, has not been previously considered. Conducting a two-way study is important, since the impacts of the two are not necessarily independent (termed *factor interaction*). Thus, the description of a method to determine the presence of any interaction effects has broader relevance to other audio domains as well.

Once the training data has been collected, several weak learning techniques can be applied. Strong label assumption training (SLAT) is a basic option that splits the audio into frames and then assigns the same parent label to each [4]. This creates more, smaller training instances, both of which benefit smaller datasets, but a portion of these may be incorrectly labelled. In this case, prior estimates of the size of this portion can be used to improve performance [14]. Other options include multiple-instance learning [15] and scalable variants [16], or the use of attention or recurrence mechanisms [17].

For many years, template matching techniques such as matched filtering [18], [19] and spectrogram correlation [19], [20] have been the PAM algorithms of choice for detecting call types with limited variation (known as stereotyped vocalisations). Following widespread adoption in many other disciplines, CNNs have also recently found success in this field [1], [10], [21], [22]. Since

<footnote>* Thanks to BAE Systems and the Engineering and Physical Sciences Research Council for funding.</footnote>
these methods differ significantly, but are both extensively used, an implementation of each will be tested. We stress that the objective of this is not to directly compare the performance of the two detection algorithms, but to identify any differences in how gradient-based learning and traditional signal processing methods are affected by the quality and quantity of data used.

In summary, this paper contributes the following:

* Design of two detectors for minke whale vocalisations, one based on spectrogram correlation, and the other a CNN.

* A quantification of how label strength affects the performance of these detectors. This is the first such study in marine bioacoustics, and also tests more datapoints than similar studies in other audio domains.

* The addition of a second dimension to the problem space, so as to explore the impact of both quantity and quality of labels simultaneously. Comments are also provided regarding interaction effects between the two factors.

## 2. DATA

The scenario considered is the detection of minke whales in towed-array data from the 2017 Hawaiian Islands Cetacean and Ecosystem Assessment Survey [23]. The dataset comprises over 23,000 60-second audio files, sampled at 500 kHz on 6 channels. The channels have varying signal-to-noise ratios (SNRs) based on the distances between the hydrophones and the ship. Like other marine mammals, minke whales are under threat from human activities including vessel strikes, fishing gear entanglement, noise pollution, and ingestion of debris, so collecting regular and accurate abundance estimates is important [24].

The minke whales in this area produce stereotyped “boing” calls. A boing comprises a brief pulse followed by a longer, frequency and amplitude modulated component, and has a peak frequency of 1.4 kHz, harmonics up to 9 kHz, and source levels around 150 dB re 1 µPa·m [25], Figure 1. Note that this image is thresholded to improve clarity for illustrative purposes, but the spectrograms used for the experiments are unthresholded to maximise detectability and for better generalisability of the results.

<figure><img src="image_81.png" /><figcaption>Figure 1: A typical minke whale boing.</figcaption></figure>

For this experiment, 40 calls are identified, and selected such that no audio file contains more than one call. These are strongly labelled by hand, with the start and end times determined to 0.1 s

precision. Call durations range from 1.4 to 4.3 s, with a mean of 2.7 s. For the non-target class, 40 files are manually verified to contain only ambient noise. All audio is taken from the same survey day to minimise data leakage. The files are split into 4 validation folds, each containing 30 training calls and 10 test calls. Hereinafter, audio will be referred to as *positive* if it contains a call, and *negative* otherwise.

In addition to the strong labels, 7 sets of weaker labels are generated by increasing the length of the label. Thus, weakly-labeled *positive* audio will contain varying amounts of ambient noise in addition to a call, depending on the label length. For the first 3 sets, the label length is variable, and is equal to the call length plus a fixed time quantity of 0.5, 1 or 1.5 s. Since no call is longer than 5 s, for weaker sets, the label length is fixed, and varies from 5 to 60 s. The extra time added is split randomly between the start and end of the strong label, so the calls can occur at any point in the audio. The strength of each set can be quantitatively measured by considering the call duration as a percentage of the overall label length, referred to as the *label density* [11]. Table 1 shows the label density for each training case.

<table><caption>Table 1: Average label density for each of the label strengths tested.</caption><thead><tr><td>Training Set</td><td>Label Density</td></tr></thead><tbody><tr><td>Strong</td><td>100%</td></tr><tr><td>Strong +0.5 s</td><td>84%</td></tr><tr><td>Strong +1 s</td><td>72%</td></tr><tr><td>Strong +1.5 s</td><td>63%</td></tr><tr><td>5 s</td><td>52%</td></tr><tr><td>10 s</td><td>26%</td></tr><tr><td>20 s</td><td>13%</td></tr><tr><td>60 s</td><td>4%</td></tr></tbody></table>

Spectrogram representations of the audio are generated using 250 ms Hamming windows with 75% overlap. The spectrograms are cropped to a narrow band between 1.3 and 1.6 kHz, containing only the peak frequency and one additional harmonic, as per Figure 1. The spectrograms are then divided into 1 s frames with 50% overlap, discarding any excess beyond the length of the label. The resulting frames measure 76 by 16 pixels. Each audio channel is treated independently, resulting in 6 times as many samples. Since the channels have varying SNRs, this acts as a form of data augmentation and helps to regularise the data. SLAT is then applied, so all the frames are assigned the label from their parent audio segment. Note that:

* The “Strong” set contains only samples where the calls entirely fill the frames.

* The “Strong +0.5 s” and “Strong +1 s” sets also include samples that only partially contain calls.

* The weaker sets include partial samples as well as samples that do not contain calls at all (but are still labelled *positive*).

For each label strength, the number of calls in the training sets is varied from 30 down to 1. Training on a single channel of one call is also considered. The test data is the same for every training case in the fold, and is always strongly labelled.

In every case, the *negative* audio contains *only* ambient sounds, which is not unrealistic for PAM data. Thus, the label noise is entirely one-sided, effectively rendering this a positive-unlabelled (PU) learning problem [26]. The classes are kept balanced by randomly undersampling from the *negative* audio.
<table><caption>Table 2: Average detection accuracies and standard deviations for different sized training sets and label strengths.</caption><thead><tr><th rowspan="2">№<br>Calls</th><th rowspan="2">Strong</th><th rowspan="2">Strong +0.5 s</th><th rowspan="2">Strong +1 s</th><th rowspan="2">Strong +1.5 s</th><th colspan="3">Spectrogram Correlation</th></tr><tr><th>5 s</th><th>10 s</th><th>20 s</th><th>60 s</th></tr></thead><tbody><tr><td>30</td><td>99.0 (0.8)</td><td>98.6 (0.7)</td><td>99.0 (0.8)</td><td>98.4 (1.2)</td><td>98.0 (1.2)</td><td>97.5 (1.4)</td><td>97.6 (1.7)</td><td>95.9 (2.1)</td></tr><tr><td>20</td><td>99.0 (0.8)</td><td>98.8 (0.9)</td><td>99.0 (0.8)</td><td>98.4 (1.1)</td><td>98.1 (1.2)</td><td>97.5 (1.5)</td><td>97.3 (1.7)</td><td>96.1 (2.1)</td></tr><tr><td>10</td><td>98.9 (0.9)</td><td>99.1 (0.7)</td><td>98.8 (0.7)</td><td>98.5 (0.8)</td><td>98.7 (1.4)</td><td>97.4 (1.5)</td><td>97.2 (1.4)</td><td>95.7 (2.0)</td></tr><tr><td>5</td><td>99.1 (0.7)</td><td>98.9 (1.0)</td><td>98.9 (0.5)</td><td>98.5 (0.8)</td><td>98.2 (1.5)</td><td>97.1 (1.5)</td><td>97.0 (1.6)</td><td>94.9 (1.7)</td></tr><tr><td>1</td><td>96.7 (1.7)</td><td>96.5 (2.1)</td><td>96.8 (1.6)</td><td>96.6 (2.2)</td><td>96.7 (2.2)</td><td>95.4 (2.9)</td><td>93.7 (3.7)</td><td>90.9 (4.6)</td></tr><tr><td>1/6</td><td>96.4 (2.2)</td><td>96.6 (2.1)</td><td>96.6 (2.1)</td><td>97.3 (2.2)</td><td>96.8 (2.1)</td><td>96.3 (2.6)</td><td>96.2 (2.3)</td><td>93.1 (3.8)</td></tr><tr><td></td><td colspan="8" style="text-align: center;">CNN</td></tr><tr><td>30</td><td>98.4 (1.5)</td><td>98.0 (2.0)</td><td>98.1 (1.1)</td><td>96.0 (1.0)</td><td>94.1 (0.6)</td><td>86.9 (1.2)</td><td>82.0 (8.2)</td><td>83.5 (3.9)</td></tr><tr><td>20</td><td>97.5 (2.3)</td><td>98.0 (2.0)</td><td>97.4 (1.7)</td><td>95.7 (1.7)</td><td>93.2 (2.0)</td><td>83.8 (6.4)</td><td>83.1 (5.3)</td><td>79.4 (5.7)</td></tr><tr><td>10</td><td>94.4 (5.7)</td><td>96.6 (3.0)</td><td>96.7 (2.2)</td><td>93.7 (1.8)</td><td>92.1 (1.8)</td><td>81.6 (1.4)</td><td>80.4 (4.0)</td><td>78.4 (8.6)</td></tr><tr><td>5</td><td>94.8 (5.8)</td><td>95.9 (3.8)</td><td>96.1 (2.7)</td><td>93.6 (1.9)</td><td>88.9 (1.9)</td><td>82.5 (3.5)</td><td>79.5 (1.7)</td><td>76.0 (6.4)</td></tr><tr><td>1</td><td>85.6 (2.8)</td><td>88.4 (6.1)</td><td>85.1 (2.5)</td><td>90.9 (5.1)</td><td>81.9 (3.7)</td><td>75.2 (1.9)</td><td>73.2 (5.0)</td><td>61.9 (11.3)</td></tr><tr><td>1/6</td><td>76.2 (8.9)</td><td>62.6 (15.0)</td><td>66.2 (11.9)</td><td>61.4 (20.5)</td><td>60.9 (15.4)</td><td>54.5 (10.9)</td><td>51.7 (10.8)</td><td>47.8 (5.6)</td></tr></tbody></table>

### 3. DETECTORS

Design of the spectrogram correlation detector follows Mellinger and Clark [20]. For each test sample, a 2D correlation is performed with each *positive* training sample and the highest correlation value is taken as the recognition score. The *negative* training samples are unused. The decision threshold is then set as the median recognition score across the test samples. Thus, this implementation implicitly assumes that half of the test samples are *positive*. However, this can easily be modified to use a fixed threshold or a threshold-moving algorithm for imbalanced data [27].

A simple CNN is designed with three convolutional layers and one dense layer. The convolutional layers have 3 by 3 kernels, [2, 2] stride, 8, 16 and 32 filters, and are followed by batch normalisation [28] and RELU activations. The network has 7,170 trainable parameters in total. Spectrogram values are rescaled to the range [0, 1] before input. Training is performed using the Adam optimiser [29] with an initial learning rate of 0.003, a batch size of 50, and early stopping.

### 4. RESULTS

The average detection accuracies and standard deviations across the 4 validation folds are shown in Table 2. The results are also given graphically, also called interaction plots, in Figure 2.

As expected, the overall trend is that the detectors perform better when trained with stronger labels, and higher quantities of data. However, the results show that both methods can tolerate some label noise (corresponding to a label density of 60 to 70%) without a meaningful reduction in performance. This has an important corollary: increasing the strength of the labels does not improve performance beyond a certain point.

Since the analysis frames are 1 s long, “Strong +1.5 s” is the first dataset for which samples can be labelled *positive* but not

contain even a partial call, and this coincides with the accuracy beginning to drop. This suggests a possible interaction between the length of the analysis frames and the strength of the labels, especially for the CNN. However, further work is needed to establish to what extent, if at all, this is the case, and determine whether frame length should also be considered when choosing a label strength.

The spectrogram correlation responds similarly to increasing label quantity, with performance plateauing beyond the use of 5 training calls. Performance of the CNN, however, continues to increase up to the maximum training set size tested. Thus, the CNN is shown to scale better with the quantity of training data available. On the other hand, the spectrogram correlation is more robust to fewer training samples, with the accuracy dropping at most only a few percent between the highest and lowest values tested.

It is observed that using all 6 audio channels instead of only one actually makes the spectrogram correlation perform worse. This is because channels for the same call are highly correlated, so comparing additional channels provides insufficient new information to compensate for the reduced SNR, which only serves to confuse the detector. On the other hand, the varying SNR is shown to be an effective regulariser for the CNN, improving accuracy by an average of 20%. When only a single channel is available, it is likely that similar gains can be achieved by varying the SNR artificially (i.e., standard data augmentation).

Both methods perform well even when labels are extremely weak. In the weakest case, only 4% of the *positive* audio distinguishes it from the non-target class. The spectrogram correlation has far better robustness to label noise, with its accuracy dropping by at most only 6%. This is likely because samples of ambient noise correlate poorly with each other, as well as with samples of minke whale call. The CNN, on the other hand, does not hold this bias, making it susceptible to overfitting, and learning to erroneously discriminate the two classes based on spurious information in the background noise. With only 7,000 parameters, the extremely small capacity of the model may have helped to avoid this.
<figure><img src="image_82.png" /><figcaption>Figure 2: Interaction plots for label quantity and strength, for the spectrogram correlation (top) and CNN (bottom).</figcaption></figure>

The presence of factor interaction is indicated by the lines in an interaction plot being nonparallel. In general, the lines in each plot of Figure 2 can be seen to spread out from left to right, showing that some interaction between label strength and quantity is occurring. Specifically, the results indicate that stronger labels are more robust to smaller training sets, and larger training sets are more robust to weaker labels. The presence of interaction also demonstrates the importance of conducting multi-factor studies.

## 5. CONCLUSION

This paper studied the effects of volume and strength of training data on the performance of two detectors for minke whale calls. The aim was to determine the relevance to marine bioacoustics of the oft-quoted principle of “quantity over quality”. Given the CNN’s good scalability to larger training sets, and the performance saturation that occurs when enhancing label strength, the study concludes that “quantity over quality” does indeed hold for CNNs for the call detection scenario presented. Consistency in other machine learning domains suggests that this conclusion is likely to be valid for other gradient-based models as well. However, this is not the case for the spectrogram correlation, which is found to be

incapable of exploiting additional training data. On the other hand, the spectrogram correlation demonstrates greater robustness to noisy labels and smaller training sets, making it more appropriate for few-shot learning scenarios.

Future work includes investigating the effects of frame length, mixing labels of different strengths, and using larger training sets to find the saturation points of the CNN’s performance. Extensions can also be made to include more advanced weak learning methods, and classification of multiple marine mammal species. Finally, statistical significance tests such as two-way analysis of variance can be used to provide a more objective measure of factor interaction.

## 6. REFERENCES

[1] P. Nguyen, “Development of artificial intelligence methods for marine mammal detection and classification of underwater sounds in a weak supervision (but) Big Data-Expert context,” Doctoral Thesis, Sorbonne University, 2020.

[2] K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image Recognition,” Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., vol. 2016-December, pp. 770–778, Dec. 2015, doi: 10.48550/arxiv.1512.03385.
[3] H. Wei, L. Tao, R. Xie, and B. An, "Open-set Label Noise Can Improve Robustness Against Inherent Label Noise," *Adv. Neural Inf. Process. Syst.*, vol. 10, pp. 7978-7992, Jun. 2021, doi: 10.48550/arxiv.2106.10891.

[4] S. Hershey et al., "CNN Architectures for Large-Scale Audio Classification," *ICASSP, IEEE Int. Conf. Acoust. Speech Signal Process.* - *Proc.*, pp. 131-135, Sep. 2016, doi: 10.1109/ICASSP.2017.7952132.

[5] J. F. Gemmeke et al., "Audio Set: An ontology and human-labeled dataset for audio events," *ICASSP, IEEE Int. Conf. Acoust. Speech Signal Process.* - *Proc.*, pp. 776-780, Jun. 2017, doi: 10.1109/ICASSP.2017.7952261.

[6] K. J. Piczak, "ESC: Dataset for Environmental Sound Classification," *MM 2015 - Proc. 2015 ACM Multimed. Conf.*, pp. 1015-1018, Oct. 2015, doi: 10.1145/2733373.2806390.

[7] E. Fonseca, X. Favory, J. Pons, F. Font, and X. Serra, "FSD50K: An Open Dataset of Human-Labeled Sound Events," *IEEE/ACM Trans. Audio Speech Lang. Process.*, vol. 30, pp. 829-852, Oct. 2020, doi: 10.48550/arxiv.2010.00475.

[8] K. E. Frasier, "A machine learning pipeline for classification of cetacean echolocation clicks in large underwater acoustic datasets," *PLoS Comput. Biol.*, vol. 17, no. 12, pp. 1-26, 2021, doi: 10.1371/JOURNAL.PCBI.1009613.

[9] K. E. Frasier, M. A. Roch, M. S. Soldevilla, S. M. Wiggins, L. P. Garrison, and J. A. Hildebrand, "Automated classification of dolphin echolocation click types from the Gulf of Mexico," *PLOS Comput. Biol.*, vol. 13, no. 12, p. e1005823, Dec. 2017, doi: 10.1371/JOURNAL.PCBI.1005823.

[10] M. Ferrari, H. Glotin, R. Marxer, and M. Asch, "DOCC10: Open access dataset of marine mammal transient studies and end-to-end CNN classification," *Proc. Int. Jt. Conf. Neural Networks*, Jul. 2020, doi: 10.1109/IJCNN48605.2020.9207085.

[11] A. Shah, A. Kumar, A. G. Hauptmann, and B. R. Fellow, "A Closer Look at Weak Label Learning for Audio Events," *CoRR*, Apr. 2018, doi: 10.48550/arxiv.1804.09288.

[12] N. Turpault, R. Serizel, and E. Vincent, "Limitations of Weak Labels for Embedding and Tagging," *ICASSP, IEEE Int. Conf. Acoust. Speech Signal Process.* - *Proc.*, vol. 2020-May, pp. 131-135, May 2020, doi: 10.1109/ICASSP40776.2020.9053160.

[13] S. Hershey et al., "The Benefit Of Temporally-Strong Labels In Audio Event Classification," *ICASSP, IEEE Int. Conf. Acoust. Speech Signal Process.* - *Proc.*, vol. 2021-June, pp. 366-370, May 2021, doi: 10.48550/arxiv.2105.07031.

[14] N. Quadrianto, A. J. Smola, T. S. Caetano, and Q. V. Le, "Estimating Labels from Label Proportions," *J. Mach. Learn. Res.*, pp. 2349-2374, 2009, Accessed: Jun. 09, 2022. [Online]. Available: https://www.jmlr.org/papers/v10/quadrianto09a.html

[15] A. Kumar and B. Raj, "Audio Event Detection using Weakly Labeled Data," *MM 2016 - Proc. 2016 ACM Multimed. Conf.*, pp. 1038-1047, May 2016, doi: 10.1145/2964284.2964310.

[16] A. Kumar and B. Raj, "Weakly Supervised Scalable Audio Content Analysis," *Proc. - IEEE Int. Conf. Multimed. Expo*, vol. 2016-August, Jun. 2016, doi: 10.48550/arxiv.1606.03664.

[17] Y. Xu, Q. Kong, Q. Huang, W. Wang, and M. D. Plumbley, "Attention and Localization based on a Deep Convolutional Recurrent Model for Weakly Supervised Audio Tagging," *Proc. Annu. Conf. Int. Speech Commun. Assoc. INTERSPEECH*, vol. 2017-August, pp. 3083-3087, Mar. 2017, doi: 10.48550/arxiv.1703.06052.

[18] G. L. Turin, "An introduction to matched filters," *IRE Trans. Inf. Theory*, vol. 6, no. 3, pp. 311-329, 1960, doi: 10.1109/TIT.1960.1057571.

[19] J. R. Potter, D. K. Mellinger, and C. W. Clark, "Marine mammal call discrimination using artificial neural networks," *J. Acoust. Soc. Am.*, vol. 96, p. 2636, 1994, doi: 10.1121/1.410274.

[20] D. K. Mellinger and C. W. Clark, "Recognizing transient low-frequency whale sounds by spectrogram correlation," *J. Acoust. Soc. Am.*, vol. 107, p. 3518, 2000, doi: 10.1121/1.429434.

[21] T. Lu, B. Han, and F. Yu, "Detection and classification of marine mammal sounds using AlexNet with transfer learning," *Ecol. Inform.*, vol. 62, p. 101277, May 2021, doi: 10.1016/J.ECOINF.2021.101277.

[22] Y. Shiu et al., "Deep neural networks for automated detection of marine mammal species," *Sci. Reports 2020* 101, vol. 10, no. 1, pp. 1-12, Jan. 2020, doi: 10.1038/s41598-020-57549-y.

[23] NOAA Pacific Islands Fisheries Science Center, "Hawaiian Islands Cetacean and Ecosystem Assessment Survey (HICEAS) towed array data. Edited and annotated for DCLDE 2022," *NOAA National Centers for Environmental Information*. 2022. doi: https://doi.org/10.25921/e12p-gj65.

[24] D. Risch, T. Norris, M. Curnock, and A. Friedlaender, "Common and Antarctic Minke Whales: Conservation Status and Future Research Directions," *Front. Mar. Sci.*, vol. 0, p. 247, May 2019, doi: 10.3389/FMARS.2019.00247.

[25] J. N. Oswald, W. W. L. Au, and F. Duennebier, "Minke whale (Balaenoptera acutorostrata) boings detected at the Station ALOHA Cabled Observatory," *J. Acoust. Soc. Am.*, vol. 129, no. 5, p. 3353, May 2011, doi: 10.1121/1.3575555.

[26] G. Blanchard, M. Flaska, G. Handy, S. Pozzi, and C. Scott, "Classification with Asymmetric Label Noise: Consistency and Maximal Denoising," *Electron. J. Stat.*, vol. 10, no. 2, pp. 2780-2824, Mar. 2013, doi: 10.48550/arxiv.1303.1208.

[27] C. Esposito, G. A. Landrum, N. Schneider, N. Stiefl, and S. Riniker, "GHOST: Adjusting the Decision Threshold to Handle Imbalanced Data in Machine Learning," *J. Chem. Inf. Model.*, vol. 61, no. 6, pp. 2623-2640, Jun. 2021, doi: 10.1021/ACS.JCIM.1C00160/ASSET/IMAGES/LARGE/CI 1C00160_0015.JPEG.

[28] S. Ioffe and C. Szegedy, "Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift," *32nd Int. Conf. Mach. Learn. ICML 2015*, vol. 1, pp. 448-456, Feb. 2015, doi: 10.48550/arxiv.1502.03167.

[29] D. P. Kingma and J. L. Ba, "Adam: A Method for Stochastic Optimization," *3rd Int. Conf. Learn. Represent. ICLR 2015 - Conf. Track Proc.*, Dec. 2014, doi: 10.48550/arxiv.1412.6980.
# DG-MIX: DOMAIN GENERALIZATION FOR ANOMALOUS SOUND DETECTION BASED ON SELF-SUPERVISED LEARNING

Ismail Nejjar¹,², Jean Meunier-Pion¹,³, Gaetan Frusque¹, Olga Fink¹,

¹ EPFL, IMOS, Lausanne, Switerzland, {ismail.nejjar, gaetan.frusque, olga.fink}@epfl.ch

² ETH Zürich, Chair of Intelligent Maintenance Systems, Zürich, Switerzland, {inejjar}@ethz.ch

³ CentraleSupélec, Gif-sur-Yvette, France, {jean.meunier-pion}@student-cs.fr

## ABSTRACT

Detecting anomalies in sound data has recently received significant attention due to the increasing number of implementations of sound condition monitoring solutions for critical assets. In this context, changing operating conditions impose significant domain shifts resulting in performance drops if a model trained on a set of operating conditions is applied to a new operating condition. An essential challenge is distinguishing between anomalies due to faults and new operating conditions. Therefore, the high variability of operating conditions or even the emergence of new operating conditions requires algorithms that can be applied under all conditions. Therefore, domain generalization approaches need to be developed to tackle this challenge. In this paper, we propose a novel framework that leads to a representation that separates the health state from changes in operating conditions in the latent space. This research introduces DG-Mix (Domain Generalization Mixup), an algorithm inspired by the recent Variance-Invariance-Covariance Regularization (VICReg) framework. Extending the original VICReg algorithm, we propose to use Mixup between two samples of the same machine type as a transformation and apply a geometric constraint instead of an invariance loss. This approach allows us to learn a representation that distinguishes between the operating conditions in an unsupervised way. The proposed DG-Mix enables the generalization between different machine types and diverse operating conditions without an additional adaptation of the hyperparameters or an ensemble method. DG-Mix provides superior performance and outperforms the baselines on the development dataset of DCASE 2022 challenge task 2. We also demonstrate that training using DG-Mix and then fine-tuning the model to a specific task significantly improves the model's performance.

**Index Terms**— Unsupervised Learning, Mixup, Domain Generalization, Self-Supervised Learning, Anomalous Sound Detection

## 1. INTRODUCTION

Anomalous sound detection (ASD) is the task of identifying whether the sound emitted by a machine is normal or abnormal [1, 2, 3]. One of the main challenges of this task is to distinguish between novel operating conditions or novel background noise and real anomalies caused by a machine fault or malfunction. Moreover, the sound emitted by machines of the same type but operated differently or installed under different conditions may differ significantly. Deep learning models have recently demonstrated excellent performance in detecting abnormal sounds under different scenarios. The directions pursued to tackle these challenges range from Unsupervised Anomalous sound detection [4, 5, 6], where

only normal sound samples are used for training, to domain adaptation techniques for bridging domain shifts [7]. Domain shifts are discrepancies in the acoustic signals between a source and a target domain mainly caused by differences in machine operating conditions or ambient noise. The shifts result in performance drops if a model trained on a set of operating conditions is applied to a new operating condition.

In real-world applications, the background noise of the machine can be affected by various sound sources surrounding the machine. Therefore, it is difficult to identify the distinct causes of the changes and attribute them to the domain shift. Consequently, it is necessary to develop a method that can be generalized to different changes in operating conditions without relying on the detection of domain shifts.

In this paper, we propose a novel algorithm DG-Mix (Domain Generalization Mixup), which aims to learn representations that distinguish between the different operating and recording conditions in an unsupervised manner. Three objectives are thereby pursued: (1) reveal the impact of attributes on the data by enforcing embeddings in the same batch to be different, (2) obtain uncorrelated embedding features containing specific information, (3) respect defined geometrical constraints between the different domains. We also investigate how self-supervised learning pre-training helps our model to learn more robust and more general representations that generalize across various operating conditions for different machine types, and are robust to different noise levels and noise types. To this end, our proposed algorithm is compared on the one hand to another popular self-supervised approach, VICReg, and on the other hand to our proposed model trained from scratch. We evaluated the proposed approach and submitted it to task 2, "Unsupervised anomalous sound detection (ASD) for machine condition monitoring applying domain generalization techniques" of the DCASE challenge 2022 [8]. In experimental evaluations, it is shown that the proposed technique significantly outperforms both baseline approaches but also the originally proposed VICReg on the source and target domains of the development set.

## 2. DATASET

The dataset of this task was generated from the MIMII DG [9] (Malfunctioning Industrial Machine Investigation and Inspection for Domain Generalization) and ToyADMOS2 [10] (Anomaly Detection in Machine Operating Sounds) datasets consisting of normal and anomalous operating sounds of seven types of toy/real machines. ToyCar and ToyTrain machine types are extracted from ToyADMOS2 dataset while fan, gearbox, bearing, slide rail, and valve are extracted from MIMII DG dataset. Each recording is a single-
channel 10-second audio signal sampled at 16 kHz. The signals originate from a mixture between machine sounds and environmental noise samples of several real-world factories. Each machine type contains six sections. The dataset is divided into a development set of three sections and an evaluation set consisting of the other three sections. For each section, there are 1000 training samples, including 990 source samples and ten target samples. In this research, we use all the training data in the development dataset and the additional training dataset for training the models.

### 3. METHOD

We propose a new framework inspired by the Variance-Invariance-Covariance Regularization (VICReg), a self-supervised algorithm proposed in [11]. We give an overview of our proposed approach in Figure 1. The framework is composed of (1) a self-supervised learning algorithm, (2) a subsequent fine-tuning step and (3) an anomaly detection phase based on k-Nearest Neighbors (k-NN) [12].

The objective of the self-supervised task is to learn an encoder that provides meaningful representations of audio samples. This pre-trained encoder is then used and fine-tuned to perform supervised classification of the section ID. Finally, we use the embeddings from the encoder to compute an anomaly score with k-NN.

#### 3.1. Audio Preprocessing

To simplify the task, all audio samples are provided with only one channel. Each audio sample is then transformed into a log-mel spectrogram. The input given to our model is a two-dimensional image-like feature $X \in \mathbb{R}^{P \times F}$. The frame size of the Short-Time Fourier Transform (STFT) is 64 ms, and the hop size is 32 ms. We also set the number of Mel bins $F$ to 128. The number of frames of the context window $P$ is fixed to 64. The context window is shifted by $L$ frames resulting in $B$ extracted images, with $B = \lceil \frac{T-P}{L} \rceil$ with $L = 8$. Given the previous parameters, the total spectrogram size $T$ is equal to 313.

#### 3.2. CNN Architectures

We used the MobileNetV2 [13] backbone trained from scratch in this work. The off-the-shelf Pytorch [14] implementation of MobileNetV2 is used. The width multiplier parameter is set to 0.5, and the last layers are adapted to obtain a 320-dimensional vector per input image. Table 1 provides a detailed summary of the applied CNN architecture, with a total parameter number of 1.45M.

#### 3.3. DG-Mix : Self-Supervised Pre-training

The first step of our approach is based on a self-supervised learning algorithm inspired by VICReg [11]. This framework provides a good feature representation for image classification problems.

**Background:** VICReg is an algorithm based on Siamese networks. VICReg aims to prevent a collapse by regularising the variance and covariance of the network outputs. The objective function of VICReg contains three main terms: a variance term, an invariance term, and a covariance term.

1. **Variance:** Regularization term that prevents mode collapse

2. **Covariance:** Regularization term that prevents dimensional collapse.

3. **Invariance:** Similarity metric to be minimized between two augmented views of the same source image.

<table><caption>Table 1: Modified MobileNetV2 architecture used for all experiments. Each row represents the sequence of layers, repeated n times, with c channels, and stride s</caption><thead><tr><th>Input</th><th>Operator</th><th>t</th><th>c</th><th>n</th><th>s</th></tr></thead><tbody><tr><td>128 × 64 × 3</td><td>conv2d 3x3</td><td>-</td><td>16</td><td>1</td><td>2</td></tr><tr><td>64 × 32 × 16</td><td>bottleneck</td><td>1</td><td>8</td><td>1</td><td>1</td></tr><tr><td>64 × 32 × 8</td><td>bottleneck</td><td>6</td><td>16</td><td>2</td><td>2</td></tr><tr><td>32 × 16 × 16</td><td>bottleneck</td><td>6</td><td>16</td><td>3</td><td>2</td></tr><tr><td>16 × 8 × 16</td><td>bottleneck</td><td>6</td><td>32</td><td>4</td><td>2</td></tr><tr><td>8 × 4 × 32</td><td>bottleneck</td><td>6</td><td>48</td><td>3</td><td>1</td></tr><tr><td>8 × 4 × 48</td><td>bottleneck</td><td>6</td><td>80</td><td>3</td><td>2</td></tr><tr><td>4 × 2 × 80</td><td>bottleneck</td><td>6</td><td>160</td><td>1</td><td>1</td></tr><tr><td>4 × 2 × 160</td><td>conv2d 1x1</td><td>-</td><td>320</td><td>1</td><td>1</td></tr><tr><td>4 × 2 × 320</td><td>conv2d 4x2</td><td>-</td><td>320</td><td>1</td><td>1</td></tr><tr><td>1 × 1 × 320</td><td>conv2d 1x1</td><td>-</td><td>320</td><td>1</td><td>1</td></tr></tbody></table>

**Proposed Approach:** Figure 1 provides an overview of DG-MIX. The proposed loss comprises three parts. The last two terms correspond to the variance and covariance losses presented in the VICReg [11] implementation, while we propose to substitute the third part (representing the invariance criterion in VICReg) with a term that enforces the embedding features of the source and target domains to be distinguishable. Thereby, we are able to learn a specific representation for both domains. For each machine type, we train a Siamese architecture where the three branches are similar and share the same weights. Each branch is composed of an encoder $f_\theta$ which corresponds to the modified MobileNetV2 presented in Table 1, followed by an expander $h_\phi$. The expander is composed of three fully-connected layers of size 1280. Each of the layers is followed by a batch normalization layer [15] and a ReLU [16] activation function.

In order to mitigate the gap between the source and target domains, we propose to extend the VICReg framework [11] by using Mixup [17] to augment the target domain. Furthermore, a novel loss is proposed to take into account the added Mixup branch, acting as a regularization term and improving domain generalization.

While in the original VICReg approach, a data augmentation approach is applied, we propose to impose similarity between the sample representations. Given all the $S$ log Mel-spectrograms from both the source and target domains of all sections for each machine type, two different samples, $X$ and $X'$ are selected. For each such pair of samples, a linear combination with respect to $\lambda$ is obtained. This combination gives rise to a new sample denoted as $X_\lambda$. Formally, $\lambda$ is a realization of a beta distribution $\text{Beta}(\alpha, \beta)$ and represents the mixup rate. In our case we set $\alpha = \beta = 0.5$.

First $X$, $X'$ and $X_\lambda$ are encoded by $f_\theta$ resulting in $Y$, $Y'$ and $Y_\lambda$, and then mapped by the expander on the embeddings, $Z$, $Z'$ and $Z_\lambda$. The loss is composed of three terms and computed at the embedding level on $Z$, $Z'$ and $Z_\lambda$. For a batch of size $N$, we denote $Z = [z_1, ..., z_N]$, with $z \in \mathbb{R}^D$ and $D$ the expander dimension.

The proposed consistency term seeks to generate new virtual domains. A linear interpolations of feature vectors should lead to linear interpolations of their corresponding domain. Therefore the proposed loss aims at minimizing the distance between the embedding vector of $X_\lambda$ and the linear combination of embeddings of $X$ and $X'$:

$$s(Z_{\lambda}, Z, Z', \lambda) = \frac{1}{N} \sum_{i=1}^{N} \|z_{\lambda, i} - (\lambda z_{i} + (1 - \lambda) z'_{i})\|_{2}^{2} \quad (1)$$
<figure><img src="image_83.png" /><figcaption>Figure 1: DG-Mix : Self-Supervised Framework for Domain Generalization</figcaption></figure>

The second loss term forces the variance inside each batch to be equal to 1, preventing a mode collapse:

$$v(Z) = \frac{1}{D} \sum_{i=1}^{D} \max(0, 1 - S(z^j)) \quad (2)$$

where $z^j$ is the jth row of the matrix Z in the batch, and $S(z) = \sqrt{Var(z)}$ is the standard deviation.

Finally, the last loss term aims to learn uncorrelated features for each embedding, by forcing the off-diagonal elements to be zero, resulting in a rich embedding. The covariance matrix is defined as:

$$C(Z) = \frac{1}{N-1} \sum_{i=1}^{N} (z_i - \bar{z})(z_i - \bar{z})^T \quad (3)$$

with $\bar{z} = \frac{1}{N} \sum_{i=1}^{N} z_i$ representing the mean embedding over a mini-batch.

$$c(z) = \frac{1}{d} \sum_{i \neq j} [C(Z)]_{i,j}^2 \quad (4)$$

The final loss used to train the model is:

$$\gamma s(Z_{\lambda}, Z, Z', \lambda) + \mu(v(Z) + v(Z')) + \nu(c(Z) + c(Z')) \quad (5)$$

$\gamma, \mu, \nu$ are hyper-parameters that we set in this report to 25, 25, 1 respectively. The networks are trained using the Large Batch Training of Convolutional Networks (LARS) [18] optimizer, with a learning rate of 0.8, weight decay of $10^{-4}$ and a batch size of 1024 for 100 epochs. In addition, ten warmup epochs were used, and the learning rate followed a cosine decay schedule starting from 0 and finishing at 0.002. The expander size and the hyperparameters were not finetuned on the task but rather taken as reported in the VICReg paper. After the pre-training, only the encoder model was used for the downstream classification task presented in the next section.

## 3.4. Fine-tuning on Section ID Classification

Similar to the baseline method, our proposed model is fine-tuned to identify the section ID of an audio sample.

The pre-trained encoder is used, and a classifier composed of two fully connected layers (320-128-6) is added. To improve the robustness of the model, a mixup strategy on the source and target data for each section is used to generate augmented data of intra-domain and inter-domain samples.

The KL-divergence loss between the classifier output and the mixed section ID is used for this task along with the geometrical constraint presented in equation 1 as a regularization term. However, this time it is directly applied to the encoder. Finally, the networks are fine-tuned using AdamW [19] optimizer, with a learning rate of $10^{-4}$, weight decay of $10^{-4}$, and a batch size of 64.

## 3.5. Anomaly Detection

After the fine-tuning step, we apply a k-NN algorithm [12] to compute the anomaly score. We use the mean embedding vector from the 10-s audio recording as input feature to the k-NN algorithm.

We used the Euclidean metric as the anomaly score, and the number of nearest neighbors was set to 1. In other words, the larger the distance from the training embeddings, the more abnormal the sample is.

# 4. RESULTS

We tested our model on the development set of the DCASE 2022 Task 2 described in section 2. We first present how our proposed method without pre-training already outperforms the baseline. The main differences between the baseline and our proposed method are (1) change of the CNN, (2) use of Mixup for data augmentation, (3) addition of a regularization term from the Mixup transformation, and (4) use of k-NN for anomaly score computation. Table 2 displays the harmonic means of the AUC Source, AUC Target, and pAUC computed over all three sections for each machine type using the baselines and the proposed method without pre-training. The harmonic means of the AUC Source, AUC Target, and pAUC are also reported for each method. An absolute improvement of more than 10% of the baseline approach is obtained.

The importance of pre-training becomes apparent from the results reported in Table 3 where the performance of two self-supervised approaches, VICReg using SpecAugment [20] and the proposed DG-Mix method is compared. Moreover, we use the SpecAugment procedure for VICReg because Mixup is not adapted for this method. VICReg provides an improvement of almost 1% in terms of the overall harmonic mean and DG-Mix has a performance gain of 4%, suggesting that they both successfully helped to get better and more robust representations.

Pre-training with DG-Mix outperforms by 3% the pre-training obtained using VICReg. This is due to Mixup and its regularization term that provide a richer representation of the dataset. This is illustrated in Figures 2a and 2b presenting the t-SNE [21] plots of the embeddings obtained from a ToyCar training sample with DG-Mix and VICReg. There are more clusters with DG-Mix, showing more granularity and a better separation of the dataset into different modalities.
<table><caption>Table 2: Results for the MobileNet-V2 baseline and our proposed anomaly detection method without pre-training (in %).</caption><thead><tr><th rowspan="2">Machine</th><th colspan="4">Baseline MobileNet-V2</th><th colspan="4">Proposed anomaly detection method without pre-training</th></tr><tr><th>AUC Source</th><th>AUC Target</th><th>pAUC</th><th>Harmonic Mean</th><th>AUC Source</th><th>AUC Target</th><th>pAUC</th><th>Harmonic Mean</th></tr></thead><tbody><tr><td>ToyCar</td><td>58,97</td><td>52,26</td><td>52,39</td><td>54,37</td><td><strong>79,82</strong></td><td><strong>73,62</strong></td><td><strong>60,11</strong></td><td><strong>70,18</strong></td></tr><tr><td>ToyTrain</td><td><strong>58,59</strong></td><td>46,07</td><td><strong>51,56</strong></td><td>51,57</td><td>46,45</td><td><strong>61,25</strong></td><td>51,43</td><td><strong>52,36</strong></td></tr><tr><td>Bearing</td><td><strong>62,88</strong></td><td>61,81</td><td><strong>57,35</strong></td><td><strong>60,58</strong></td><td>59,24</td><td><strong>69,06</strong></td><td>50,13</td><td>58,47</td></tr><tr><td>Fan</td><td>71,35</td><td>48,53</td><td>57,10</td><td>57,54</td><td><strong>88,85</strong></td><td><strong>70,27</strong></td><td><strong>69,45</strong></td><td><strong>75,22</strong></td></tr><tr><td>Gearbox</td><td>69,98</td><td>56,60</td><td>56,18</td><td>60,29</td><td><strong>79,83</strong></td><td><strong>70,32</strong></td><td><strong>60,78</strong></td><td><strong>69,44</strong></td></tr><tr><td>Slider</td><td>66,03</td><td>40,72</td><td>54,77</td><td>51,76</td><td><strong>90,70</strong></td><td><strong>66,02</strong></td><td><strong>64,89</strong></td><td><strong>72,15</strong></td></tr><tr><td>Valve</td><td>67,75</td><td>58,01</td><td><strong>62,70</strong></td><td>62,57</td><td><strong>71,74</strong></td><td><strong>65,99</strong></td><td>59,05</td><td><strong>65,18</strong></td></tr><tr><td><strong>Overall</strong></td><td>64,73</td><td>51,06</td><td>55,80</td><td>56,65</td><td><strong>72,71</strong></td><td><strong>70,93</strong></td><td><strong>62,27</strong></td><td><strong>68,32</strong></td></tr></tbody></table>

<table><caption>Table 3: Results obtained when pre-training with VICReg compared to DG-Mix (in %).</caption><thead><tr><th rowspan="2">Machine</th><th colspan="4">VICReg with SpecAugment</th><th colspan="4">DG-Mix</th></tr><tr><th>AUC Source</th><th>AUC Target</th><th>pAUC</th><th>Harmonic Mean</th><th>AUC Source</th><th>AUC Target</th><th>pAUC</th><th>Harmonic Mean</th></tr></thead><tbody><tr><td>ToyCar</td><td>85,09</td><td>78,68</td><td>59,05</td><td>72,47</td><td><strong>93,28</strong></td><td><strong>81,73</strong></td><td><strong>68,32</strong></td><td><strong>79,80</strong></td></tr><tr><td>ToyTrain</td><td>48,34</td><td><strong>56,20</strong></td><td><strong>53,69</strong></td><td>52,54</td><td><strong>52,50</strong></td><td>55,48</td><td>53,20</td><td><strong>53,70</strong></td></tr><tr><td>Bearing</td><td><strong>77,99</strong></td><td>70,49</td><td><strong>73,22</strong></td><td><strong>73,77</strong></td><td>66,99</td><td><strong>82,90</strong></td><td>64,75</td><td>70,70</td></tr><tr><td>Fan</td><td><strong>83,83</strong></td><td><strong>77,17</strong></td><td>65,93</td><td>74,9</td><td>83,37</td><td><strong>76,73</strong></td><td><strong>70,74</strong></td><td><strong>76,60</strong></td></tr><tr><td>Gearbox</td><td><strong>94,99</strong></td><td>68,77</td><td>68,30</td><td>75,55</td><td>88,96</td><td><strong>82,74</strong></td><td><strong>71,22</strong></td><td><strong>80,28</strong></td></tr><tr><td>Slider</td><td><strong>94,99</strong></td><td><strong>68,74</strong></td><td>68,25</td><td><strong>75,51</strong></td><td>94,52</td><td>67,35</td><td><strong>68,91</strong></td><td>75,11</td></tr><tr><td>Valve</td><td>75,02</td><td>66,48</td><td>60,60</td><td>66,86</td><td><strong>85,09</strong></td><td><strong>81,93</strong></td><td><strong>71,65</strong></td><td><strong>79,13</strong></td></tr><tr><td><strong>Overall</strong></td><td>76,44</td><td>68,78</td><td>63,53</td><td>69,18</td><td><strong>77,55</strong></td><td><strong>74,08</strong></td><td><strong>66,33</strong></td><td><strong>72,34</strong></td></tr></tbody></table>

<figure><img src="image_84.png" /><figcaption>(a) t-SNE plot of the embedding result when Pre-training with DG-Mix</figcaption></figure>

<figure><img src="image_85.png" /><figcaption>(b) t-SNE plot of the embedding result when Pre-training with VICReg</figcaption></figure>

However, SpecAugment is less suited than Mixup for anomaly detection because it uses time warping, frequency masking, and time masking transformations that can interfere with the anomaly patterns. In contrast, Mixup does not assume anything about possible anomalies but only mixes samples. Using Mixup between and across multiple domains [22] allows us to sample the augmented training data from the heterogeneous Mixup distribution and get a more robust feature extractor at the end, which improves the results.

log-Mel spectrograms. In this work, our goal was to develop a unified framework that is robust and performs well across all machine types. We used the same hyperparameters for each machine type to achieve this goal. Experimental evaluation shows that the proposed approach significantly outperforms all baseline approaches. In addition, we demonstrated that pre-training an encoder improved the generalization ability of this encoder. Extending the framework to other tasks and datasets is left for future research.

## 5. CONCLUSION

In this paper, we proposed a novel sound anomaly detection framework for domain generalization composed of a self-supervised algorithm followed by a supervised task using Mixup on the input

## 6. ACKNOWLEDGMENTS

This work was supported by the Swiss National Science Foundation under Grant PP00P2_176878
7. REFERENCES

[1] Y. Koizumi, S. Saito, H. Uematsu, Y. Kawachi, and N. Harada, "Unsupervised detection of anomalous sound based on deep learning and the neyman-pearson lemma," *IEEE/ACM Transactions on Audio, Speech, and Language Processing*, vol. 27, no. 1, pp. 212–224, 2018.

[2] Y. Kawaguchi, R. Tanabe, T. Endo, K. Ichige, and K. Hamada, "Anomaly detection based on an ensemble of dereverberation and anomalous sound extraction," in *ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. IEEE, 2019, pp. 865–869.

[3] K. Suefusa, T. Nishida, H. Purohit, R. Tanabe, T. Endo, and Y. Kawaguchi, "Anomalous sound detection based on interpolation deep neural network," in *ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. IEEE, 2020, pp. 271–275.

[4] Y. Koizumi, Y. Kawaguchi, K. Imoto, T. Nakamura, Y. Nikaido, R. Tanabe, H. Purohit, K. Suefusa, T. Endo, M. Yasuda, et al., "Description and discussion on dcase2020 challenge task2: Unsupervised anomalous sound detection for machine condition monitoring," *arXiv preprint arXiv:2006.05822*, 2020.

[5] R. Giri, F. Cheng, K. Helwani, S. V. Tenneti, U. Isik, and A. Krishnaswamy, "Group masked autoencoder based density estimator for audio anomaly detection," in *Detection and Classification of Acoustic Scenes and Events Workshop 2020*, 2020.

[6] K. Wilkinghoff, "Sub-cluster adacos: Learning representations for anomalous sound detection," in *2021 International Joint Conference on Neural Networks (IJCNN)*. IEEE, 2021, pp. 1–8.

[7] Y. Kawaguchi, K. Imoto, Y. Koizumi, N. Harada, D. Niizumi, K. Dohi, R. Tanabe, H. Purohit, and T. Endo, "Description and discussion on dcase 2021 challenge task 2: Unsupervised anomalous sound detection for machine condition monitoring under domain shifted conditions," *arXiv preprint arXiv:2106.04492*, 2021.

[8] K. Dohi, K. Imoto, N. Harada, D. Niizumi, Y. Koizumi, T. Nishida, H. Purohit, T. Endo, M. Yamamoto, and Y. Kawaguchi, "Description and discussion on DCASE 2022 challenge task 2: Unsupervised anomalous sound detection for machine condition monitoring applying domain generalization techniques," *In arXiv e-prints: 2206.05876*, 2022.

[9] K. Dohi, T. Nishida, H. Purohit, R. Tanabe, T. Endo, M. Yamamoto, Y. Nikaido, and Y. Kawaguchi, "MIMII DG: Sound dataset for malfunctioning industrial machine investigation and inspection for domain generalization task," *In arXiv e-prints: 2205.13879*, 2022.

[10] N. Harada, D. Niizumi, D. Takeuchi, Y. Ohishi, M. Yasuda, and S. Saito, "ToyADMOS2: Another dataset of miniature-machine operating sounds for anomalous sound detection under domain shift conditions," in *Proceedings of the 6th Detection and Classification of Acoustic Scenes and Events 2021 Workshop (DCASE2021)*, Barcelona, Spain, November 2021, pp. 1–5.

[11] A. Bardes, J. Ponce, and Y. LeCun, "Vicreg: Variance-invariance-covariance regularization for self-supervised learning," 2022. [Online]. Available: https://hal.inria.fr/hal-03541297/file/vicreg_iclr_2022.pdf

[12] E. Fix and J. L. Hodges, "Discriminatory analysis. nonparametric discrimination: Consistency properties," *International Statistical Review/Revue Internationale de Statistique*, vol. 57, no. 3, pp. 238–247, 1989.

[13] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, "Mobilenetv2: Inverted residuals and linear bottlenecks," 2018. [Online]. Available: https://arxiv.org/abs/1801.04381

[14] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer, "Automatic differentiation in pytorch," in *NIPS-W*, 2017.

[15] S. Ioffe and C. Szegedy, "Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift," *CoRR*, vol. abs/1502.03167, 2015. [Online]. Available: http://arxiv.org/abs/1502.03167

[16] A. F. Agarap, "Deep Learning using Rectified Linear Units (ReLU)," *arXiv preprint arXiv:1803.08375*, 2018.

[17] H. Zhang, M. Cissé, Y. N. Dauphin, and D. Lopez-Paz, "mixup: Beyond Empirical Risk Minimization," *CoRR*, vol. abs/1710.09412, 2017. [Online]. Available: http://arxiv.org/abs/1710.09412

[18] Y. You, I. Gitman, and B. Ginsburg, "Scaling SGD Batch Size to 32k for ImageNet Training," *CoRR*, vol. abs/1708.03888, 2017. [Online]. Available: http://arxiv.org/abs/1708.03888

[19] I. Loshchilov and F. Hutter, "Decoupled weight decay regularization," 2017. [Online]. Available: https://arxiv.org/abs/1711.05101

[20] D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le, "Specaugment: A simple data augmentation method for automatic speech recognition," *arXiv preprint arXiv:1904.08779*, 2019.

[21] L. Van der Maaten and G. Hinton, "Visualizing data using t-sne." *Journal of machine learning research*, vol. 9, no. 11, 2008.

[22] Y. Wang, H. Li, and A. C. Kot, "Heterogeneous domain generalization via domain mixup," in *ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. IEEE, 2020, pp. 3622–3626.
FEW-SHOT BIOACOUSTIC EVENT DETECTION
AT THE DCASE 2022 CHALLENGE

I. Nolasco¹, S. Singh¹, E. Vidaña-Vila², E. Grout³,⁴, J. Morford⁵, M.G. Emmerson⁶ F. H. Jensen⁷, I. Kiskin⁸
H. Whitehead⁹, A. Strandburg-Peshkin³,⁴, L. Gill¹⁰, H. Pamuła¹¹, V. Lostanlen¹², V. Morfi¹,¹³, D. Stowell¹⁴

¹ Centre for Digital Music (C4DM), Queen Mary University of London, London, UK

² La Salle, University Ramon Llull, Barcelona, ES

³ Dept. of Biology & Centre for the Advanced Study of Collective Behaviour, University of Konstanz, DE

⁴ Dept. for the Ecology of Animal Societies, Max Planck Institute of Animal Behavior, DE

⁵ The Oxford Navigation group, Dept. of Zoology, Oxford University, Oxford, UK

⁶ School of Biological and Behavioural Sciences, Queen Mary University of London, London, UK

⁷ Biology Dept, Syracuse University, NY, USA

⁸ Institute for People-Centred AI, FHMS, University of Surrey, Surrey, UK

⁹ School of Science, Engineering and Environment, University of Salford, Manchester, UK

¹⁰ BIOTOPIA Naturkundemuseum Bayern, Munich, DE

¹¹ AGH University of Science and Technology, Kraków, PL

¹² Nantes Université, École Centrale Nantes, CNRS, LS2N, UMR 6004, F-44000 Nantes, FR

¹³ Sonantic Limited, London, UK

¹⁴ Tilburg University, Tilburg, The Netherlands; Naturalis Biodiversity Centre, Leiden, NL

ABSTRACT

Few-shot sound event detection is the task of detecting sound events, despite having only a few labelled examples of the class of interest. This framework is particularly useful in bioacoustics, where often there is a need to annotate very long recordings but the expert annotator time is limited. This paper presents an overview of the second edition of the few-shot bioacoustic sound event detection task included in the DCASE 2022 challenge. A detailed description of the task objectives, dataset, and baselines is presented, together with the main results obtained and characteristics of the submitted systems. This task received submissions from 15 different teams from which 13 scored higher than the baselines. The highest F-score was of 60.2% on the evaluation set, which leads to a huge improvement over last year's edition. Highly-performing methods made use of prototypical networks, transductive learning, and addressed the variable length of events from all target classes. Furthermore, by analysing results on each of the subsets we can identify the main difficulties that the systems face, and conclude that few-show bioacoustic sound event detection remains an open challenge.

***Index Terms***— Few-shot learning, bioacoustics, sound event detection, DCASE challenge

# 1. INTRODUCTION

The task of bioacoustic sound event detection refers to the retrieval of animal vocalisations from audio recordings in terms of onset and offset times. It shares a common methodology with other sound event detection (SED) contexts, yet, the application domain of bioacoustics is particularly challenging for SED. Deep learning contributed to overcome some of these difficulties in bioacoustic SED, however it also established strong requirements regarding the amount of annotated data needed [1]. Collecting and annotating a

large dataset of animal vocalisations is often not feasible given that species are unequally abundant [2] and may be rarely observed; and audio annotation is costly and time-consuming [3]. In contrast to traditional deep learning approaches that use a large amount of data to train models, few-shot learning tries to build accurate models with very few training data [4]. Few-shot learning is usually studied using *N*-way-*k*-shot classification, where *N* denotes the number of classes and *k* the number of known examples for each class.

This problem was first evaluated as a task on the DCASE 2021 challenge [5]. This year, the setup and goal remain the same: Given the first 5 events of a target class, can systems detect the subsequent events of the same class in the remaining of the audio recording? Diverse approaches have been used to address the few-shot learning problem for classification. Some use prior knowledge about similarity between sounds by computing embeddings (learnt representation spaces) designed to help discriminate between unseen classes [4], while others exploit prior knowledge about the structure of the data by using augmentation to synthesise new data [6]. Finally, some approaches can learn models with parameters that can be fine-tuned to smaller datasets [7]. More recent works use meta-learning and/or prototypical networks for acoustic few-shot learning [8], [9]. All of the above approaches deal with classification tasks rather than detection. Indeed, SED in a few shot setup is commonly approximated as an audio tagging task and few works have addressed the actual detection of onsets and offsets of events [10]. At last year's task edition, the best ranked system improved over the baseline prototypical approach by applying a transductive inference method and a mutual learning framework designed to make the feature extraction network more task dependent [11]. The overall best results were just below 40% F-score which indicates the difficulty of this task. This year, we added more and diverse datasets, and
increased the task difficulty (dataset diversity); yet the task doubled the amount of participants and the best overall F-score in the evaluation set reached the 60% level. This paper is structured as follows. Section 2 presents the bioacoustic datasets used for developing and evaluating submitted systems. Section 3 presents the two baseline methods proposed for the task, followed by the evaluation procedure. Finally, section 4 presents the results of the submitted systems and a discussion about the overall task and future steps in the field of few-shot bioacoustic event detection.

## 2. DATASETS

A development dataset consists of predefined training and validation sets to be used for system development<fnref n="1" />. The training set contains multi-class temporal annotations, provided for each recording as: positive (POS), negative (NEG) and unknown (UNK). For the validation set only single-class temporal annotations (POS/UNK) were provided for each recording. A separate evaluation set was kept for evaluating the performance of the systems<fnref n="2" />. During the challenge, only the first five POS events of the class of interest were provided for each of the recordings. Table 1 presents an overview of all the datasets in the development and evaluation sets.

**BirdVox-DCASE-10h (BV):** This dataset contains audio files recorded in 2015, during the fall migration season by four different autonomous recording units located in Tompkins County, NY, USA. An expert ornithologist, Andrew Farnsworth, has annotated flight calls from four families of passerines, namely: American sparrows, cardinals, thrushes, and New World warblers. These flight calls have a duration in the range 50–150 ms and a fundamental frequency in the range 2–10 kHz.

**Hyenas (HT):** Hyenas use a variety of types of vocalisations to coordinate with one another over both short and long distances. Spotted hyenas were recorded on custom-developed audio tags designed by Mark Johnson and integrated into combined GPS/acoustic collars (Followit Sweden AB) by Frants Jensen and Mark Johnson. Collars were deployed on female hyenas of the Talek West hyena clan at the MSU-Mara Hyena Project (directed by Kay Holekamp) in the Masai Mara, Kenya as part of a multi-species study on communication and collective behaviour. The recordings contain 5 different vocalisations which were identified and classified into types based on the established hyena vocal repertoire [12]. Field work was carried out by Kay Holekamp, Andrew Gersick, Frants Jensen, Ariana Strandburg-Peshkin, and Benson Pion; labelling was done by Kenna Lehmann and colleagues.

**Meerkats (MT, ME):** Meerkats are a highly social mongoose species that live in stable social groups and use a variety of distinct vocalisations to communicate and coordinate with one another. Recordings of meerkats were acquired at the Kalahari Meerkat Project (Kuruman River Reserve, South Africa; directed by Marta Manser and Tim Clutton-Brock), as part of a multi-species study on communication and collective behaviour. Recordings of the training set (MT) were recorded on small audio devices (TS Market, Edic Mini Tiny+ A77, 8 kHz) integrated into combined GPS/audio collars which were deployed on multiple members of meerkat groups. Recordings of the validation set (ME) were recorded by an observer following a focal meerkat with a Sennheiser ME66 directional microphone (44.1 kHz) from a distance of less than 1m. Recordings were carried out during daytime hours while meerkats were primarily foraging and include several different call types. Field work was

carried out by Ariana Strandburg-Peshkin, Baptiste Averly, Vlad Demartsev, Gabriella Gall, Rebecca Schaefer and Marta Manser. Audio recordings were labelled by Baptiste Averly, Vlad Demartsev, Ariana Strandburg-Peshkin, and colleagues.

**Jackdaws (JD):** Jackdaws are corvid songbirds that usually breed, forage and sleep in large groups. In a multi-year field study (Max-Planck-Institute for Ornithology, Seewiesen, Germany), wild jackdaws were equipped with small backpacks containing minia-ture voice recorders (Edic Mini Tiny A31, TS-Market Ltd., Russia) to investigate the vocal behaviour of individuals interacting with their group and behaving freely in their natural environment. Field work was conducted by Lisa Gill, Magdalena Pelayo van Buuren and Magdalena Maier. Sound files were annotated by Lisa Gill.

**Western Mediterranean Wetlands Bird Dataset (WMW):** Contains bird sounds from 20 endemic species that are typically in-habitants of the “Aiguamolls de l’Empordà” natural park in Girona, Spain. The audio files that compose this dataset were originally retrieved from the Xeno-Canto portaland were manually cleaned and labelled by Juan Gómez-Gómez, Ester Vidaña-Vila and Xavier Sevillano using the Audacity software [13].

**HumBug (HB):** Mosquitoes produce sound both as a by-product of their flight and as a means for communication and mating. Fundamental frequencies vary in the range of 150 to 750 Hz [14]. As part of the HumBug project, acoustic data was recorded with a high specification field microphone (Telinga EM-23) coupled with an Olympus LS-14. The recordings used in this challenge are a subset of the datasets marked as ‘OxZoology’ and ‘Thailand’ from HumBugDB [15]<fnref n="3" />. The recordings contain the sound of lab-cultured *Culex quinquefasciatus* mosquitoes from Oxford, UK, and various species captured in the wild in Thailand, placed into plastic cups.[16]

**Polish Baltic Sea bird flight calls (PB):** The PB dataset consists of bird flight calls recordings from Hanna Pamula's project focused on the acoustic monitoring of birds migrating at night along the Polish Baltic Sea coast. Three autonomous recording units (Song Meters SM2, Wildlife Acoustics, Inc) were deployed close to each other (<100m) near a lake, on the dune, and on a forest clearing. The passerines night flight calls were annotated by Hanna Pamula. Target classes belong to: song thrush, *Turdus philomelos* and blackbird, *Turdus merula*. Event lengths vary between 8 to 400 milliseconds and the usual fundamental frequency range for calls is 5 to 9 kHz.

**Transfer-Exposure-Effects dataset (CHE):** This dataset comes from the Transfer Exposure-Effects (TREE) research project<fnref n="4" />. Data were collected using unattended acoustic recorders (Songmeter 3) in the Chornobyl Exclusion Zone (CEZ) to capture the Chornobyl soundscape and investigate the longterm effects of the nuclear plower plant accident on the local ecology. The fieldwork was designed and undertaken by Mike Wood (University of Salford), Nick Beresford (UK Centre for Ecology & Hydrology) and Sergey Gashchak (Chornobyl Center). Common Chiffchaff (Phylloscopus collybita) and Common Cuckoo (Cuculus canorus) vocalisations were manually annotated and labelled from these recordings by Helen Whitehead.

**BIOTOPIA Dawn Chorus (DC):** The Dawn Chorus project is a worldwide citizen science and arts project bringing together amateurs and experts to experience and record the dawn chorus at their doorstep. The DC dataset stems from dawn chorus recordings,

<footnote n="1">¹Dev set: https://doi.org/10.5281/zenodo.6012309</footnote>

<footnote n="2">²Eval set: https://doi.org/10.5281/zenodo.6517413</footnote>

<footnote n="3">³https://github.com/HumBug-Mosquito/HumBugDB/</footnote>

<footnote n="4">⁴https://tree.ceh.ac.uk/</footnote>
<table><caption>Table 1: Information on each dataset.</caption><thead><tr><th></th><th>Dataset</th><th>mic type</th><th># audio files</th><th>total duration</th><th># labels (excl. UNK)</th><th># events</th></tr></thead><tbody><tr><td rowspan="5">Development Set: Training</td><td>BV</td><td>fixed</td><td>5</td><td>10 hours</td><td>11</td><td>9026</td></tr><tr><td>HT</td><td>various</td><td>5</td><td>5 hours</td><td>5</td><td>611</td></tr><tr><td>MT</td><td>animal mounted</td><td>2</td><td>70 mins</td><td>4</td><td>1294</td></tr><tr><td>JD</td><td>mobile</td><td>1</td><td>10 mins</td><td>1</td><td>357</td></tr><tr><td>WMW</td><td>various</td><td>161</td><td>5 hours</td><td>26</td><td>2941</td></tr><tr><td rowspan="3">Development Set: Validation</td><td>HB</td><td>handheld</td><td>10</td><td>2.38 hours</td><td>1</td><td>712</td></tr><tr><td>PB</td><td>fixed</td><td>6</td><td>3 hours</td><td>2</td><td>292</td></tr><tr><td>ME</td><td>animal mounted</td><td>2</td><td>20 mins</td><td>2</td><td>73</td></tr><tr><td rowspan="6">Evaluation Set</td><td>CHE</td><td>fixed</td><td>18</td><td>3 hours</td><td>3</td><td>2550</td></tr><tr><td>DC</td><td>fixed</td><td>10</td><td>95 mins</td><td>3</td><td>967</td></tr><tr><td>CT</td><td>handheld</td><td>3</td><td>48 mins</td><td>3</td><td>365</td></tr><tr><td>MS</td><td>fixed</td><td>4</td><td>40 mins</td><td>1</td><td>1087</td></tr><tr><td>QU</td><td>animal mounted</td><td>8</td><td>74 mins</td><td>1</td><td>3441</td></tr><tr><td>MGE</td><td>fixed</td><td>3</td><td>32 mins</td><td>2</td><td>1195</td></tr></tbody></table>

made using Zoom H2 recorders at 44100 Hz, at three different locations in Southern Germany (Haspelmoor, Munich's Nymphenburg Schlosspark, and Nantesbuch), by Moritz Hertel and Rudi Schlich. The vocalisations of three target species were annotated by Lisa Gill (Common cuckoo, *Cuculus canorus*; European robin, *Erithacus rubecula*; Eurasian wren, *Troglodytes troglodytes*). A challenging aspect of this data is related to recordings being very busy with various other birds vocalising at the same time.

**Coati (CT):** Coatis are omnivorous diurnal mammals that live in stable social groups ranging from 5 to 30 individuals. The target calls used in this dataset are growls, chitters and chirp-grunts. Several other call types that might be confused with the targets were captured in the recordings which configures the main challenging aspect of this data. Audio recordings were collected from two adult females from the same group on Barro Colorado Island, Panama in March 2020. These individuals wore collars which recorded high resolution GPS data with an external attachment of a small audio recording device (TS Market, Edic Mini Tiny+ A77, 22050 Hz). Audio data were recorded during their active foraging period in daytime hours. Fieldwork was carried out by Emily Grout, José Ortega and Ben Hirsch.

**Manx Shearwater (MS):** Manx shearwaters are procellariform seabirds that breed in dense island colonies in the North Atlantic and winter in the South Atlantic off the South American coast. Adult Manx shearwaters make loud, distinctive vocalisations while present at their breeding colony in various contexts. The target class is Chick begging vocalisations which typically comprise bouts of short, high-pitched 'peeps'. In a multi-year study, Audiomoth recorders were placed in burrows during the breeding season. Fieldwork on Skomer Island was undertaken by various members of the Oxford Navigation Group (OxNav) and annotation was carried out by Joe Morford.

**Dolphin Quacks (QU):** Bottlenose dolphins are highly acoustic animals with an expansive repertoire of acoustic signals used for social interactions. The target class is Quacks which are short signals (around 100 ms), low-frequency and emitted at relatively high rates, often with 100s of quacks in a single short vocal bout. The recordings were obtained using sound-and-movement recording DTAGs (Johnson and Tyack 2003) attached with suction cups to bottlenose dolphins by F. Jensen in collaboration with Profs. Peter Tyack, Vincent Janik, and Laela Sayigh. All tags were deployed during routine health assessments conducted by the Sarasota dol-

phin research project and under a National Marine Fisheries Service research permit to Dr. Randall Wells of Chicago Zoological Society. Individual quacks were labelled by Austin Dziki and validated by F. Jensen.

**Chick calls (MGE):** Chickens are a precocial bird and upon hatching undergo a process of 'filial imprinting' whereby they establish a strong attachment to their mother. Chicks are active participants in this filial imprinting process and use their calls to signal they are in close proximity to their mother and other family members (i.e. pleasure calls) and to signal distress during social separation in order to solicit maternal contact (i.e. contact calls). The dataset includes three chicks with each chick recorded for 10 minutes; pleasure calls were annotated in recordings from chicks one and two, contact calls were annotated in recordings from chick three. All data was collected by Elisabetta Versace, Shu Wang, and Laura Freeland as part of a project from the Prepared Minds Lab from Queen Mary University of London<fnref n="5" />. All data were annotated by Shu Wang, Laura Freeland, and Michael Emmerson.

## 3. BASELINE METHODS AND EVALUATION

The baseline systems proposed did not change considerably from last year's edition [5]. Template Matching is based on spectrogram cross-correlation and still commonly used in bioacoustics. This approach scored surprisingly well on last edition evaluation set and thus it remains relevant as a baseline for this task. The second system proposed is based on prototypical networks which remain the state of the art for few-shot learning [4]. The changes from last year's system are the use of a ResNet, and adapting segment size depending on the target class in the query set. These changes mainly address the problem of high variation of event lengths and create a more adaptive system.

The evaluation of this task is based on an event-level *F*-measure with macro-averaged metric across all classes [5]. A positive match between predicted events and reference is found by applying the Intersection over Union (IoU) with 30% minimum overlap, followed by Hopcroft-Karp-Karzanov algorithm [18] for bipartite graph matching. True Positives (TP), False Positives (FP), and False Negatives (FN) can be computed after the matching step. These are defined as: TP - predicted events that match ground truth events; FP - predicted events that do not match any ground truth events;

<footnote n="5">⁵https://www.preparedmindslab.org/home</footnote>