To overcome these limitations, we propose to consider all the candidates produced by the model and select the best SPIDEr score between them with a new metric.

# 6. SPIDEr-max

## 6.1. Definition

We propose SPIDEr-max, defined by the following equation:

$$ \text{SPIDEr-max}(C, R) = \max_{i} \text{SPIDEr}(C_i, R) \quad (3) $$

where C is a list of N caption candidates and R a list of references.

It consists of retaining the largest SPIDEr score among the scores calculated for a set of caption candidates, to avoid having to choose a single hypothesis. The SPIDEr-max values are between 0 and 5.5, like the SPIDEr score. The source code in PyTorch will be made available on GitHub<fnref n="5" /> upon paper acceptance.

## 6.2. Results

The score of SPIDEr-max highly depends on how many candidates we use, so we report results with various beam sizes in figures 2 and 3 for AudioCaps and Clotho, respectively. We varied the beam size from 1 to 10. If we imagine a human end-user, proposing at most 5 candidates captions would be reasonable, in our opinion.

<figure><img src="image_50.png" /><figcaption>Figure 2: SPIDEr and SPIDEr-max scores with different beam sizes, calculated on the AudioCaps testing subset with CNN10-Transformer.</figcaption></figure>

On AudioCaps, the SPIDEr-max score increases rapidly above the score of our model from 0.401 to 0.473 with only a beam size of two. The scores continue to rise above the human SPIDEr score (0.565), meaning that our model is already producing human-like captions, but fail to select them, if we take the maximum likelihood criterion.

On Clotho, the scores also increase with a higher beam size, but they do not reach the cross-reference score on the first beam sizes. This is probably due to the references of Clotho, which show more diversity in terms of vocabulary and n-grams than AudioCaps.

We also tried to compute the SPIDEr-max score for a beam size equal to 100, which gave 0.953 on AudioCaps and 0.535 on Clotho, but we decided to focus on a few candidates, as it would be more realistic in a real scenario, where, for instance, automatic captions are proposed to an end-user.

<figure><img src="image_51.png" /><figcaption>Figure 3: SPIDEr and SPIDEr-max scores with different beam sizes, calculated on the Clotho development-testing subset with CNN10-Transformer.</figcaption></figure>

## 6.3. Why such a boost in SPIDEr-max?

As we saw in the previous section, SPIDEr-max increases rapidly and even outreaches the human SPIDEr score on AudioCaps. We also noticed that predicting a correct infrequent n-gram seems to drastically improves the score of a candidate, probably due to the CIDEr-D metric based on the TF-IDF of the n-grams. To see if there is a relation between TF-IDF and the SPIDEr and SPIDEr-max scores, we computed the difference between them with the best candidate given by the model and the best one given by the SPIDEr score for various beam sizes.

The correlation value between this variation of TF-IDF and SPIDEr scores is almost one for AudioCaps and Clotho. It suggests that the candidates selected by SPIDEr-max have a much higher TF-IDF than those selected by the model probabilities, which appears to significantly increase the CIDEr-D score and, thus, also the SPIDEr-max score.

# 7. CONCLUSION

In this paper, we showed that the SPIDEr score is very sensitive to the words used in the caption candidates, so we proposed a new metric, SPIDEr-max, that takes into account multiple candidates for each audio recording. The scores of SPIDEr-max compared to human scores of SPIDEr show that our model already produces human-like caption candidates, but selecting the caption with the highest SPIDEr score is not trivial. As future work, we are interested to study other metrics that do not use TF-IDF, such as model-based metrics like BERTScore [23] or FENSE [24]. We also look forward to testing SPIDEr-max with new models to see if our findings are repeated across architectures and training methods.

# 8. ACKNOWLEDGMENT

This work was partially supported by the Agence Nationale de la Recherche LUDAU (Lightly-supervised and Unsupervised Discovery of Audio Units using Deep Learning) project (ANR-18-CE23-0005-01). Experiments presented in this paper were carried out using the OSIRIM platform that is administered by IRIT and supported by CNRS, the Region Midi-Pyrénées, the French Government, ERDF (see http://osirim.irit.fr/).

<footnote n="5">⁵https://github.com/Labbeti/spider-max</footnote>
## 9. REFERENCES

[1] X. Xu, M. Wu, and K. Yu, "A comprehensive survey of automated audio captioning," arXiv preprint arXiv:2205.05357, 2022.

[2] Q. Kong, Y. Cao, T. Iqbal, Y. Wang, W. Wang, and M. D. Plumbley, "PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition," arXiv:1912.10211 [cs, eess], Aug. 2020, arXiv: 1912.10211.

[3] X. Mei, Q. Huang, X. Liu, G. Chen, J. Wu, Y. Wu, J. Zhao, S. Li, T. Ko, H. L. Tang, X. Shao, M. D. Plumbley, and W. Wang, "An encoder-decoder based audio captioning system with transfer and reinforcement learning for DCASE challenge 2021 task 6," DCASE2021 Challenge, Tech. Rep., July 2021.

[4] F. Gontier, R. Serizel, and C. Cerisara, "Automated audio captioning by fine-tuning bart with audioset tags," in DCASE 2021 - 6th Workshop on Detection and Classification of Acoustic Scenes and Events, Virtual, Spain, Nov. 2021.

[5] S. Liu, Z. Zhu, N. Ye, S. Guadarrama, and K. Murphy, "Improved Image Captioning via Policy Gradient optimization of SPIDER," 2017 IEEE International Conference on Computer Vision (ICCV), pp. 873-881, Oct. 2017, arXiv: 1612.00370.

[6] R. Vedantam, C. L. Zitnick, and D. Parikh, "CIDEr: Consensus-based Image Description Evaluation," arXiv:1411.5726 [cs], June 2015, arXiv: 1411.5726.

[7] P. Anderson, B. Fernando, M. Johnson, and S. Gould, "SPICE: Semantic Propositional Image Caption Evaluation," arXiv:1607.08822 [cs], July 2016, arXiv: 1607.08822.

[8] D. Klein and C. D. Manning, "Accurate unlexicalized parsing," in Proceedings of the 41st Annual Meeting on Association for Computational Linguistics - ACL '03, vol. 1. Sapporo, Japan: Association for Computational Linguistics, 2003, pp. 423-430.

[9] W.-N. Hsu, D. Harwath, C. Song, and J. Glass, "Text-Free Image-to-Speech Synthesis Using Learned Segmental Units," Dec. 2020, arXiv:2012.15454 [cs, eess].

[10] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, "BLEU: a method for automatic evaluation of machine translation," in Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02. Philadelphia, Pennsylvania: Association for Computational Linguistics, 2001, p. 311.

[11] C.-Y. Lin, "ROUGE: A package for automatic evaluation of summaries," in Text Summarization Branches Out. Barcelona, Spain: Association for Computational Linguistics, July 2004, pp. 74-81.

[12] M. Denkowski and A. Lavie, "Meteor Universal: Language Specific Translation Evaluation for Any Target Language," in Proceedings of the Ninth Workshop on Statistical Machine Translation. Baltimore, Maryland, USA: Association for Computational Linguistics, 2014, pp. 376-380.

[13] K. Drossos, S. Lipping, and T. Virtanen, "Clotho: An Audio Captioning Dataset," arXiv:1910.09387 [cs, eess], Oct. 2019, arXiv: 1910.09387.

[14] C. D. Kim, B. Kim, H. Lee, and G. Kim, "AudioCaps: Generating captions for audios in the wild," in Proceedings of the

2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Minneapolis, Minnesota: Association for Computational Linguistics, June 2019, pp. 119-132.

[15] J. F. Gemmeke, D. P. W. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, "Audio Set: An ontology and human-labeled dataset for audio events," in 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). New Orleans, LA: IEEE, Mar. 2017, pp. 776-780.

[16] I. Montani, M. Honnibal, M. Honnibal, S. V. Landeghem, A. Boyd, H. Peters, P. O. McCann, M. Samsonov, J. Geovedi, J. O'Regan, G. Orosz, D. Altinok, S. L. Kristiansen, Roman, E. Bot, L. Fiedler, G. Howard, W. Phatthiyaphaibun, Y. Tamura, S. Bozek, murat, M. Amery, B. Böing, P. K. Tippa, L. U. Vogelsang, B. Vanroy, R. Balakrishnan, V. Mazaev, and GregDubbin, "explosion/spaCy: v3.2.1: doc_cleaner component, new Matcher attributes, bug fixes and more," Dec. 2021.

[17] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, "Attention is all you need," CoRR, vol. abs/1706.03762, 2017.

[18] D. P. Kingma and J. Ba, "Adam: A Method for Stochastic Optimization," arXiv:1412.6980 [cs], Jan. 2017, arXiv: 1412.6980.

[19] D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le, "SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition," in Proc. Interspeech 2019, 2019, pp. 2613-2617.

[20] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, "Pytorch: An imperative style, high-performance deep learning library," in proc. NeurIPS, 2019, pp. 8026-8037.

[21] W. Falcon and al, "Pytorch lightning," GitHub. Note: https://github.com/PyTorchLightning/pytorch-lightning, vol. 3, 2019.

[22] X. Xu, Z. Xie, M. Wu, and K. Yu, "The SJTU system for DCASE2022 challenge task 6: Audio captioning with audio-text retrieval pre-training," DCASE2022 Challenge, Tech. Rep., July 2022.

[23] T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi, "BERTScore: Evaluating Text Generation with BERT," arXiv:1904.09675 [cs], Feb. 2020, arXiv: 1904.09675.

[24] Z. Zhou, Z. Zhang, X. Xu, Z. Xie, M. Wu, and K. Q. Zhu, "Can Audio Captions Be Evaluated with Image Caption Metrics?" Jan. 2022, number: arXiv:2110.04684 arXiv:2110.04684 [cs, eess].
# MULTI-SCALE ARCHITECTURE AND DEVICE-AWARE DATA-RANDOM-DROP BASED FINE-TUNING METHOD FOR ACOUSTIC SCENE CLASSIFICATION

Joo-Hyun Lee*, Jeong-Hwan Choi*, Pil Moo Byun*, and Joon-Hyuk Chang

Department of Electronic Engineering, Hanyang University, Seoul, Republic of Korea
{jhyun42, brent1104, fordream0309, jchang}@hanyang.ac.kr

## ABSTRACT

We propose a low-complexity acoustic scene classification (ASC) model structure suitable for short-segmented audio and fine-tuning methods for generalization to multiple recording devices. Based on the state-of-the-art architecture of the ASC, broadcasting-ResNet (BC-ResNet), we introduce BC-Res2Net that uses hierarchical residual-like connections within the frequency- and temporal-wise convolutions to extract multiscale features while using fewer parameters. We also incorporate the attention and aggregation method proposed in short-utterance speaker verification with BC-Res2Net to achieve high performance. In addition, we train the model with a novel fine-tuning method using a device-aware data-random-drop to avoid optimization for only a few devices. When the amount of data differed for each device in the training dataset, the proposed method gradually dropped the data of the primary device from the mini-batch. The experimental results on the TAU Urban Acoustic Scenes 2022 Mobile development dataset demonstrated the effectiveness of multi-scale modeling in short audio. Furthermore, the proposed training strategy significantly reduced the multi-class cross-entropy loss for various devices.

**Index Terms**— Acoustic scene classification, multi-scale, data imbalance, fine-tuning, short-segmented audio

## 1. INTRODUCTION

Remarkable progress in acoustic scene classification (ASC) has been accomplished with the development of deep learning, and several studies have recently been conducted to implant deep neural networks (DNNs) into low-resource devices that are suitable for practical applications [1, 2]. Notably, the Detection and Classification of Acoustic Scenes and Events (DCASE) Challenge have been held with various audio tasks, including ASC, and contributes to advance computational environmental audio analysis techniques. In DCASE Task 1, ASC systems should satisfy constraints such as data imbalance depending on the recording device, low complexity limitations, and short audio input while ensuring high classification accuracy [3, 4].

Owing to the limitation on the number of parameters, convolutional neural networks (CNNs) are preferred over DNNs for ASC models. To further enhance the feature extraction capability of CNNs, the ResNet [5] structure and depth-wise separable CNNs (DW-CNNs) [6] were adopted in [7] and [8], respectively. Moreover, the modified MobileNet [9], EfficientNet [10], and broadcasting-ResNet (BC-ResNet) [11] structures, which were designed to consider computational power, exhibited excellent performance [3, 12]. To improve the generalization of the model to the

multiple devices, ResNorm was proposed with BC-ResNet, which normalized the frequency bands with the residual path [12].

Several studies introduced the model architecture in the speaker verification field, which focused on the frequency of speech and channels in CNN to improve the verification performance in short-duration speech [13–15]. Liu et al. [14] conducted a multiscale frequency-channel attention (MFA) framework with frequency-channel attention and Res2Net structure [16], which learned multiscale features to emphasize the significant frequency and channel components. Jung et al. [15] proposed a feature pyramid module (FPM) that upsamples in a top-down pathway to effectively aggregate various-resolution feature maps.

This paper introduces two strategies for generalized ASC for various devices under low complexity and short input time conditions: improving the model structure, and training with a novel fine-tuning method. Inspired by Res2Net, we propose BC-Res2Net that are accomplished by modifying the BC-ResNet structure with multiscale modeling to increase the receptive field size of each CNN. We integrate the BC-Res2Net with MFA and FPM, which effectively extract and aggregate features from short speech signals, to construct the ASC model. Subsequently, we perform device-aware data-random-drop-based fine-tuning that drops data of the selected device in batch-level processing for the pretrained model to obtain consistent performance in various recording devices. We choose the device that recorded the most in the training dataset. We do not drop the data at the beginning of the fine-tuning but gradually increased the drop rate to a given parameter. In addition, we add regularization with a cross-entropy loss to avoid overfitting devices that are not selected.

## 2. ASC MODEL ARCHITECTURE

### 2.1. BC-Res2Net structure

Broadcasted residual learning [11] resiliently connects two-dimensional (2D) and one-dimensional (1D) feature maps with the input. These different sized feature maps are extracted by frequency-wise 2D and temporal-wise 1D DW-CNNs, respectively, and they contain frequency and frequency-aware temporal features. To correct the size mismatch, the output of the 1D DW-CNN, a 1D feature map is broadcast along the frequency axis. This residual connection was combined with basic structures such as ResNet and Transformer [17] to obtain state-of-the-art results in various audio and speech fields [11, 12, 18] where it is paramount to effectively capture frequency-time characteristics.

Res2Net, however, computes more efficiently than the conventional convolution-based structure because it comprises several CNNs connected in a hierarchical residual-like manner. In particular, the input feature map is sliced precisely with the same channel

<footnote>*Equal contributions.</footnote>
<figure><img src="image_52.png" /><figcaption>Figure 1: BC-Res2Net structure.</figcaption></figure>

size of CNNs. Each partitioned feature subset has a corresponding CNN that is fed while being sequentially added to the output of the previous CNN. The CNN outputs have various receptive field sizes owing to the multiscale operation. To extract the frequency and frequency-aware temporal features in a multiscale manner, we propose the BC-Res2Net that converts the frequency- and temporal-wise convolution of the BC-ResNet into a Res2Net structure. Figure 1 shows the network blocks of the BC-Res2Net comprising $\mathbf{F}_2$, $\mathbf{F}_1$, frequency average pooling, and point-wise 1D CNN, where $\mathbf{F}_2$ and $\mathbf{F}_1$ denote the Res2Net style 2D and 1D convolutions including nonlinear functions, respectively. Because the transition block is used when the given input and output feature map sizes are assigned differently, we describe the proposed structure based on the normal blocks.

The input feature map $\mathbf{X} \in \mathbb{R}^{C \times F \times T}$ is sliced into $S$ subfeature maps along with the channel axis, and the feature map subset is denoted by $\{\mathbf{x}_1, \mathbf{x}_2, ..., \mathbf{x}_S\}$. Except for the first subfeature map $\mathbf{x}_1$, each $\mathbf{x}_i \in \mathbb{R}^{C/S \times F \times T}$ has a corresponding frequency-wise 2D DW-CNN with a kernel size of $3 \times 1$ and subspectral normalization [19], denoted by $f_{2C,i}(\cdot)$ and $f_{SSN,i}(\cdot)$, respectively. The feature subset $\mathbf{x}_i$ is sequentially fed into $f_{2C,i}(\cdot)$ and $f_{SSN,i}(\cdot)$, after adding the output of $f_{SSN,i-1}(\cdot)$. The overall process for $\mathbf{F}_2$ can be formulated as follows:

$$ \mathbf{F}_2(\mathbf{x}_i) = \begin{cases} \mathbf{x}_i, & \text{if } i = 1 \\ f_{SSN,i}(f_{2C,i}(\mathbf{x}_i)), & \text{if } i = 2 \end{cases} \quad (1) $$

Frequency average pooling is applied after concatenating the set of outputs $\{\mathbf{F}_2(\mathbf{x}_1), \mathbf{F}_2(\mathbf{x}_2), ..., \mathbf{F}_2(\mathbf{x}_S)\}$ along the channel axis. The obtained feature map $Y \in \mathbb{R}^{C \times 1 \times T}$ is then fed into $\mathbf{F}_1$ to extract the temporal characteristics. $\mathbf{F}_1$ can be expressed as follows:

$$ \mathbf{F}_1(y_i) = \begin{cases} y_i, & \text{if } i = 1 \\ f_{ReLU,i}(f_{1C,i}(y_i)), & \text{if } i = 2 \\ f_{ReLU,i}(f_{1C,i}(y_i + \mathbf{F}_1(y_{i-1}))), & \text{otherwise} \end{cases} \quad (2) $$

where $y_i \in \mathbb{R}^{C/S \times 1 \times T}$ denotes the subfeature map of $Y$ that is sliced into $S$ along with the channel axis, and $f_{1C,i}$, and $f_{ReLU,i}(\cdot)$

denote a corresponding temporal-wise 1D DW-CNN with a kernel size of 3 and the ReLU activation, respectively. The following operations are performed sequentially: concatenating outputs of $\mathbf{F}_1$ into one, point-wise convolution, channel-wise dropout, and expanding the feature map size $\mathbb{R}^{C \times 1 \times T}$ to $\mathbb{R}^{C \times F \times T}$ along with the frequency axis. Finally, batch normalization with ReLU activation is applied after combining the output with two auxiliary residuals that the input identity and result of $\mathbf{F}_1$. Note that the BC-Res2Net operates with fewer CNN parameters than the BC-ResNet because the first subfeature map sliced from $\mathbf{F}_2$ and $\mathbf{F}_1$ does not proceed with convolution. The transition block differs from the normal block in two ways: auxiliary point-wise 2D CNN is applied before $\mathbf{F}_2$ to change the input channel size, and there is no residual connection for identity due to the size difference between the input and output.

## 2.2. ASC model for MFA and FPM

Short audio or speech makes feature extraction difficult because of insufficient temporal information. Several studies have focused on adding or enhancing the DNN structure in speaker verification fields to overcome performance degradation under short-utterance situations. In this study, we combined MFA and FPM, which improved the feature maps using the attention mechanism and aggregated the features from multiple resolutions, respectively, with the BC-Res2Net structure to introduce the ASC model for short audio.

<table><caption>Table 1: Architectures of proposed BC-Res2Net-based ASC model. $T$, $F$, and $C$ denote the number of time sequences, frequency bins, and CNN channel respectively. Input feature size is $1 \times F \times T$.</caption><thead><tr><th>Output size</th><th>Stage</th><th>Operator</th></tr></thead><tbody><tr><td>2C × F/2 × T/2</td><td>Stem</td><td>Conv2D [5 × 5], stride 2<br>BatchNorm + MFA</td></tr><tr><td>C × F/2 × T/2</td><td>Stage 1</td><td>BC-Res2Net × 2<br>ResNorm + MFA</td></tr><tr><td>1.5C × F/4 × T/4</td><td>Stage 2</td><td>Max-pool [2 × 2]<br>BC-Res2Net × 2<br>ResNorm + MFA</td></tr><tr><td>2C × F/8 × T/8</td><td>Stage 3</td><td>Max-pool [2 × 2]<br>BC-Res2Net × 2<br>ResNorm + MFA</td></tr><tr><td>2.5C × F/8 × T/8</td><td>Stage 4</td><td>BC-Res2Net × 3<br>ResNorm + MFA</td></tr><tr><td>4C × 1 × 1</td><td>Aggregation</td><td>FPM</td></tr><tr><td># Classes × 1 × 1</td><td>Classifier</td><td>Linear</td></tr></tbody></table>

The overall architecture is presented in Table 1. Assigning the importance of the frequency channel components for all the outputs is necessary because the output of each stage has a different resolution and receptive field. Therefore, we apply MFA to Stem and every stage after ResNorm. Figure 2 shows the FPM aggregating the MFA output of each stage in a bottom-up pathway, where the base CNN channel of the proposed model is set to 40. The last three feature maps are upsampled to the size of $\mathbb{R}^{C \times F/2 \times T/2}$, which is the same as the output of Stage 1. We considered the average of four feature maps across the frequency-temporal dimension and concatenated them into a single feature map with a size of $\mathbb{R}^{4C \times 1 \times 1}$. The upsampling method of the FPM is converted into the pixel shuffle method [20] for greater efficiency compared to the transposed convolution method proposed in [15]. The output of FPM is reshaped through a linear layer according to the number of classes.
<figure><img src="image_53.png" /><figcaption>Figure 2: Proposed BC-Res2Net-based ASC model architecture.</figcaption></figure>

### 3. FINE-TUNING METHOD USING DEVICE-AWARE DATA-RANDOM-DROP

When we train the ASC model with a significantly different amount of data for each device, the model undertakes greater effort to characterize a few specific devices occupying a large percentage of the dataset than to represent the general features of the overall device. To alleviate this issue, we load a pre-learning model that neglected the device types and fine-tuned the model by removing data from specific devices. Before fine-tuning, the last linear layer that classified the scenes is initialized and retrained to become more device-agnostic. Next, we fine-tune the model using the suggested device-aware data-random-drop method. It selects the device with the most data from the dataset and manages the mini-batch for every step by randomly removing data from it. However, an excessive drop of specific devices from the training process can lead to poor ASC performance owing to a lack of data diversity. Therefore, inspired by curriculum learning [21], we design a method in which the data-drop rate is initialized with zero and then increased step-wise. The drop rate gradually increases with the shape of the sigmoid function from zero to the given parameter. Furthermore, we add regularization to minimize the square weight difference between the fine-tuning and pretrained model parameters to prevent excessive loss of information from the selected device as given by:

$$L = L_C + \lambda \sum_i (\theta_i - \tilde{\theta}_{P,i})^2, \quad (3)$$

where $L$, $L_C$, and $\lambda$ denote the total loss, classification loss, and scaling factor of regularization, respectively. $\theta$ and $\tilde{\theta}_P$ denote the fine-tuned and pretrained model parameters, respectively, except for the classifier layer. The pretrained model parameters are stored in advance.

### 4. EXPERIMENTAL SETUP

#### 4.1. Datasets and preprocessing

The TAU Urban Acoustic Scenes 2022 Mobile Development dataset [4] had the same format as the 2020 development dataset [3], same sample rate of 44.1 kHz and 24 bits. However, this 2022 dataset segments were significantly shorter (1 s) compared to the last 2020 development set (10 s). In addition, the number of segments grew tenfold as the 2020 dataset split the 2022 dataset by 1 s.

<table><caption>Table 2: Ablation study of the BC-Res2Net evaluated on the TAU Urban Acoustic Scenes 2022 Mobile development dataset. (Acc. indicates the top-1 test accuracy(%).)</caption><thead><tr><th>Systems</th><th># Params</th><th>MACs</th><th>Log Loss</th><th>Acc.</th></tr></thead><tbody><tr><td>BC-ResNet-40</td><td>88.1K</td><td>17.21M</td><td>1.327</td><td>57.1</td></tr><tr><td>BC-Res2Net-40</td><td>85.8K</td><td>15.89M</td><td>1.235</td><td>59.1</td></tr><tr><td>w/ MFA</td><td>123.6K</td><td>17.45M</td><td>1.198</td><td>59.3</td></tr><tr><td>w/ FPM</td><td>93.6K</td><td>17.06M</td><td>1.212</td><td>59.5</td></tr><tr><td>w/ MFA & FPM</td><td>126.6K</td><td>26.76M</td><td><strong>1.167</strong></td><td><strong>60.8</strong></td></tr></tbody></table>

<table><caption>Table 3: Log loss and top-1 test accuracy (%) comparison for different duration of test audio on the TAU Urban Acoustic Scenes 2020 Mobile development dataset. (Dur. indicates durations.)</caption><thead><tr><th>Dur.</th><th>BC-ResNet-40</th><th>BC-Res2Net-40</th><th>BC-Res2Net-40 w/ MFA & FPM</th></tr></thead><tbody><tr><td>1 s</td><td>1.327 / 57.1</td><td>1.235 / 59.1</td><td><strong>1.167 / 60.8</strong></td></tr><tr><td>2 s</td><td>1.285 / 57.8</td><td>1.190 / 60.3</td><td><strong>1.146 / 61.6</strong></td></tr><tr><td>5 s</td><td>1.301 / 56.7</td><td>1.185 / 59.7</td><td><strong>1.172 / 60.5</strong></td></tr><tr><td>10 s</td><td>1.315 / 56.3</td><td>1.195 / 58.7</td><td><strong>1.192 / 59.5</strong></td></tr></tbody></table>

The audio segments were ten types of acoustic scenes from ten cities, recorded from three real devices (A, B, and C) and six simulated devices (S1-S6). According to the train-split method of [4], development dataset 2022 was separated into training and test subsets comprising 139,970 and 29,680 segments, respectively. In the training subset, the data for Device A accounted for 73% of the total. In the test split, the data from all the devices were evenly distributed. The test split contained data recorded with devices S4-S6, which were excluded in the training data split. The evaluation dataset was provided without labels for submitting the results.

We used the log Mel spectrum as the input feature for our system. The input features were prepared through three steps: downsampling from 44.1 kHz to 16.0 kHz, log Mel spectrum feature extraction, and data augmentation. The log Mel spectrograms were 256-dimensional, extracted with 2048 samples of the Hanning window, and 512 sample shifts. The input feature size obtained using the preprocessing method mentioned was [1, 256, 32]. The time-rolling method was used for time-domain augmentation. The input audio was randomly rolled along the time axis, ranging from -0.5-0.5 s, with out-of-range parts shifted to the other side. Specaugment [22], except time wrapping, was also employed with two frequency and temporal masks each. Mask parameters of 40 and 4 were used for the frequency and temporal masks, respectively. Each time-rolling and Specaugment mask was applied with a probability of 0.8. We also applied Mixup [23] with $\alpha = 0.3$ to the acoustic feature space.

#### 4.2. Implementation details

We trained the BC-Res2Net-based ASC model with pretraining and fine-tuning phases. In the pretraining phase, the AdamW optimizer [24] with a weight decay of 0.05 was used over 300 epochs, and the mini-batch size was set to 512. Warmup [25] was applied, where the learning rate linearly increased from 1e-8 to 0.01 over the first ten epochs and decayed to zero with a cosine annealing scheduler [26]. We applied a device-aware data-random-drop, treating the selected Device A as an excluded recording device in the fine-tuning phase. The mixup was disabled to correct the mismatch between the training and test conditions. The scaling factor of regularization was 0.4, and the AdamW optimizer with a weight decay of 1e-8 and fixed
<table><caption>Table 4: Device-wise top-1 test accuracy (%) and overall log loss comparison of proposed fine-tuning method according to maximum drop rate on the TAU Urban Acoustic Scenes 2022 Mobile development dataset. (Acc. indicates the top-1 test accuracy (%).)</caption><thead><tr><th rowspan="2">Systems</th><th rowspan="2">Fine-tuning<br>(Drop rate)</th><th colspan="6">Seen device</th><th colspan="3">Unseen device</th><th rowspan="2">Average<br>Log loss / Acc.</th></tr><tr><th>A</th><th>B</th><th>C</th><th>S1</th><th>S2</th><th>S3</th><th>S4</th><th>S5</th><th>S6</th></tr></thead><tbody><tr><td rowspan="4">BC-Res2Net-40<br>w/ MFA & FPM</td><td>✗</td><td>72.0</td><td>64.5</td><td>68.2</td><td>62.7</td><td>59.5</td><td>64.2</td><td>54.4</td><td>56.4</td><td>45.3</td><td>1.167 / 60.8</td></tr><tr><td>✓ (0.00)</td><td>72.8</td><td><b>68.1</b></td><td>69.7</td><td>63.1</td><td>62.0</td><td>66.2</td><td>53.4</td><td>56.0</td><td>47.6</td><td>1.083 / 62.1</td></tr><tr><td>✓ (0.50)</td><td>73.0</td><td>67.8</td><td>70.1</td><td>62.9</td><td>61.6</td><td><b>66.5</b></td><td>55.2</td><td>56.8</td><td>48.4</td><td>1.085 / 62.5</td></tr><tr><td>✓ (0.90)</td><td><b>73.2</b></td><td>68.0</td><td>69.4</td><td><b>63.7</b></td><td>61.6</td><td>66.2</td><td>55.2</td><td>57.3</td><td><b>49.0</b></td><td><b>1.076 / 62.6</b></td></tr><tr><td></td><td>✓ (0.99)</td><td>72.7</td><td>67.2</td><td><b>70.3</b></td><td>63.0</td><td><b>62.2</b></td><td>66.0</td><td><b>55.9</b></td><td><b>57.4</b></td><td>48.8</td><td>1.081 / 62.6</td></tr></tbody></table>

<figure><img src="image_54.png" /><figcaption>Figure 3: Top-1 test accuracy comparison of the BC-Res2Net and the BC-ResNet according to model parameters and MACs.</figcaption></figure>

learning rate of 1e-5 was used. For the model structure, both $F_2$ and $F_1$ of the BC-Res2Net were sliced into four subchannels. Sub-spectral normalization [19] with four sub-bands and ResNorm with 0.1 (hyperparameter of the identity shortcut path,) were applied to the BC-Res2Net.

## 5. RESULT

We evaluated the result in terms of the top-1 test accuracy and multiclass cross-entropy (log loss). We also reported the number of model parameters and multiply-accumulate operations (MACs) were used to observe computational complexity. Figure 3 shows the comparison of the BC-ResNet and BC-Res2Net when the base CNN channel size increases from 40 to 80. For all the CNN channel sizes, the BC-Res2Net achieves higher accuracy than the BC-ResNet while having small MACs and model parameters. Table 2 presents the effects of the proposed structural modifications on the ASC model. The BC-Res2Net-40 requires 2.6% fewer parameters and 7.7% fewer MACs than the BC-ResNet-40 but performs better in terms of log loss and accuracy. When MFA and FPM were applied to the BC-Res2Net, the accuracy improved by 0.2% and 0.3%, respectively; when both were applied, the accuracy improved by 1.7%. Table 3 shows the results for the short audio conditions. We evaluated the cropped test data with the given duration within each 10 s audio of the 2020 data. The BC-Res2Net performed better than the BC-ResNet for all the test lengths; in particular, the model that added MFA and FPM to the BC-Res2Net obtained better results at shorter durations of 1 s and 2 s. These results show that the BC-Res2Net extracts the information required to classify the scenes more effectively than the BC-ResNet, and using MFA and FPM additionally assists in classifying sound in short-segmented audio. Table 4 presents the effect of the fine-tuning method based on the maximum drop rate. Compared with the pretrained model, the overall accuracy and log loss improved by 0.084 and 1.3%, respectively; when the fine-tuning was applied without data-random-drop,

and better performance was achieved when the drop was applied to the selected Device A. Maximum drop rate of 0.9 exhibited the best average log loss and accuracy, and achieved significant improvements of 0.091 and 1.8%, respectively, compared to the case when fine-tuning was not applied. In particular, on the seen device, the performance of multiple devices including Device A was improved evenly, and the performance improvement was observed even in the unseen device, showing that the proposed fine-tuning method benefits generalization of the device.

## 6. RELATIONSHIPS WITH TECHNICAL REPORT

In a technical report [27], quantization-aware training (QAT) [28] was additionally introduced to satisfy the quantization conditions of INT8. The log loss and accuracy of the QAT-applied BC-Res2Net-40-based ASC model were degraded to be 1.193 and 60.3%, respectively, compared with the results without quantization. We submitted the outputs of the two systems with the proposed fine-tuning in QAT environment. To investigate the effect of the proposed fine-tuning according to the regularization scaling factor, we submitted systems results trained with the fine-tuning method with a drop rate of 0.9 and the regularization factors with 0.04 and 0.4. Each result achieved log losses of 1.072 (Acc. 62.2%) and 1.065 (Acc. 62.6%), respectively, for the test set of the development dataset. For the rest of the two trials, we additionally applied knowledge distillation [29] introduced in [7] to improve the performance. The best result was assigned the teacher model size equal to the student model and the scaling factor of regularization to 0.4 and achieved a log loss of 0.835 and accuracy of 70.1% from the development dataset. Finally, this result achieved a log loss of 1.147 and accuracy of 60.8% in the challenge evaluation and placed second in the competition.

## 7. CONCLUSION

We proposed the BC-Res2Net by modifying the BC-ResNet in a multiscale manner. Moreover, we improved ASC performance under short audio evaluation conditions by using the MFA and the FPM method, which finds important components among frequency and channel components and effectively aggregates feature maps of different resolutions. Finally, we suggested the device-aware data-random-drop method-based fine-tuning method to promote optimization for multiple devices.

## 8. ACKNOWLEDGMENT

This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) (No. 2021-0-00456, Development of Ultra-high Speech Quality Technology for Remote Multi-speaker Conference System)
## 9. REFERENCES

[1] M. Valenti et al., “DCASE 2016 acoustic scene classification using convolutional neural networks,” in Proc. Workshop Detection Classif. Acoust. Scenes Events, 2016, p. 95–99.

[2] S. Chu, S. Narayanan, and C.-C. J. Kuo, “Environmental sound recognition with time-frequency audio features,” IEEE Transactions on Audio, Speech, and Language Processing, vol. 17, no. 6, pp. 1142–1158, 2009.

[3] I. Martín-Morató, A. Ancilotto, T. Heittola, A. Mesaros, and T. Virtanen, “Low-complexity acoustic scene classification for multi-device audio: analysis of DCASE 2021 challenge systems,” arXiv:2105.13734, 2021.

[4] I. Martín-Morató, F. Paissan, A. Ancilotto, T. Heittola, A. Mesaros, E. Farella, A. Brutti, and T. Virtanen, “Low-complexity acoustic scene classification in DCASE 2022 challenge,” arXiv:2206.03835, 2022.

[5] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” arXiv:1512.03385, 2015.

[6] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, “MobileNets: Efficient convolutional neural networks for mobile vision applications,” arXiv:1704.04861, 2017.

[7] B. Lehner and K. Koutini, “Acoustic scene classification with reject option based on ResNets,” DCASE 2019 Challenge, Tech. Rep., 2019.

[8] T. Heittola, A. Mesaros, and T. Virtanen, “Acoustic scene classification in DCASE 2020 challenge: generalization across devices and low complexity solutions,” in Proc. Detection and Classification of Acoustic Scenes and Events (DCASE) Workshop, 2020, p. 56–60.

[9] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, “MobileNetV2: Inverted residuals and linear bottlenecks,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, p. 4510–4520.

[10] M. Tan and Q. V. Le, “EfficientNet: Rethinking model scaling for convolutional neural networks,” in Proc. International Conference on Machine Learning (ICML), vol. 97, 2019, pp. 6105–6114.

[11] B. Kim, S. Yang, J. Lee, and D. Sung, “Broadcasted residual learning for efficient keyword spotting,” in Proc. INTERSPEECH, 2021, p. 4538–4542.

[12] B. Kim, S. Yang, J. Ki, and S. Chang, “Domain generalization on efficient acoustic scene classification using residual normalization,” in Proc. Detection and Classification of Acoustic Scenes and Events (DCASE) Workshop, 2021, p. 21–25.

[13] J.-H. Choi, J.-Y. Yang, and J.-H. Chang, “Short-utterance embedding enhancement method based on time series forecasting technique for text-independent speaker verification,” in IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 2021, pp. 130–137.

[14] T. Liu, R. K. Das, K. Aik Lee, and H. Li, “MFA: TDNN with multi-scale frequency-channel attention for text-independent speaker verification with short utterances,” in Proc. IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2022, pp. 7517–7521.

[15] Y. Jung, S. M. Kye, Y. Choi, M. Jung, and H. Kim, “Improving multi-scale aggregation using feature pyramid module for robust speaker verification of variable-duration utterances,” in Proc. INTERSPEECH, 2020, pp. 1501–1505.

[16] S.-H. Gao, M.-M. Cheng, K. Zhao, X.-Y. Zhang, M.-H. Yang, and P. Torr, “Res2Net: A new multi-scale backbone architecture,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 43, no. 6, p. 652–662, Feb. 2021.

[17] A. Vaswani et al., “Attention is all you need,” in Proc. Advances in Neural Information Processing Systems (NIPS), 2017, p. 3830–3834.

[18] J.-H. Choi, J.-Y. Yang, Y.-R. Jeoung, and J.-H. Chang, “Improved CNN-Transformer using broadcasted residual learning for text-independent speaker verification,” in Proc. INTERSPEECH, 2022.

[19] S. Chang, H. Park, J. Cho, H. Park, S. Yun, and K. Hwang, “Subspectral normalization for neural audio data processing,” in Proc. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021, pp. 850–854.

[20] W. Shi et al., “Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 1874–1883.

[21] Y. Bengio, J. Louradour, R. Collobert, and J. Weston, “Curriculum learning,” in Proc. International Conference on Machine Learning (ICML), 2009, p. 41–48.

[22] D. S. Park et al., “SpecAugment: A simple data augmentation method for automatic speech recognition,” in Proc. INTERSPEECH, 2019, pp. 2613–2618.

[23] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, “Mixup: Beyond empirical risk minimization,” in Proc. International Conference on Learning Representations (ICLR), 2018.

[24] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in Proc. International Conference on Learning Representations (ICLR), 2019.

[25] P. Goyal, “Accurate, large minibatch SGD: Training imagenet in 1 hour,” arxiv:1706.02677, 2017.

[26] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, “SGDR: Stochastic gradient descent with warm restarts,” in Proc. International Conference on Learning Representations (ICLR), 2017.

[27] J.-H. Lee, J.-H. Choi, P. M. Byun, and J.-H. Chang, “HYU submission for the DCASE 2022: fine-tuning method using device-aware data-random-drop for device-imbalanced acoustic scene classification,” DCASE 2022 Challenge, Tech. Rep., 2022.

[28] https://pytorch-lightning.readthedocs.io/en/stable/_modules/pytorch Lightning/callbacks/quantization.html.

[29] G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” in Proc. NIPS Deep Learning and Representation Learning Workshop, 2015.
# UNSUPERVISED ANOMALOUS SOUND DETECTION FOR MACHINE CONDITION MONITORING USING TEMPORAL MODULATION FEATURES ON GAMMATONE AUDITORY FILTERBANK

Kai Li, Quoc-Huy Nguyen, Yasuji Ota, and Masashi Unoki*

School of Information Science,
Japan Advanced Institute of Science and Technology
1-1, Asahidai, Nomi, Ishikawa, 923-1292, Japan
{kai_li, hqnguyen, y_ota, unoki}@jaist.ac.jp

## ABSTRACT

Anomalous sound detection (ASD) is a technique to determine whether the sound emitted from a target machine is anomalous or not. Subjectively, timbral attributes, such as sharpness and roughness, are crucial cues for human beings to distinguish anomalous and normal sounds. However, the feature frequently used in existing methods for ASD is the log-Mel-spectrogram, which is difficult to capture temporal information. This paper proposes an ASD method using temporal modulation features on the gammatone auditory filterbank (TMGF) to provide temporal characteristics for machine-learning-based methods. We evaluated the proposed method using the area under the ROC curve (AUC) and the partial area under the ROC curve (pAUC) with sounds recorded from seven kinds of machines. Compared with the baseline method of the DCASE2022 challenge, the proposed method provides a better ability for domain generalization, especially for machine sounds recorded from the valve.

**Index Terms—** Anomalous sound detection, gammatone filterbank, temporal modulation features, timbre information, deep learning

## 1. INTRODUCTION

Anomalous sound detection (ASD) is a technique to determine whether the sound recorded from a target machine is anomalous or not. It enables workers to arrange maintenance work to fix machine problems in the earliest stages, thus reducing maintenance costs and preventing consequential damages. ASD for machine condition monitoring purposes has received increasing attention.

ASD is often viewed as an unsupervised problem due to difficulties in collecting anomalous sounds that can cover all possible types of anomalies. Autoencoder (AE)-based unsupervised methods, such as those in [1, 2, 3], were popularly used. These methods simulated the distribution of normal sounds by minimizing the reconstruction error of normal training data. Then, the reconstruction scores from the testing data were used to detect the anomalies. Some improved AE models, such as Heteroskedastic Variational AE (HVAE) [4] and Conformer-based AE [5], have also been proposed to improve the performance of ASD. However, the performance of

AE-based ASD systems depends significantly on the discrimination of input features.

The log-Mel-spectrogram (LMS) is widely used as input feature in ASD [1, 3, 6]. It is designed in accordance with the pitch perception of the human ear and has high resolution in the low frequency and low resolution in the high frequency [7]. However, the discriminative information of sounds emitted from different kinds of machines may be encoded non-uniformly in the frequency domain. The Mel filterbank may filter out important information concealed in the high-frequency components and hence decrease the performance of an ASD system. Furthermore, the LMS focuses on discriminative information from the frequency domain, making it difficult to capture temporal information.

Because of the drawbacks of the LMS, other ASD methods considered temporal information to improve detection results. In [8], a temporal feature is extracted from the raw waveform by a CNN-based network (TgramNet) to compensate for the anomalous information unavailable from the LMS. This complementary information can further improve the results of ASD systems. However, there is still a lot of redundant information with the raw waveform as a front-end feature and cannot distinguish between normal and anomalous sounds well.

For human beings, it is pretty easy to distinguish anomalous and normal sounds by perceiving auditory attributes (loudness, pitch, and timbre), especially timbral attributes, such as sharpness and roughness [9]. A feature that includes more timbral information is crucial for perceptually distinguish anomalous and normal sounds. However, a specially designed feature from the perspective of human perception for ASD has not been developed.

This paper proposes a method to use temporal modulation features on the gammatone auditory filterbank (TMGF) [10] combined with a simple AE-based detector for the ASD task. This paper assumes that the TMGF feature can provide much more information related to human perception, especially timbral attributes. The proposed method is evaluated by experiments on the Task 2 dataset of the DCASE2022 challenge [1]. The results show that the proposed method outperforms the baseline system in the target evaluation.

## 2. BASELINE METHOD

The AE-based system was selected as a baseline [1]. In the baseline system, the LMS of the input audio $X = \{X_t\}_{t=1}^T$ was extracted and fed into an AE-based detector, where $X_t \in \mathbb{R}^F$, $F$ and $T$ are the number of Mel-filters and time-frames, respectively.

<footnote>*Corresponding author. This work was supported by SCOPE Program of Ministry of Internal Affairs and Communications (Grant Number: 201605002) and the Fund for the Promotion of Joint International Research (Fostering Joint International Research (B))(20KK0233).</footnote>
<figure><img src="image_55.png" /><figcaption>Figure 1: Proposed system using temporal modulation features on the gammatone auditory filterbank (TMGF, $S_{m,k}(t)$) for anomalous sound detection (ASD).</figcaption></figure>

Then, the acoustic feature at $t$ is obtained by concatenating consecutive frames of the log-Mel-spectrogram as $\delta_t = (X_t, ..., X_{t+P-1})$, where $D = P \times F$, $P$ is the number of frames of the context window. The anomaly score is calculated as

$$A_{\theta}(X) = \frac{1}{DT} \sum_{t=1}^{T} \|\delta_t - \mathbb{F}(\delta_t)\|_{2}^{2}, \quad (1)$$

where $\mathbb{F}(\cdot)$ is the vector reconstruct function using the AE model, and $\|\cdot\|_2$ is $\ell_2$ norm. As shown in Fig. 1, the AE model includes an encoder, bottleneck layer, and decoder modules. All modules consist of fully-connected layers. The training of the AE model is a regression mission due to the fact that only normal sounds can be used in model training. Finally, the mean squared error (MSE) is used as the cost function to optimize the overall system.

To determine the anomaly detection threshold, the baseline method assumes that $A_{\theta}$ follows a gamma distribution. The gamma distribution parameters are estimated from the histogram of $A_{\theta}$, and the anomaly detection threshold is determined as the 90th percentile of the gamma distribution. If $A_{\theta}$ for each test clip is greater than this threshold, the clip is judged to be abnormal; otherwise, it is judged to be normal.

### 3. PROPOSED TMGF FEATURES

The temporal modulation on an auditory filterbank contains important information related to the timbre of a sound, such as the sharpness, roughness, and fluctuation strength [11, 12, 13]. Such information visualizes how humans perceive a sound as well as how we judge a sound (i.e., as "anomalous" or "normal"). Also, different frequencies of temporal modulation contain different levels of speech information such as speech intelligibility, speaker identity, and emotion. Thus, we aim to utilize the temporal modulation feature for detecting anomalous sound. The extraction processes are based on those from Huy. et al. [10].

The gammatone filter [14] is a well-known auditory filter model. The impulse response of a gammatone analysis filter at the center frequency $f_c$ is defined as

$$g(t) = at^{n-1}e^{-2\pi b\text{ERB}(f_c)t}e^{j2\pi f_c t}, \quad (2)$$

where $t \ge 0$ is time in seconds, $a$ is the amplitude, $n$ is the filter order, and $b$ is the bandwidth coefficient. The equivalent rectangular bandwidth ERB($f_c$) is defined as

$$\text{ERB}(f_c) = 24.7 + 0.108f_c. \quad (3)$$

Using K gammatone filters $\{g^{(k)}(t)\}_{k=0}^{K-1}$ with different center frequencies, from an input signal $x(t)$, the output of the filterbank

$X_k(t)$ can be expressed as the product of the amplitude modulation $A_k(t)$ and the complex carrier $e^{j\phi_k(t)}$, as

$$\begin{aligned} X_k(t) &= x(t) * g^{(k)}(t) \\ &= A_k(t)e^{j\phi_k(t)}. \end{aligned} \quad (4)$$

The gammatone filterbank can be implemented using a wavelet transform where the mother wavelet is $\psi(t) = g(t)$ [15]. Then, with an $\alpha > 1$, the k-th filter $g^{(k)}(t)$ can be defined by scaling $\psi(t)$ with a factor $\alpha_k$ of $t$, as

$$g^{(k)}(t) = \psi(\alpha_k t), \quad (5)$$

$$\alpha_k = \alpha^{\frac{2k}{K-1}-1}. \quad (6)$$

To analyze different frequency components of $A_{k,t}$, we use a modulation filterbank [16, 17] consisting of M filters $\{h^{(m)}(t)\}_{m=1}^M$. The first filter $h^{(1)}(t)$ is a low-pass filter with a cut-off frequency of $f_1$. For each $m \ge 2$, the filter $h^{(m)}(t)$ is a band-pass filter of which the frequency ranges from $2^{m-2}f_1$ to $2^{m-1}f_1$. Using the designed modulation filterbank, the TMGF features can be extracted from the amplitude modulation $A_{k,t}$ as

$$S_{m,k}(t) = A_k(t) * h^{(m)}(t). \quad (7)$$

### 4. EXPERIMENTAL SETUP

#### 4.1. Datasets

The datasets used in this task were provided by the DCASE2022 organizers [18, 19]. The data includes normal and anomalous sounds recorded from seven machines: fan, gearbox, bearing, slide, tor car, toy train, and valve. Each recorded sound includes the target machine's sounds and environmental sounds. To simplify the task, only the first channel of multi-channel audio is used. The length of each recorded sound is fixed to 10 s, and the sampling rate is 16 kHz.

The data is divided into three datasets: development, additional training, and evaluation. Each dataset includes audio from these seven types of machines. Machines in the development dataset include sections 01, 02, and 03. Machines in the additional training dataset and evaluation dataset include sections 04, 05, and 06. Each section was divided into source and target domains due to the differences in operating speed, machine load, viscosity, heating temperature, type of environmental noise, signal-to-noise ratio (SNR), etc. Different domains are split into a training and testing subset—the training dataset includes normal sounds only, but the testing dataset includes normal and abnormal sounds. In our experiments, training data in the development dataset was used for model training, and test data in the development dataset was used for testing.
<table><caption>Table 1: Overall results of the proposed (TMGF) and baseline (BL) methods in terms of AUC and pAUC.</caption><thead><tr><th rowspan="2">Machines</th><th rowspan="2">Sections</th><th colspan="2">AUC (source)</th><th colspan="2">AUC (target)</th><th colspan="2">pAUC</th></tr><tr><th>BL (%)</th><th>TMGF (%)</th><th>BL (%)</th><th>TMGF (%)</th><th>BL (%)</th><th>TMGF (%)</th></tr></thead><tbody><tr><td rowspan="4">Toy car</td><td>0</td><td>85.54</td><td>62.62</td><td>45.06</td><td>40.78</td><td>51.89</td><td>47.79</td></tr><tr><td>1</td><td>87.22</td><td>67.66</td><td>42.02</td><td>39.76</td><td>53.53</td><td>48.42</td></tr><tr><td>2</td><td>99.04</td><td>71.62</td><td>26.44</td><td><strong>42.66</strong></td><td>54.32</td><td><strong>55.53</strong></td></tr><tr><td>Arithmetic mean</td><td>90.60</td><td>67.30</td><td>37.84</td><td><strong>41.07</strong></td><td>53.25</td><td>50.58</td></tr><tr><td>Harmonic mean</td><td>90.22</td><td>67.10</td><td>35.79</td><td><strong>41.03</strong></td><td>53.23</td><td>50.35</td></tr><tr><td rowspan="4">Toy train</td><td>0</td><td>66.78</td><td>44.26</td><td>32.94</td><td>25.84</td><td>51.63</td><td>48.74</td></tr><tr><td>1</td><td>77.56</td><td>61.82</td><td>30.58</td><td><strong>45.92</strong></td><td>50.37</td><td>49.37</td></tr><tr><td>2</td><td>83.42</td><td>45.86</td><td>15.92</td><td><strong>49.76</strong></td><td>49.47</td><td><strong>51.05</strong></td></tr><tr><td>Arithmetic mean</td><td>75.92</td><td>50.65</td><td>26.48</td><td><strong>40.51</strong></td><td>50.49</td><td>49.72</td></tr><tr><td>Harmonic mean</td><td>75.27</td><td>49.53</td><td>23.83</td><td><strong>37.23</strong></td><td>50.48</td><td>49.70</td></tr><tr><td rowspan="4">Bearing</td><td>0</td><td>50.24</td><td>62.86</td><td>62.88</td><td><strong>63.46</strong></td><td>51.53</td><td><strong>52.84</strong></td></tr><tr><td>1</td><td>66.12</td><td><strong>66.44</strong></td><td>63.96</td><td>62.42</td><td>52.79</td><td>49.53</td></tr><tr><td>2</td><td>42.14</td><td><strong>55.70</strong></td><td>54.74</td><td><strong>62.64</strong></td><td>48.47</td><td><strong>66.05</strong></td></tr><tr><td>Arithmetic mean</td><td>52.83</td><td><strong>61.67</strong></td><td>60.53</td><td><strong>62.84</strong></td><td>50.93</td><td><strong>56.14</strong></td></tr><tr><td>Harmonic mean</td><td>51.06</td><td><strong>61.33</strong></td><td>60.23</td><td><strong>62.84</strong></td><td>50.86</td><td><strong>55.29</strong></td></tr><tr><td rowspan="4">Fan</td><td>0</td><td>82.04</td><td><strong>84.20</strong></td><td>38.66</td><td><strong>42.00</strong></td><td>59.63</td><td>50.11</td></tr><tr><td>1</td><td>72.46</td><td>51.84</td><td>46.04</td><td><strong>49.48</strong></td><td>51.63</td><td>50.95</td></tr><tr><td>2</td><td>81.84</td><td>78.58</td><td>65.64</td><td><strong>67.50</strong></td><td>63.89</td><td><strong>64.37</strong></td></tr><tr><td>Arithmetic mean</td><td>78.78</td><td>71.54</td><td>50.11</td><td><strong>52.99</strong></td><td>58.39</td><td>55.14</td></tr><tr><td>Harmonic mean</td><td>78.52</td><td>68.35</td><td>47.75</td><td><strong>50.99</strong></td><td>57.93</td><td>54.43</td></tr><tr><td rowspan="4">Gearbox</td><td>0</td><td>64.34</td><td>36.02</td><td>65.00</td><td>49.60</td><td>61.26</td><td>49.60</td></tr><tr><td>1</td><td>65.84</td><td>59.22</td><td>57.40</td><td>54.86</td><td>53.63</td><td>50.58</td></tr><tr><td>2</td><td>74.64</td><td>67.96</td><td>66.04</td><td><strong>66.22</strong></td><td>62.11</td><td>58.05</td></tr><tr><td>Arithmetic mean</td><td>68.27</td><td>54.40</td><td>62.81</td><td>56.89</td><td>59.00</td><td>52.74</td></tr><tr><td>Harmonic mean</td><td>67.98</td><td>50.54</td><td>62.57</td><td>56.08</td><td>58.74</td><td>52.48</td></tr><tr><td rowspan="4">Slider</td><td>0</td><td>80.42</td><td>46.26</td><td>56.82</td><td>45.12</td><td>62.21</td><td>48.26</td></tr><tr><td>1</td><td>67.04</td><td>50.22</td><td>50.18</td><td><strong>63.06</strong></td><td>53.05</td><td>53.05</td></tr><tr><td>2</td><td>86.78</td><td>23.88</td><td>40.82</td><td><strong>53.60</strong></td><td>54.37</td><td>48.37</td></tr><tr><td>Arithmetic mean</td><td>78.08</td><td>40.12</td><td>49.27</td><td><strong>53.93</strong></td><td>56.54</td><td>49.89</td></tr><tr><td>Harmonic mean</td><td>77.17</td><td>35.97</td><td>48.37</td><td><strong>52.93</strong></td><td>56.27</td><td>49.80</td></tr><tr><td rowspan="4">Valve</td><td>0</td><td>54.66</td><td><strong>98.66</strong></td><td>51.96</td><td><strong>98.30</strong></td><td>52.26</td><td><strong>94.37</strong></td></tr><tr><td>1</td><td>50.58</td><td><strong>59.80</strong></td><td>52.06</td><td><strong>60.94</strong></td><td>49.95</td><td><strong>54.16</strong></td></tr><tr><td>2</td><td>50.88</td><td><strong>95.86</strong></td><td>43.40</td><td><strong>97.08</strong></td><td>48.79</td><td><strong>89.11</strong></td></tr><tr><td>Arithmetic mean</td><td>52.04</td><td><strong>84.77</strong></td><td>49.14</td><td><strong>85.44</strong></td><td>50.33</td><td><strong>79.21</strong></td></tr><tr><td>Harmonic mean</td><td>51.98</td><td><strong>80.45</strong></td><td>48.78</td><td><strong>81.34</strong></td><td>50.29</td><td><strong>74.47</strong></td></tr><tr><td rowspan="2">Average</td><td>Arithmetic mean</td><td>70.93</td><td>61.49</td><td>48.03</td><td><strong>56.24</strong></td><td>54.13</td><td><strong>56.20</strong></td></tr><tr><td>Harmonic mean</td><td>67.57</td><td>55.53</td><td>42.53</td><td><strong>51.56</strong></td><td>53.76</td><td><strong>54.26</strong></td></tr></tbody></table>

## 4.2. Metrics

To evaluate the performance of an ASD system, the area under the curve (AUC) and partial-AUC (pAUC) for receiver operating characteristic (ROC) curves are used. The pAUC is an AUC calculated from a portion of the ROC curve over a pre-specified range of interest. To increase the reliability, the pAUC is calculated as the AUC over a low false-positive-rate (FPR) range [0, p], where p = 0.1 is used. According to [20], the AUC and pAUC for each machine type, section, and domain can be calculated as

$$AUC_{m,n,d} = \frac{1}{N_d^- N_n^+} \sum_{i=1}^{N_d^-} \sum_{l=1}^{N_n^+} \mathcal{H}(A_{\theta}(x_l^+) - A_{\theta}(x_i^-)), \quad (8)$$

$$pAUC_{m,n} = \frac{1}{[pN_n^-] N_n^+} \sum_{i=1}^{[pN_n^-]} \sum_{l=1}^{N_n^+} \mathcal{H}(A_{\theta}(x_l^+) - A_{\theta}(x_i^-)), \quad (9)$$

where *m* represents the index of a machine type, *n* represents the index of a section, *d* = {source, target} represents a domain, |·| is the flooring function, and $\mathcal{H}(x)$ returns 1 when $x > 0$ and 0 otherwise. ${x_i^-}_{i=1}^{N_-}$ and ${x_l^+}_{l=1}^{N_+}$ are normal and anomalous test clips in domain *d* in section *n* in machine type *m*, respectively. $N_-$ and $N_+$ are the number of normal and anomalous test clips in domain *d* in section *n* in machine type *m*, respectively.

## 4.3. Experimental conditions

To extract the LMS feature, 10-s audio clips were first split into different frames with frame lengths of 64 ms and hop lengths of 32 ms. Then, the Mel-spectrogram feature is extracted using the *melspectrogram* module in the *librosa* library with the following parameters: n resc = 1024, hop length = 512, T = 128, and power = 2.0. Finally, five Mel-spectrogram features (*P* = 5) were concatenated into one feature vector with a dimension of 640 and fed into the detector.
<figure><img src="image_56.png" /><figcaption>Figure 2: Comparison of the log-Mel-spectrogram (LMS) and the proposed TMGF feature using normal (N) and anomalous (A) sounds emitted from valve. Both sounds are selected from the target domain and have the same pattern. MF: modulation frequency, freq.: frequency.</figcaption></figure>

In the TMGF feature extraction, we used the gammatone filter-bank with $K = 65$ and $\alpha = 10$. For the mother wavelet $\psi(t)$, we set $n = 4$, $b = 1.019$, and $f_c = 600$ Hz. For the modulation filterbank, we used $M = 6$ and $f_1 = 2$ Hz. To decrease the dimension of TMGF feature, downsampling was conducted to decrease the temporal dimension to 1600 Hz. Finally, feature vectors with a fixed dimension of 390 were fed into the detector.

The model had four dense layers with 128 dimensions for the encoder, one bottleneck layer with eight dimensions, and four dense layers with 128 dimensions for the decoder. We trained the model for 100 epochs using the Adam optimizer [21] with a learning rate of 0.0001 and a batch size of 128. The anomaly scores were calculated by the averaged reconstruction error.

## 5. RESULTS

The overall results are shown in Table 1. This paper compares the results using our proposed method with that of the baseline method. The improved results are highlighted in the table. From these results, we can see that the LMS feature provides better performance in the source evaluations, but the performance significantly degrades in the target evaluation. The proposed method performs better in the target evaluation; even degradation occurs in the source evaluation. This is because of the TMGF feature can capture the sound variances in the time domain easily. It is sensitive to some background noises and irrelevant information. Therefore, the over-fitting problem in the training stage could be alleviated to some extent by using the proposed TMGF feature, hence improving the robustness of a trained ASD system.

The results of the TMGF feature achieve a much better performance in both the source and target evaluation in the valve. This is because timbral information captured by the TMGF feature, such as the sharpness and roughness, is useful for a learning system to find the variance of the 'click' sounds emitted from a valve. By using the TMGF feature, we improved the average arithmetic mean of AUC from 48.03% to 56.24% and the average harmonic mean of

<figure><img src="image_57.png" /><figcaption>Figure 3: Results of DCASE 2022 challenge using sounds recorded from the valve. Results from both the development (Dev.) dataset and evaluation (Eva.) dataset are depicted. Blue and red circles correspond to baseline and proposed systems, respectively.</figcaption></figure>

AUC from 42.53% to 51.56% in the target evaluation.

Figure 2 shows the differences between the LMS feature and the proposed TMGF using normal and anomalous sounds emitted from the valve. The pattern of these two sounds was consistent. TMGF can capture not only the frequency feature as 'click' sounds but also the time domain feature as timbre-related property.

The results of the DCASE2022 challenge using sounds recorded from the valve are shown in Fig. 3. Each dot corresponds to a different system in the challenge. As we can see, the TMGF can obtain competitive results in the valve even if a simple AE-based detector is used. The AE-based detector has to assume that the learned model cannot reconstruct sounds that are not used in training, that is, unknown anomalous sounds. This assumption is hard to satisfy because the training procedure does not involve anomalous sounds [8, 22]. Therefore, we believe that the performance can be further improved if a more reasonable detector can be used for the TMGF feature.

## 6. CONCLUSION

This paper presented a method that combines the temporal modulation features on the gammatone auditory filterbank (TMGF) with an AE-based detector in the ASD challenges. With the proposed method, this paper aims to make up for the deficiency of the log-Mel-spectrogram (LMS) feature and provide the TMGF feature, including more timbral information related to timbral attributes such as sharpness and roughness. Experimental results in the DCASE2022 Challenge Task 2 showed that the proposed method could provide a better ability for domain generalization. For machine sounds recorded from the valve, results from both the source and target evaluation have significant improvements compared with the baseline method. Future work will focus on investigating the model architecture of the ASD system to extract more discriminative information from the proposed TMGF feature.
7. REFERENCES

[1] K. Dohi, K. Imoto, N. Harada, D. Niizumi, Y. Koizumi, T. Nishida, H. Purohit, T. Endo, M. Yamamoto, and Y. Kawaguchi, "Description and discussion on dcase 2022 challenge task 2: Unsupervised anomalous sound detection for machine condition monitoring applying domain generalization techniques," *arXiv preprint arXiv:2206.05876*, 2022.

[2] K. Suefusa, T. Nishida, H. Purohit, R. Tanabe, T. Endo, and Y. Kawaguchi, "Anomalous sound detection based on interpolation deep neural network," *Proc. IEEE-ICASSP*, pp. 271–275, 2020.

[3] S. Kapka, "Id-conditioned auto-encoder for unsupervised anomaly detection," *arXiv preprint arXiv:2007.05314*, 2020.

[4] P. Daniluk, M. Goździewski, S. Kapka, and M. Kośmider, "Ensemble of auto-encoder based and wavenet like systems for unsupervised anomaly detection," *Challenge on Detection and Classification of Acoustic Scenes and Events (DCASE 2020 Challenge)*, Tech. Rep, 2020.

[5] T. Hayashi, T. Yoshimura, and Y. Adachi, "Conformer-based id-aware autoencoder for unsupervised anomalous sound detection," *DCASE2020 Challenge, Tech. Rep.*, 2020.

[6] K. Dohi, T. Endo, H. Purohit, R. Tanabe, and Y. Kawaguchi, "Flow-based self-supervised density estimation for anomalous sound detection," *Proc. IEEE-ICASSP*, pp. 336–340, 2021.

[7] S. Davis and P. Mermelstein, "Comparison of parametric representations for monosyllabic word recognition in continuously spoken sentences," *IEEE Transactions on Acoustics, Speech, and Signal processing*, vol. 28, no. 4, pp. 357–366, 1980.

[8] Y. Liu, J. Guan, Q. Zhu, and W. Wang, "Anomalous sound detection using spectral-temporal information fusion," *Proc. IEEE-ICASSP*, pp. 816–820, 2022.

[9] R. Sottek and K. Genuit, "Perception of roughness of time-variant sounds," *Journal of the Acoustical Society of America*, vol. 19, no. 1, p. 050195, 2013.

[10] Q.-H. Nguyen, K. Li, and M. Unoki, "Automatic mean opinion score estimation with temporal modulation features on gammatone filterbank for speech assessment," *Proc. INTERSPEECH*, 2022.

[11] H. Fastl and E. Zwicker, *Psychoacoustics - Facts and Models*. Berlin: Springer, 2007.

[12] B. C. Moore, *An introduction to the psychology of hearing*. Brill, 2012.

[13] A. Pearce, T. Brookes, and R. Mason, "Timbral attributes for sound effect library searching," in *Audio Engineering Society Conference: 2017 AES International Conference on Semantic Audio*, 2017.

[14] R. D. Patterson and J. Holdsworth, "A functional model of neural activity patterns and auditory images," *Advances in speech, hearing and language processing*, vol. 3, pp. 547–563, 1996.

[15] M. Unoki and M. Akagi, "A method of signal extraction from noisy signal based on auditory scene analysis," *Speech Communication*, vol. 27, no. 3-4, pp. 261–279, 1999.

[16] T. Dau, B. Kollmeier, and A. Kohlrausch, "Modeling auditory processing of amplitude modulation. i. detection and masking with narrow-band carriers," *Journal of the Acoustical Society of America*, vol. 102, no. 5, pp. 2892–2905, 1997.

[17] ——, "Modeling auditory processing of amplitude modulation. ii. spectral and temporal integration," *Journal of the Acoustical Society of America*, vol. 102, no. 5, pp. 2906–2919, 1997.

[18] K. Dohi, T. Nishida, H. Purohit, R. Tanabe, T. Endo, M. Yamamoto, Y. Nikaido, and Y. Kawaguchi, "Mimii dg: Sound dataset for malfunctioning industrial machine investigation and inspection for domain generalization task," *arXiv preprint arXiv:2205.13879*, 2022.

[19] N. Harada, D. Niizumi, D. Takeuchi, Y. Ohishi, M. Yasuda, and S. Saito, "Toyadmos2: Another dataset of miniature-machine operating sounds for anomalous sound detection under domain shift conditions," *arXiv preprint arXiv:2106.02369*, 2021.

[20] Y. Koizumi, Y. Kawaguchi, K. Imoto, T. Nakamura, Y. Nikaido, R. Tanabe, H. Purohit, K. Suefusa, T. Endo, M. Yasuda, et al., "Description and discussion on dcase2020 challenge task2: Unsupervised anomalous sound detection for machine condition monitoring," *arXiv preprint arXiv:2006.05822*, 2020.

[21] D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization," *arXiv preprint arXiv:1412.6980*, 2014.

[22] Y. Koizumi, S. Saito, H. Uematsu, Y. Kawachi, and N. Harada, "Unsupervised detection of anomalous sound based on deep learning and the neyman-pearson lemma," *IEEE/ACM Transactions on Audio, Speech, and Language Processing*, vol. 27, no. 1, pp. 212–224, 2018.
# FEW-SHOT BIOACOUSTIC EVENT DETECTION:
## ENHANCED CLASSIFIERS FOR PROTOTYPICAL NETWORKS

Ren Li<fnref n="1" />, Jinhua Liang<fnref n="1" />, Huy Phan<fnref n="1" />,<fnref n="2" />

## ABSTRACT

Few-shot learning has emerged as a novel approach to bioacoustic event detection since it is useful when training data is insufficient, and the cost of labelling data is high. In this paper, we explore the Prototypical Networks for developing a few-shot learning system to detect mammal and bird sounds from audio recordings. To enhance the deep networks, we use a ResNet-18 variant as the classifier, which can learn the embedding mapping better with stronger architecture. Another method is proposed to focus on domain shift problem during learning the embedding by taking advantage of autoencoders to learn the low-dimensional representations of input data. A reconstruction loss is added to the training loss to perform regularization. We also utilize various data augmentation techniques to boost the performance. Our proposed systems are evaluated on the validation set of DCASE 2022 task 5 and improve the F1-score from 29.59% to 47.88%.

**Index Terms**— Few-shot learning, sound event detection, Prototypical Networks, embedding space

In recent years, an increasing number of meta-learning approaches for few-shot learning have been proposed and applied in many domains, such as sound event detection, image classification and text classification [4]. Among them, the Prototypical Network (ProtoNet) proposed by Snell et al. [3] is simple in principle but effective in practice. The ProtoNet transforms the input into an embedding space where the embeddings are simply clustered to the nearest “prototype” [3]. Therefore, it is desirable for the deep networks to produce adequate embedded features and calculate a useful prototype for each class.

In this work, we propose two enhanced methods to build a stronger ProtoNet. The first method uses a ResNet-18 variant as the embedding features extractor, which is capable of learning and extracting more advanced features with deeper and wider residual networks. The second method merges the ideas from autoencoders and ProtoNet to learn low-dimensional representations and to preserve the information contained in original low-level features. We also apply various spectrogram augmentation techniques to increase the amount of training data for model generalization. Our proposed systems are evaluated on the validation set of DCASE 2022 task 5 and achieve the best F1-score of 47.88%.

## 1. INTRODUCTION

Bioacoustic event detection is the task of recognizing biological sound events present in a set of audio recordings and predicting their time boundaries [1]. This technology is now benefitting from the power of deep learning and becomes an effective way to gain information on the activities of animals that reflects human's impact on the environment [2]. Traditionally, researchers have conducted the work through manually labelling on huge datasets, which is consuming both in time and resources [2]. In addition, collecting labelled data in some certain animal sounds can be challenging, and the scarcity of supervised data can lead to poor generalization and overfitting problem [1].

To address the data scarcity and reduce the cost of labelling data, few-shot learning has been proposed; this approach learns a classifier that can recognize new classes with a limited amount of labelled data [3]. One applicable advantage of few-shot learning is its ability to gain experience from prior similar tasks, so few-shot learning can be characterized as a kind of meta-learning [4]. A meta-learning algorithm gains experience over a set of learning “episodes” and uses this experience to improve its future performance for a new task [4]. For N-way-K-shot classification, each episode includes N classes with K examples. For the DCASE 2022 task 5, the first K=5 events are used for the class of interest for each test file to detect all the events of this class in the rest of the recording [1].

## 2. RELATED WORK

The use of convolutional neural layers allows feature extractor to extract complex features that express the raw data in much more detail and learn representations more efficiently. However, as the layers get deeper, the learned features can deteriorate due to vanishing gradient, leading to performance deterioration [5]. For this reason, residual networks (ResNets) were proposed and widely applied in deep learning tasks [5]. Sharma et al. [6] used a pretrained ResNet-50 model for bird song classification, producing an accuracy of 97.1%, which was far superior to that of the VGG16 model. Soumya et al. [7] improved the ProtoNet using a customized ResNet as the feature embedding network for facial emotion recognition and proved its capability of extracting minute details.

In addition, most few-shot learning methods can suffer from domain shift problems during learning the embedding [8]. Since the embedding is only learned from the seen classes, when performing testing with the unseen classes, the embedding features are likely to be shifted due to the bias of the seen classes used for training [8]. Sometimes it can make the query data points far away from the correct corresponding unseen class prototypes, thus affecting the accuracy of the k-NN search. An effective Semantic Autoencoder (SAE) [8] is proposed to solve this problem by adding a reconstruction constraint to learn the low-dimensional representation. Moreover, Liu et al. [9] improved the SAE using graph structure and another L2-norm constraint, which preserves

<footnote n="1">¹ School of Electronic Engineering and Computer Science, Queen Mary University of London, United Kingdom</footnote>

<footnote n="2">² The Alan Turing Institute, United Kingdom</footnote>

<footnote>{ml20986, jinhua.liang, h.phan}@qmul.ac.uk</footnote>
the intrinsic data structure and has more discriminating power. Inspired by the ideas of above works, we enhance the ProtoNet for the few-shot bioacoustic event detection.

### 3. METHOD

This section introduces our methods developed upon the baseline system, including model design and data augmentation. First, we present a modified version of ResNet-18 used as a feature embedding network. After that, we describe how to combine autoencoders to perform data reconstruction from embeddings for better generalization to new unseen classes. Finally, we describe spectrogram augmentation techniques to boost the system's performance.

#### 3.1. ResNet-based prototypical network

For the task of detecting bird and mammal sounds, it is important for the embedding module to extract adequate features since the sound samples are often too short and imperceptible to detect and distinguish them. However, the embedding module of the original ProtoNet only consists of four Conv blocks. If we use the original ProtoNet with multiple Conv blocks to learn the sound's features, it is prone to encounter gradient vanishing problems, which reduces the quality of the embeddings.

We thus choose residual networks as the embedding encoder, which can avoid the vanishing gradients thanks to skip connections. The skip connections add the output from a preceding layer to a later layer, allowing information to get fast-forwarded and go deeper with less deterioration [6]. Another salient feature of ResNets is the use of batch normalization (BN) to normalize the input of the activation function of the previous layer, which helps mitigate the covariate shift problem [6].

Our implementation is based on the ResNet-18. We modify the network to obtain a less deep model which only has 3 residual blocks to fit the size of the features. Each residual block contains 3 convolution layers using a kernel size of 3 × 3, followed by a batch normalization and a Leaky ReLU activation. Importantly, a shortcut with a 1 × 1 convolutional layer is added over the 3 layers. The architecture of our residual network is shown in Table 1, while Table 2 shows the architecture of the original embedding module, which provides a comparison.

<table><caption>Table 1. Architecture of the presented residual network</caption><thead><tr><th colspan="2">Encoder</th><th colspan="2">Residual Block</th></tr><tr><th>Layers</th><th>Channels</th><th>Layers</th><th>Kernel</th></tr></thead><tbody><tr><td>Conv2D+ BN +ReLU</td><td>16</td><td>Conv2D+ BN +ReLU</td><td>3 × 3</td></tr><tr><td>Residual Block</td><td>64</td><td>Conv2D+ BN +ReLU</td><td>3 × 3</td></tr><tr><td>Residual Block</td><td>128</td><td>Conv2D+ BN</td><td>3 × 3</td></tr><tr><td>Residual Block</td><td>64</td><td>Shortcut: Conv 2D+BN</td><td>1 × 1</td></tr><tr><td>Adaptive</td><td>-</td><td>ReLU+MaxPool-ing+Dropout</td><td>1 × 1</td></tr><tr><td>AvgPooling+SoftMax</td><td>-</td><td></td><td></td></tr></tbody></table>

<table><caption>Table 2. Architecture of original encoder and Conv block</caption><thead><tr><th colspan="2">Encoder</th><th colspan="2">Conv Block</th></tr><tr><th>Layers</th><th>Channels</th><th>Layers</th><th>Kernel Size</th></tr></thead><tbody><tr><td>Conv Block</td><td>64</td><td>Conv2D</td><td>3 × 3</td></tr><tr><td>Conv Block</td><td>64</td><td>BatchNorm</td><td>-</td></tr><tr><td>Conv Block</td><td>64</td><td>ReLU</td><td>-</td></tr><tr><td>Conv Block</td><td>64</td><td>Max pool</td><td>2 × 2</td></tr></tbody></table>

#### 3.2. Combination of Autoencoder and ProtoNet

To overcome the domain shift problems described in Section 2, we enhance the network based on the encoder-decoder paradigm. The encoder compresses the spectrograms of input data into an embedding space while the decoder reconstructs the expected original input features from the embedding space [10]. The output of the decoder is then compared with the original input features. This additional reconstruction task imposes a new constraint in learning the input features that guarantees the embedding features preserve more distinctive information contained in the original input features [8]. Therefore, it is effective in mitigating the domain shift problem. Although the appearance of features changes from seen classes to unseen classes, the demand for a more truthful reconstruction of the input features is unchanged; thus, the embedding function is generalizable across seen and unseen domains [8].

Inspired by AutoProtoNet proposed by Sandoval-Segura et al. [11], we use the 4 original sequential convolution blocks for the encoder and 4 sequential transpose convolution blocks for the decoder. The transpose convolution blocks are utilized to reproduce the high-dimensional low-level features by deconvolutional operations. The detail of these blocks is displayed in Table 2. Each convolutional block consists of a Conv2D layer, a Batch Normalization, a ReLU activation and followed by a Max Pooling layer with pool-size of 2x2. Each transpose convolutional block is made up of a transpose layer, a Batch Normalization layer, and a Conv2D layer and followed by a ReLU activation.

<table><caption>Table 2. Components of Conv Block and Transpose Conv Block</caption><thead><tr><th colspan="2">Conv Block</th><th colspan="2">Transpose Conv Block</th></tr><tr><th>Layers</th><th>Kernel Size</th><th>Layers</th><th>Kernel Size</th></tr></thead><tbody><tr><td>Conv2D</td><td>3 × 3</td><td>Conv2D</td><td>2 × 2,</td></tr><tr><td>BatchNorm</td><td>-</td><td>Transpose</td><td>stride 2</td></tr><tr><td>ReLU</td><td>-</td><td>BatchNorm</td><td>-</td></tr><tr><td>Max pool</td><td>2 × 2</td><td>Conv2D</td><td>3 × 3</td></tr><tr><td></td><td></td><td>ReLU</td><td>-</td></tr></tbody></table>

<figure><img src="image_58.png" /><figcaption>Figure 1: Overview of the forward pass through the autoencoder model</figcaption></figure>

The training procedure of the autoencoder model is based upon the original training framework presented by Snell et al [3]. The main improvement is that we tailor the training loop with a reconstruction loss to regularize the embedding features and purpose it to preserve more useful details. The overview of the forward pass through the autoencoder is shown in Figure 1.

In the new training framework, the support data $x_{support}$ and query data $x_{query}$ are randomly sampled from the current episode in form of 5 classes with 5 examples. They are then passed through the encoder and decoder to produce a reconstruction set
$\hat{x}_i$. This reconstruction set is then compared with the original input features $x_i$ using mean square error (MSE) loss [11], defined as:

$$MSE = (x_i - \hat{x}_i)^2. \qquad (1)$$

The finetuning algorithm A computes a set of prototypes $p_k$ for each class k by computing the class-wise mean of embedded support examples and updates the model's parameters [3], and both are contained in $\theta_i$. Eq. (2) defines the computation of prototypes:

$$p_k = \frac{1}{|S_k|} \sum_{x \in S_k} F_{\theta}(x), \qquad (2)$$

where the $S_k$ denotes the set of support samples for class k and x denotes the embeddings of class k.

Given the Euclidean distance function d and a set of query sound samples, the ProtoNet produces a probability distribution over classes for a query sample x belonging to true class k by Eq. (3) [3]. Then the training proceeds by minimizing the negative log-likelihood (NLL) of the true class k by Eq. (4). Finally, the classification loss $L_C$ and the reconstruction loss $L_R$ are summed to jointly optimize the training.

$$p_{\theta}(y = k|x) = \frac{\exp(-d(F_{\theta}(x), p_k))}{\sum_{k'} \exp(-d(F_{\theta}(x), p_{k'})} \qquad (3)$$

$$L(\theta) = -\log p_{\theta}(y = k|x). \qquad (4)$$

### 3.3. Data augmentation

In order to increase the diversity of data and the generalization ability of the model, we use SpecAugment [12] as the data augmentation technique. It essentially consists of three transformations: time warping, frequency masking, and time masking. Specifically, they modify a spectrogram by warping it in the time direction with a distance factor, masking blocks of consecutive frequency channels, and masking blocks of time steps, respectively [12]. In our case, we warp the feature to the left by 0.5 s and mask one block of one frequency mel bin and one block of 10 time steps. We choose these values according to the size of the time-frequency representation, which is appropriate to produce the diversity of the training data. If the values are too large or small, the augmented features could be very different from the originals or not changing enough; thus, the model is unable to achieve improved performance.

<figure><img src="image_59.png" /><figcaption>Figure 2: Example of spectrogram augmentation: original spectrogram (left) and augmented spectrogram with SpecAugment (right).</figcaption></figure>

## 4. EXPERIMENTS

### 4.1. Dataset

The DCASE 2022 challenge provides a development set which is predefined as a training set and validation set. They were acquired from multiple bioacoustic sources, including sounds of worldwide birds, spotted hyenas, jackdaws, meerkats, and wetlands birds [1]. As a result, the sounds can be long or very short across the subsets; the sampling rate of each audio varies from 6 kHz to 44 kHz [1]. The training set consists of 174 audio recordings, 47 classes and 14,229 event instances. In addition, multi-class annotations are provided for the training set with positive, negative, and unknown; we only extracted and made use of the positive event instances for training. The validation set consists of 18 audio recordings, 5 classes and 1,077 positive event instances [1].

### 4.2. Data pre-processing

**Mel-spectrogram.** All audio files in both the training set and validation set were first resampled to a sampling rate of 22,050 Hz. The audio files were then transformed to Mel-spectrograms with 128 Mel bins using an FFT size of 1024 samples and a hop size of 256 samples. The *librosa* library was employed for this purpose. Afterwards, spectrogram images of size $F \times T$ where $F=17$ by $T=128$ were used as inputs.

**PCEN.** PCEN has been proposed to normalize a time-frequency representation by performing automatic gain control, followed by nonlinear compression [13]. Former research used PCEN to mitigate the effects of background noise, demonstrating its effectiveness as a preprocessing step prior to convolutional methods in sound event detection [13]. Bioacoustic data recorded in the wild often have multiple sound sources and uncleaned background. Therefore, we utilized PCEN to reduce noise presented in the Mel-spectrograms and improve robustness to channel distortion.

### 4.3. Training

Prototypical networks adopt an episodic training procedure where in each episode, a mini batch is randomly sampled from the training data [4]. A subset of mini batch was used as the support set and the remaining is used as query set. The models were trained with 2,000 episodes and 5 classes in each minibatch with the Adam optimizer and the learning rate of 0.001. Euclidean distance was selected as the metric that measures the distance between query samples to a prototype.

### 4.4. Post-processing

The preliminary experiments confirm that the task of detecting bioacoustics events from nature is challenging; most classification methods can produce a large number of false-positive predictions, substantially reducing the model's F1-score [14]. Therefore, we applied post-processing to the outputs to remove possible false positives. Specifically, we removed the predictions that were shorter than 20% of the average duration calculated by the first 5 shots for each audio file. It was because participants were expected to treat the task as a 5-shot setting.
<table><caption>Table 3. Comparison of models using different classifier and feature. The best results are highlighted in **boldface**.</caption><thead><tr><th colspan="3">Model components</th><th colspan="3">Validation set scores (%)</th><th colspan="3">Subset F1-score (%)</th></tr><tr><th>Exp No.</th><th>Classifier</th><th>Feature</th><th>F1-score</th><th>Precision</th><th>Recall</th><th>HB</th><th>ME</th><th>PB</th></tr></thead><tbody><tr><td>1</td><td>CNN (Baseline)</td><td>PCEN</td><td>29.59</td><td>36.34</td><td>24.96</td><td>/</td><td>/</td><td>/</td></tr><tr><td>2</td><td>ResNet</td><td>PCEN</td><td>45.64</td><td>48.34</td><td>43.22</td><td>50.00</td><td>57.14</td><td>26.18</td></tr><tr><td>3</td><td>Autoencoder</td><td>PCEN</td><td>37.94</td><td>38.95</td><td>36.97</td><td>44.53</td><td>52.05</td><td>25.68</td></tr><tr><td>4</td><td>CNN</td><td>PCEN+Augment</td><td>37.16</td><td>42.09</td><td>33.26</td><td>38.86</td><td><strong>72.01</strong></td><td>15.33</td></tr><tr><td>5</td><td>ResNet</td><td>PCEN+Augment</td><td><strong>47.88</strong></td><td><strong>52.11</strong></td><td><strong>44.30</strong></td><td><strong>53.45</strong></td><td><strong>50.98</strong></td><td><strong>17.65</strong></td></tr><tr><td>6</td><td>Autoencoder</td><td>PCEN+Augment</td><td><strong>47.61</strong></td><td><strong>50.18</strong></td><td><strong>45.34</strong></td><td><strong>52.68</strong></td><td><strong>53.10</strong></td><td><strong>22.44</strong></td></tr></tbody></table>

# 5. RESULTS AND DISCUSSIONS

We conducted several ablation experiments on the validation set to verify the effectiveness of the components and tricks in our proposed models. To further investigate the capabilities of our models, we computed the F1-score for three difference subsets. The experiments results are shown in Table 3. The brief information about each subset is as follows: the HB subset records the mosquito's events that are very long with low noise; the ME subset contains the sounds of meerkats that are short with low noise; the PB subset records the bird flight calls that are very short and unclear with high noise.

## 5.1. Effects of using ResNets

From the results of Experiments 1 and 2, it can be seen that the ResNet model outperforms the baseline CNN and achieves a noticeable improvement of over 15%. It is primarily due to the ResNet's deep architecture, which has many more parameters to capture the features better, allowing the learned features to fit the input data better. In addition, the residual networks make use of skip connections, enabling the model to carry gradients to a very deep layer. It also allows the model optimally tuning the number of the layers during training, so that the model parameters can be updated more optimally. However, since it adopts a complex networks architecture, the computation and memory cost increase intensively. To compare the model complexity, we measure the number of trainable parameters for different models as shown in Table 4. Compared to CNN with 112k parameters, the number of parameters for ResNet has grown significantly to 724k, and the number of parameters for the autoencoder is roughly twice that of the CNN.

<table><caption>Table 4. The number of parameters for different models</caption><thead><tr><th>Model</th><th>Parameters</th></tr></thead><tbody><tr><td>CNN (Baseline)</td><td>111,936</td></tr><tr><td>ResNet</td><td>724,096</td></tr><tr><td>Autoencoder</td><td>272,717</td></tr></tbody></table>

## 5.2. Effects of adding reconstruction loss

As shown by the results of Experiments 1, 2 and 3, the autoencoder model also improves the performance but with a slightly lower gain compared to that of the ResNet model. By adding a reconstruction loss, it is likely that the autoencoder model can learn the high-level low-dimensional representation and preserve more useful details, resulting in a better generalization to unseen classes for few-shot learning task. However, the reconstruction loss is served as a constraint during learning, so its influence is limited. Making an embedding classifier to learn the

representation of the input data in a fundamentally different way is more meaningful and challenging. That could be the reason why the ResNet model outperforms the autoencoder in this case.

## 5.3. Effects of data augmentation

Applying *SpecAugment* has been the popular choice for sound event detection. When training data is unbalanced and insufficient, it has shown to be effective. Overall, this technique also works well for our systems. Despite being augmented by *SpecAugment*, the performance of detecting the very short bird sounds in PB subset decreases. The reason could be that since the masking and warping were randomly applied from a uniform distribution over the value of factors, the blocks and warping steps could be too excessive in some cases. This could lead to some useful information in the minor sounds being masked, or even the augmented features became quite different from the originals, thus preventing the networks from learning. It gets even worse under high-noise conditions. Furthermore, the model trained on CNN and augmented data was observed to achieve much higher results for the ME subset than other approaches. This is because ResNet and autoencoder have many more parameters to optimize, likely leading to overfitting the training set. In contrast, the CNN with a simple architecture is just capable of generalizing better to ME data.

# 6. CONCLUSION

In this paper, we present two solutions to enhance Prototypical Networks for the task of bioacoustics sound event. We show a list of ablation studies and discussed the effects of each component or trick used in our systems. Overall, using ResNets and autoencoder (or construction loss) contribute to learn a more adequate embedding space and boost the model's performance. Joining with data augmentation techniques, our enhanced models achieve the best F1-score of 47.88%, which improves over the baseline by a large margin.

Our studies also imply that it is challenging to detect the very short and unclear bioacoustics sound events. This can be an important subject to be explored in the future work. Recent research show that *model adaptation* is an effective solution to improve the general robustness of SED method [15].

# 7. REFERENCES

[1] https://dcase.community/challenge2022/task-few-shot-bio-acoustic-event-detection

[2] C. Chalmers, P. Fergus, S. Wich and S. N. Longmore, "Modelling Animal Biodiversity Using Acoustic
Monitoring and Deep Learning," in *2021 International Joint Conference on Neural Networks (IJCNN)*, 2021, pp. 1-7.

[3] J. Snell, K. Swersky, R. Zemel, "Prototypical networks for few-shot learning," *Advances in neural information processing systems*. 2017, vol. 30.

[4] Z. Zhan, J. Zhou, and B. Xu, "Fabric defect classification using prototypical network of few-shot learning algorithm," *Computers in Industry*. 2022, vol. 138, p.103628.

[5] C. S. Wickramasinghe, D. L. Marino and M. Manic, "Res-Net Autoencoders for Unsupervised Feature Learning From High-Dimensional Data: Deep Models Resistant to Performance Degradation," *IEEE Access*, 2021 vol. 9, pp. 40511-40520.

[6] N. Sharma, A. Vijayeendra, V. Gopakumar, P. Patni and A. Bhat, "Automatic Identification of Bird Species using Audio/Video Processing," in *2022 International Conference for Advancement in Technology (ICONAT)*, 2022, pp. 1-6.

[7] K. Soumya and S. Palaniswamy, "Emotion Recognition from Partially Occluded Facial Images using Prototypical Networks," in *2020 2nd International Conference on Innovative Mechanisms for Industry Applications (ICIMIA)*, 2020, pp. 491-497.

[8] E. Kodirov, T. Xiang, and S. Gong, "Semantic autoencoder for zero-shot learning," in *Proceedings of the IEEE conference on computer vision and pattern recognition*, 2017, pp. 3174-3183.

[9] Y. Liu, D.Y. Xie, Q. Gao, J. Han, S. Wang, and X. Gao, "Graph and autoencoder based feature extraction for zero-shot learning." In *IJCAI*, 2019, vol.1(2), pp.6.

[10] S. Hong and S. -K. Song, "Kick: Shift-N-Overlap Cascades of Transposed Convolutional Layer for Better Autoencoding Reconstruction on Remote Sensing Imagery," *IEEE Access*, 2020, vol. 8, pp. 107244-107259.

[11] P. Sandoval-Segura, W. Lawson. AutoProtoNet: Interpretability for Prototypical Networks. arXiv preprint arXiv:2204.00929. 2022 Apr 2.

[12] D.S. Park, W. Chan, Y. Zhang, C.C. Chiu, B. Zoph, E.D. Cubuk, and Q.V. Le. Speaugment: A simple data augmentation method for automatic speech recognition. 2019, arXiv preprint arXiv:1904.08779.

[13] C. Ick and B. McFee, "Sound Event Detection in Urban Audio with Single and Multi-Rate Pcen," *ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, 2021, pp. 880-884.

[14] V. Morfi, I. Nolasco, V. Lostanlen, S. Singh, A. Strandburg-Peshkin, L.F. Gill, H. Pamula, D. Benvent, and D. Stowell. Few-Shot Bioacoustic Event Detection: A New Task at the DCASE 2021 Challenge. In DCASE, 2021, November, pp. 145-149.

[15] A. Mesaros, T. Heittola, T. Virtanen and M. D. Plumbley, "Sound Event Detection: A tutorial," in *IEEE Signal Processing Magazine*, 2021, vol. 38, pp. 67-83.
# A HYBRID SYSTEM OF SOUND EVENT DETECTION TRANSFORMER AND FRAME-WISE MODEL FOR DCASE 2022 TASK 4

Yiming Li¹,², Zhifang Guo¹,², Zhirong Ye¹,², Xiangdong Wang¹,†, Hong Liu¹, Yueliang Qian¹,
Rui Tao³, Long Yan³, Kazushige Ouchi³

¹ Beijing Key Laboratory of Mobile Computing and Pervasive Device,
Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China,
eamon.y.li@gmail.com, {guozhifang21s, yezhirong19s, xdwang, hliu, ylqian} @ict.ac.cn
² University of Chinese Academy of Sciences, Beijing, China
³ Toshiba China R&D Center, Beijing, China,
{taorui, yanlong} @toshiba.com.cn, kazushige.ouchi@toshiba.co.jp

## ABSTRACT

In this paper, we describe in detail our system for DCASE 2022 Task4. The system combines two considerably different models: an end-to-end Sound Event Detection Transformer (SEDT) and a frame-wise model, Metric Learning and Focal Loss CNN (MLFL-CNN). The former is an event-wise model which learns event-level representations and predicts sound event categories and boundaries directly, while the latter is based on the widely-adopted frame-classification scheme, under which each frame is classified into event categories and event boundaries are obtained by post-processing such as thresholding and smoothing. For SEDT, self-supervised pre-training using unlabeled data is applied, and semi-supervised learning is adopted by using an online teacher, which is updated from the student model using the Exponential Moving Average (EMA) strategy and generates reliable pseudo labels for weakly-labeled and unlabeled data. For the frame-wise model, the ICT-TOSHIBA system of DCASE 2021 Task 4 is used. Experimental results show that the hybrid system considerably outperforms either individual model, and achieves psds1 of 0.420 and psds2 of 0.783 on the validation set without external data. The code is available at https://github.com/965694547/Hybrid-system-of-frame-wise-model-and-SEDT.

**Index Terms—** Sound Event Detection Transformer, Online Pseudo-labelling, Hybrid System

## 1. INTRODUCTION

Sound Event Detection (SED) aims at identifying the category of foreground sound events as well as their corresponding onset and offset timestamps. Task4 of the DCASE challenge has been focusing on weakly supervised SED for several years. The DCASE 2022 Task4 [1] is a follow up of last year's challenge [2]. This year, in addition to exploring a heterogeneous development dataset containing unlabeled data, synthetic data and weakly labeled data, participants are allowed to incorporate external dataset or pre-trained embeddings. As last year, the SED system will be evaluated by Polyphonic Sound Detection Score (PSDS) [3] under two different real-life settings.

For weakly supervised SED, most existing works follow the Multiple Instance Learning (MIL) framework, and formulate SED as a seq2seq classification task. They usually design Convolutional

Neural Networks (CNNs) or Convolutional Recurrent Neural Networks (CRNNs) to obtain frame-level classification probability and then apply pooling mechanism to aggregate frame-level predictions to event-level results. However, such methods do not take sound events as a whole, which may ignore some global information, such as the correlation between frames or event duration. Recently, an event-wise model, namely SEDT, is proposed to handle such problems [4]. It models SED as a set prediction problem, which directly maps audio spectrogram to a set of candidate events, thus freeing SED models from trivial post-processing, namely frame-level thresholding or median filtering. Empirical study has shown that SEDT can achieve competitive performance compared with its frame-wise counterparts [4]. Moreover, we find that the two models can supplement each other, as they solve the SED task in different ways. Therefore, combining them together may be an intuitive approach to reach promising SED performance.

In this paper, we describe our system participating in DCASE 2022 Task 4. It is a combination of SEDT and frame-wise CNN model. For SEDT, specially-designed training formulas, including supervised learning, self-supervised learning and semi-supervised learning, are studied to help it learn from the heterogeneous development dataset. For frame-wise CNN model, metric learning is applied to narrow the domain gap between real and synthetic data, mean-teacher framework is implemented to provide supervision for unlabeled data and a tag-conditioned CNN model is used to generate final predictions based on audio tags. After obtaining each well-trained model, we explore the fusion strategy and post-processing methods of the ensemble model. By using the methods above, the hybrid system achieves competitive results on the validation dataset.

## 2. SEMI-SUPERVISED SEDT

### 2.1. Sound Event Detection Transformer

An overview of SEDT is shown in Fig. 1. It represents each sound event as $y_i = (c_i, b_i)$, where $c_i$ is the event category and $b_i = (m_i, l_i)$ denotes the event temporal boundary containing normalized event center $m_i$ and duration $l_i$, and directly seeks a mapping between input features and ground-truth events. Given the input spectrogram, the backbone CNN is adopted to extract its feature map, which is then added with one-dimensional positional encoding and fed into transformer encoder for further feature processing. The transformer decoder takes $N + 1$ learnable embeddings ($N$ event queries and 1 audio query) as input event query, where each of them
<figure><img src="image_60.png" /><figcaption>Figure 1: Overview of Sound Event Detection Transformer</figcaption></figure>

gathers information of a potential event from the encoder output feature via encoder-decoder cross-attention mechanism to generate event-level representations, while audio query gathers the whole audio information to generate clip-level representations. Finally, Feed Forward Networks (FFNs) are utilized to transform the event-level representations and clip-level representations from the decoder into event detection and audio tagging results, which are then fused together to get the candidate detection results. De-overlapping is implemented on overlapped candidate events of the same category. Specifically, it only reserves the events with the highest class probability. More details can be found in [4].

## 2.2. Supervised learning for SEDT

SEDT incorporates event-level loss and clip-level loss to optimize its event detection and audio tagging performance. For strongly-labeled data, both loss terms will be involved during the SEDT model training, while for weakly-labeled data, the event-level loss will be excluded since the strong annotations are not available.

**Event-level loss.** SEDT adopts a label assignment scheme before computing event-level loss: it tries to find a matching $\hat{\sigma}_i$ between each event prediction $\hat{y}_i$ and its corresponding ground-truth annotation $y_i$ through Hungarian algorithm, which is efficient for above bipartite graph matching problem. To equip SEDT with sound event classification and localization ability, the loss for SEDT supervised training is formulated as the weighted linear combination of localization loss $\mathcal{L}_{\text{loc}}$ and classification loss $\mathcal{L}_{\text{cls}}$. For each event prediction, the two loss functions are calculated as:

$$ \mathcal{L}_{\text{loc}} = \sum_{i}^{N} \left( \lambda_{\text{IOU}} \mathcal{L}_{\text{IOU}} (b_i, \hat{b}_{\hat{\sigma}(i)}) + \lambda_{\text{L1}} \left\| b_i - \hat{b}_{\hat{\sigma}(i)} \right\|_{1} \right) \quad (1) $$

$$ \mathcal{L}_{\text{cls}} = \frac{1}{N} \sum_{i=1}^{N} -\log \hat{p}_{\hat{\sigma}(i)}(c_i) \quad (2) $$

where $\lambda_{\text{IOU}}$ and $\lambda_{\text{L1}}$ are weights for Intersection Over Union (IOU) loss [5] and L1 loss.

**Clip-level loss.** The audio tagging loss is defined as the binary cross-entropy between the clip-level class label $l_{\text{tag}}$ and predicted audio tagging $y_{\text{tag}}$:

$$ \mathcal{L}_{\text{at}} = \text{BCE}(l_{\text{tag}}, y_{\text{tag}}) \quad (3) $$

## 2.3. Self-supervised learning for SEDT

To better use the unlabeled or external datasets, such as AudioSet and SINS, we adopt a self-supervised learning method to pre-train SEDT on unlabeled data, which is named as Self-supervised Pre-training SEDT (SP-SEDT). Specifically, we randomly crop spectrogram along the time axis to obtain several patches, and then pre-train the model to predict corresponding locations of the patches. To

preserve the category information in SP-SEDT, classification loss and feature reconstruction loss are also adopted as sub-objective terms. By means of such pre-text task, we hope that SEDT can localize sound event and maintain most category-related features at the same time. More details can be found in [6].

## 2.4. Semi-supervised learning for SEDT

Pseudo-labelling [7] is one of the mainstream approaches of semi-supervised learning. It requires a well-trained model to generate pseudo labels on unlabeled data, so that in the next stage, the converged model can be re-optimized on both labelled data and unlabeled data jointly. Based on that, we propose an improved pseudo-labelling method for the Semi-Supervised learning of SEDT (SS-SEDT). SS-SEDT splits the training process into two stages: the burn-in stage and the teacher-guided stage. In the burn-in stage, SEDT is simply trained on the labeled dataset to initialize the model. At the beginning of the teacher-guided stage, the initialized model is copied into two models (a student model and a teacher model), and then the teacher model generates pseudo labels on unlabeled data so that the student model can gain knowledge from both labeled data and unlabeled data. To guarantee the quality of the pseudo labels, we revisit the following off-the-shelf techniques, and apply them in the teacher-guided process. The detailed training process of teacher-guided stage is shown in Algorithm 1.

* **EMA:** Unlike previous methods supervised by offline pseudo labels, we resort to a progressing teacher model to generate pseudo labels. The teacher model is updated from the student model through EMA and thus can be viewed as implicit ensemble models and provide more reliable guidance. Notice that although the usage of EMA is similar to that in the mean-teacher framework, the proposed method is different since pseudo labels involved are hard ones and no consistency loss is adopted.

* **Asymmetric augmentation:** Asymmetric augmentation has been introduced into semi-supervised image recognition [8] and SED [9]. Inspired by that, we adopt similar idea in the teacher-guided stage, during which weakly-augmented (frequency mask and frequency shift) spectrograms are fed into the teacher model to get pseudo labels and the student model make predictions on the strongly augmented (frequency mask, frequency shift, time mask and gaussian noise) version of the same data batch.

* **Mixup [10]:** We mix labeled data with ground-truth and unlabeled data with pseudo annotations together, which is supposed to improve the model robustness to pseudo annotation noise and alleviate the overfitting problem in model training.

* **Focal loss [11]:** Focal loss is adopted to handle the unbalanced event categories in SED, without which the model may be overwhelmed by easily classified samples and produce biased out-
**Algorithm 1:** Pseudocode for teacher-guided stage

Require: $\mathcal{B}_L$ = labeled batch, $\mathcal{B}_U$ = unlabeled batch

Require: $S_\theta(x)$ = student model, $T_{\theta'}(x)$ = teacher model

Require: $A_w(x)$ = weak augmentation function

Require: $A_s(x)$ = strong augmentation function

Require: $\alpha$ = learning rate, $\gamma$ = EMA ratio

Require: $\mathcal{L}$ = loss function

Ensure : $\theta, \theta'$

1 for $i \to 1$ to $max$
2   $\mathcal{B}_L \cup \mathcal{B}_U \in \mathcal{B}$ do

3   $\mathcal{J}_{\text{sup}} \leftarrow \frac{1}{|\mathcal{B}_L|} \sum_{(x_i, y_i) \in \mathcal{B}_L} \mathcal{L}(S_\theta(A_w(x_i)), y_i)$;

4   $\mathbf{foreach} \ x_i \in \mathcal{B}_U \ \mathbf{do} \ y_i \leftarrow T_{\theta'}(A_w(x_i))$;

5   $\hat{\mathcal{B}} \leftarrow \text{Mixup}(\mathcal{B}_L, \mathcal{B}_U)$;

6   $\mathcal{J}_{\text{unsup}} \leftarrow \frac{1}{|\mathcal{B}|} \sum_{(\hat{x}_i, \hat{y}_i) \in \mathcal{B}} \mathcal{L}(S_\theta(A_s(\hat{x}_i)), \hat{y}_i)$;

7   $\theta \leftarrow \theta - \alpha(\frac{\partial \mathcal{J}_{\text{sup}}}{\partial \theta} + \frac{\partial \mathcal{J}_{\text{unsup}}}{\partial \theta})$;

8   $\theta' \leftarrow \gamma\theta' + (1 - \gamma)\theta$;

9   end

10 end

puts. It should be noted that focal loss is merely used in the teacher-guided stage, we believe such curriculum learning pattern may help our model learn from easy to difficult.

3. FRAME-WISE CNN MODEL

The pipeline of the frame-wise CNN model is illustrated in Fig. 2. At first, MLFL-CNN is preliminarily trained with weakly labeled data and strongly labeled synthetic data to acquire basic event detection and audio tagging ability. Then, it attaches pseudo strong labels to the weakly-labeled and unlabeled data, and the model is jointly trained with all these data in a self-training manner. Finally, the trained MLFL-CNN provides audio tags and strong pseudo labels for the weakly-labeled data and unlabeled data to train the tag-conditioned CNN [12], which gives the final SED results.

<figure><img src="image_61.png" /><figcaption>Figure 2: Overview of the frame-wise model</figcaption></figure>

The MLFL-CNN model contains three branches. The first branch is the embedding-level attention pooling branch based on the MIL framework, which is the same with [13]. The second branch is the sound event detection branch which is introduced to exploit the strong labels of synthetic data and uses focal loss as its supervision. The third branch is the domain adaptation branch which uses metric learning by inter-frame distance contrastive loss, more details of which can be found in [14]. During training process, the MLFL-CNN adopts the mean-teacher architecture and pseudo-labelling

framework simultaneously [15]. It combines clip-level loss (for weakly-labeled data), frame-level loss (for data with strong labels and pseudo strong labels), inter-frame distance contrastive loss (for real data and synthetic data), and consistency loss together. And the tag-conditioned CNN takes spectrograms and audio tags predicted by the MLFL-CNN as inputs, and uses the strong labels of synthetic data and pseudo strong labels of real data as ground-truth to train.

4. FUSION OF THE TWO MODELS

4.1. Preliminary: Class-specific PSDS

The essence of PSDS is to obtain a function $r(e)$ of effective TP rate (eTPR) changing with effective FP rate (eFPR), and calculate the integral of this function over $(0, e_{max})$, where $e_{max}$ represents the maximum value of eFPR value [3]. We notice that the original calculation of eTPR relies on two class-averaged indicators $\mu_{TP}$ and $\sigma_{TP}$. To decouple the eTPR according to the event category, we simply replace class-averaged indicators with class-dependent ones and finally redefine the PSDS value of given category as follows:

$$
\mu_{\text{TP},c} = r_{\text{TP},c} \qquad \sigma_{\text{TP},c} = r_{\text{TP},c} - \mu_{\text{TP},c} \tag{4}
$$

$$
\text{eTPR}_c : \quad r_c(e) \triangleq \mu_{\text{TP},c}(e) - \alpha_{\text{ST}} * \sigma_{\text{TP},c}(e) \quad (5)
$$

$$
\text{PSDS}_c \triangleq \frac{1}{e_{\text{max}}} \int_{0}^{e_{\text{max}}} r_c(e) \, de \qquad (6)
$$

where PSDS<sub>c</sub>, eTPR<sub>c</sub>, μ<sub>TP,c</sub> and σ<sub>TP,c</sub> are corresponding class-wise indicators for specific event class c.

4.2. Model fusion method

The core of model fusion is to calculate the class-wise fusion coefficients of each model's prediction during the evaluation stage. Assume that there are *N* models $m_i(i = 1, 2, ..., N)$, for each sound event class *c*, the PSDS of model $m_i$ on *c* is denoted as PSDS<sub>*i*,c</sub>. Then the fusion coefficient of model *i* on category *c* is defined as:

$$
w_{i,c} = \frac{\text{PSDS}_{i,c}}{\sum_{i=1}^{N} \text{PSDS}_{i,c}} \qquad (7)
$$

Therefore, for specific event category *c*, the final fusion probability $\hat{p}_c$ is formulated as the weighted linear combination of each model's predicted probability $p_{i,c}$:

$$
\hat{p}_c = \sum_{i=1}^{N} w_{i,c} * p_{i,c} \qquad (8)
$$

It is noteworthy that the above PSDS in Eq.(7) can be interpreted as PSDS1 or PSDS2 for this year's DCASE task4, so two different sets of parameters $w_{i,c}$ can be obtained on the development set and utilized to improve PSDS1 and PSDS2 respectively.

5. POST-PROCESSING

In order to reduce the noise in frame-level probability and make
sound events continuous, it is necessary to perform a smoothing
operation, such as mean filter or median filter, on the frame-level
probability. Currently, median filtering with a fixed window length
or with the average length of each event calculated on the develop-
ment set is generally utilized [16]. In this paper, we perform median
filtering and mean filtering (with larger window size) on frame-level
probabilities in sequence, and propose a method to search for opti-
mal class-wise window lengths on the development set.
<table><caption>Table 1: The PSDS on the validation set</caption><thead><tr><th>System</th><th>Extra data</th><th>PSDS1</th><th>PSDS2</th></tr></thead><tbody><tr><td>Baseline 1</td><td></td><td>0.336</td><td>0.536</td></tr><tr><td>Baseline 2</td><td>✓</td><td>0.351</td><td>0.552</td></tr><tr><td>System 1</td><td>✓</td><td>0.449</td><td>0.645</td></tr><tr><td>System 2</td><td>✓</td><td>0.115</td><td>0.816</td></tr><tr><td>System 3</td><td></td><td>0.420</td><td>0.618</td></tr><tr><td>System 4</td><td></td><td>0.099</td><td>0.783</td></tr></tbody></table>

<table><caption>Table 2: Ablation study on techniques in SS-SEDT</caption><thead><tr><th>MU</th><th>FL</th><th>AA</th><th>EMA</th><th>PSDS1</th><th>PSDS2</th></tr></thead><tbody><tr><td></td><td>✓</td><td>✓</td><td>✓</td><td>0.372</td><td>0.570</td></tr><tr><td>✓</td><td></td><td>✓</td><td>✓</td><td>0.349</td><td>0.540</td></tr><tr><td>✓</td><td>✓</td><td></td><td>✓</td><td>0.369</td><td>0.566</td></tr><tr><td>✓</td><td>✓</td><td>✓</td><td></td><td>0.357</td><td>0.538</td></tr><tr><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>0.388</td><td>0.573</td></tr></tbody></table>

Specifically, for a given event class *c*, we enumerate window length $wl_c$ from 1 to 500, and find the optimal length $wl_c^*$ to optimize PSDS1 and PSDS2 respectively:

$$wl_c^* = \arg \max_{wl_c} \frac{\text{PSDS}_c}{\text{PSDS}} \quad (9)$$

Finally, in the hybrid system, the event-level predictions of SEDT are firstly obtained in an end-to-end manner and then converted into frame-level probabilities, before being fused with frame-wise model and finally post-processed to get the ultimate results.

# 6. EXPERIMENT

## 6.1. Experiment Setup

For SEDT not using external data, we firstly pre-train it on unlabeled real subset (14412 clips), then simply train it on the weakly labeled training set (1578 clips) and synthetic 2019 subset (2045 clips) during burn-in stage, and finally use weakly labeled set, synthetic 2019 subset, synthetic 2021 subset (10000 clips), and unlabeled subset to conduct teacher-guided learning. For SEDT using external data, the two main differences compared to the above lie in 1) models are pre-trained on both unlabeled real subset and SINS subset (72894 clips), 2) an additional strongly labeled set (3470 clips) is further included in the teacher-guided stage. The detailed settings can be found in our repository<fnref n="1" />.

For frame-wise model not using external data, the training set contains the weakly labeled training set, the unlabeled training set, and synthetic 2021 subset. While for systems using external data, we add the same strongly labeled set taken from AudioSet to the original strongly labeled set. The detailed settings of training hyper-parameters and configurations can be found in [17].

## 6.2. Results of Submitted Systems

Table 1 shows the performance of our submitted systems, all of which are fused models of ensemble frame-wise CNN models and ensemble SEDT. Among them, system 1 and 2 incorporate external data, while system 3 and 4 do not. Besides, model fusion and

window tuning methods proposed in Section 4 and Section 5 are utilized in system 1, 3 to improve their PSDS1 and in system 2, 4 to improve their PSDS2 separately. As shown in Table 1, our hybrid systems outperform the official baseline considerably whatever the usage of external data. Moreover, our systems ranked 6th / 9th in the challenge respectively. While they are inferior to the winner models, our designed components are orthogonal to network architecture and data augmentation, which means that they may generalize to other models and bring about promising improvements.

<table><caption>Table 3: Ablation study on window tuning and model fusion</caption><thead><tr><th>Id</th><th>Model</th><th>MF</th><th>WT</th><th>PSDS1</th><th>PSDS2</th></tr></thead><tbody><tr><td>1</td><td>Single SEDT</td><td></td><td></td><td>0.415</td><td>0.582</td></tr><tr><td>2</td><td>Ensemble SEDT</td><td></td><td></td><td>0.431</td><td>0.607</td></tr><tr><td>3</td><td>Single frame</td><td></td><td></td><td>0.349</td><td>0.668</td></tr><tr><td>4</td><td>Ensemble frame</td><td></td><td></td><td>0.392</td><td>0.673</td></tr><tr><td>5</td><td>Hybrid system</td><td>✓</td><td></td><td>0.437</td><td>0.740</td></tr><tr><td>6</td><td>Hybrid system</td><td>✓</td><td>✓</td><td>0.449</td><td>0.816</td></tr></tbody></table>

## 6.3. Ablation Study

**Techniques in SS-SEDT.** To verify the effectiveness of techniques in SS-SEDT, we conduct ablation study using single SS-SEDT model without external data. Table 2 shows the results of models trained without specific technique, where MU, FL, AA denotes Mixup, Focal Loss, Asymmetric Augmentation mentioned in Section 2.4 respectively, and the model trained without AA means that the inputs of teacher and student model are both weakly augmented. It can be seen that all techniques can improve the performance of SS-SEDT and it can finally reach a PSDS1 of 0.388 and a PSDS2 of 0.573 while incorporating all techniques.

**Window tuning and model fusion.** To investigate the effects of window tuning and model fusion strategy, we conduct ablation study using SEDT and frame-wise model trained with external data. Table 3 compares the performance between models under different settings. In the above table, MF and WT denote Model Fusion and Window Tuning methods proposed in Section 4 and 5 respectively, and frame-wise model is abbreviated to “frame”. Among all these models, model 2 and 4 are ensemble models of top 1-5 single models, while hybrid system represents the fused model of ensemble SEDT and ensemble frame-wise model. By comparing model 1, 2 with model 3, 4, it is obvious that SEDT can achieve higher PSDS1 while frame-wise model is better at PSDS2. Moreover, by comparing model 5 with model 2, 4, we can see that while SEDT and frame-wise model have their own edges, they can complement each other, since the hybrid system achieve further improvements compared to single ensemble models. By comparing model 6 with model 5, the effectiveness of window tuning can be validated, since model 6 provides the best PSDS1 (0.449) and PSDS2 (0.816).

# 7. CONCLUSIONS

In this paper, we developed a framework to fuse the detection results of the frame-wise model and event-wise model, which leads to an improved PSDS1 of 0.420 and PSDS2 of 0.783 on the validation set compared to individual ensemble models.

# 8. ACKNOWLEDGMENT

This work was partly supported by the National Natural Science Foundation of China (62276250) and the Beijing Natural Science Foundation (Z190020).

<footnote n="1">¹https://github.com/Anaesthesiaye/sound_event_detectionTransformer</footnote>
## 9. REFERENCES

[1] https://dcase.community/challenge2022/
task-sound-event-detection-in-domestic-environments.

[2] https://dcase.community/challenge2021/
task-sound-event-detection-in-domestic-environments.

[3] Ç. Bilen, G. Ferroni, F. Tuveri, J. Azcarreta, and S. Krstulović, "A framework for the robust evaluation of sound event detection," in *ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. IEEE, 2020, pp. 61–65.

[4] Z. Ye, X. Wang, H. Liu, Y. Qian, R. Tao, L. Yan, and K. Ouchi, "Sound event detection transformer: An event-based end-to-end model for sound event detection," *arXiv preprint arXiv:2110.02011*, 2021.

[5] J. Yu, Y. Jiang, Z. Wang, Z. Cao, and T. Huang, "Unitbox: An advanced object detection network," in *Proceedings of the 24th ACM International Conference on Multimedia*, 2016, pp. 516–520.

[6] Z. Ye, X. Wang, H. Liu, Y. Qian, R. Tao, L. Yan, and K. Ouchi, "Sp-sedt: Self-supervised pre-training for sound event detection transformer," *arXiv preprint arXiv:2111.15222*, 2021.

[7] D.-H. Lee et al., "Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks," in *Workshop on Challenges in Representation Learning, ICML*, vol. 3, no. 2, 2013, p. 896.

[8] K. Sohn, D. Berthelot, N. Carlini, Z. Zhang, H. Zhang, C. A. Raffel, E. D. Cubuk, A. Kurakin, and C.-L. Li, "Fixmatch: Simplifying semi-supervised learning with consistency and confidence," *Advances in Neural Information Processing Systems*, vol. 33, pp. 596–608, 2020.

[9] T. K. Chan and C. S. Chin, "Multi-branch convolutional macaron net for sound event detection," *IEEE/ACM Transactions on Audio, Speech, and Language Processing*, vol. 29, pp. 2972–2985, 2021.

[10] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, "mixup: Beyond empirical risk minimization," in *International Conference on Learning Representations*, 2018.

[11] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, "Focal loss for dense object detection," in *Proceedings of the IEEE International Conference on Computer Vision*, 2017, pp. 2980–2988.

[12] J. Ebbers and R. Haeb-Umbach, "Forward-backward convolutional recurrent neural networks and tag-conditioned convolutional neural networks for weakly labeled semi-supervised sound event detection," in *Proceedings of the Detection and Classification of Acoustic Scenes and Events 2020 Workshop*, 2020, pp. 41–45.

[13] L. Lin, X. Wang, H. Liu, and Y. Qian, "Specialized decision surface and disentangled feature for weakly-supervised polyphonic sound event detection," *IEEE/ACM Transactions on Audio, Speech, and Language Processing*, vol. 28, pp. 1466–1478, 2020.

[14] Y. Huang, L. Lin, X. Wang, H. Liu, Y. Qian, M. Liu, and K. Ouchi, "Learning generic feature representation with synthetic data for weakly-supervised sound event detection by inter-frame distance loss," *arXiv preprint arXiv:2011.00695*, 2020.

[15] T. Rui, Y. Long, O. Kazushige, and X. Wang, "Couple learning for semi-supervised sound event detection," in *Proc. Inter-speech 2022*, 2022, pp. 2398–2402.

[16] L. Lin, X. Wang, H. Liu, and Y. Qian, "Guided learning convolution system for dcase 2019 task 4," in *Workshop on Detection and Classification of Acoustic Scenes and Events 2019*, 2019, p. 134.

[17] G. Tian, Y. Huang, Z. Ye, S. Ma, X. Wang, H. Liu, Y. Qian, R. Tao, L. Yan, K. Ouchi, and R. Ebbers, *Janek Haeb-Umbach*, "Sound event detection using metric learning and focal loss for dcase 2021 task 4," DCASE2021 Challenge, Tech. Rep., June 2021.
# LEVERAGING LABEL HIERARCHIES FOR FEW-SHOT EVERYDAY SOUND RECOGNITION

Jinhua Liang, Huy Phan, Emmanouil Benetos

Centre for Digital Music, Queen Mary University of London, United Kingdom
{jinhua.liang, h.phan, emmanouil.benetos}@qmul.ac.uk

## ABSTRACT

Everyday sounds cover a considerable range of sound categories in our daily life, yet for certain sound categories it is hard to collect sufficient data. Although existing works have applied few-shot learning paradigms to sound recognition successfully, most of them have not exploited the relationship between labels in audio taxonomies. This work adopts a hierarchical prototypical network to leverage the knowledge rooted in audio taxonomies. Specifically, a VGG-like convolutional neural network is used to extract acoustic features. Prototypical nodes are then calculated in each level of the tree structure. A multi-level loss is obtained by multiplying a weight decay with multiple losses. Experimental results demonstrate our hierarchical prototypical networks not only outperform prototypical networks with no hierarchy information but yield a better result than other state-of-the-art algorithms. Our code is available in: https://github.com/JinhuaLiang/HPNs_tagging

**Index Terms—** Everyday sound recognition, few shot learning, hierarchical prototypical network

## 1. INTRODUCTION

Everyday sound recognition (or audio tagging) is to classify the types of environmental sound events in a recording or online stream, which involves many potential scenarios such as hearing aids [1], smart cities [2], and advanced healthcare [3]. In the past decades, a great amount of deep learning methods have emerged [4, 5] exploring how to boost audio networks' performance using large-scale datasets [6, 7]. While many works turned to focus on some more practical scenarios, such as mismatched domains [8], weakly supervised learning [9], and noisy labels [10], most of these methods are still restricted by the size of available datasets. This is a practical problem in the field of everyday sound recognition as it usually takes annotators more effort to mark the categories in a recording. In addition, everyday sounds cover thousands of categories, which makes it impossible to collect sufficient instances per class for supervised learning. This is thus how few-shot learning comes into the picture.

Inspired by the human ability to learn novel items with just a few examples, few-shot learning aims to capture the pattern of an unseen category using a handful of instances [11]. A typical few-shot learning framework is depicted as an *N*-way *K*-shot problem where there are *N* classes in a task and each class contains *K* instances for training. Currently only a few studies have attempted to apply few-shot learning to environmental sound recognition tasks. Although these works pioneered few-shot audio recognition, most of them were restricted to implementing off-the-shelf few-shot learning methods from other fields explicitly, which ignores exploiting the relationship between labels in audio taxonomy.

This work is motivated by the fact that we humans learn unseen concepts not only by observing their own features, but also by connecting them to existing knowledge. We thus assume that leveraging a priori knowledge helps a model to learn an unseen category with a few examples. Based on this assumption, this paper applies hierarchical prototypical networks (HPNs) to leverage the audio taxonomy knowledge drawn from the taxonomy of the dataset. Specifically, a few-shot classification problem is considered as a multi-task classification problem where both ancestor classes and descendant classes are used in separate classification tasks. Furthermore, prototypical networks are adopted as classifiers by measuring distance between query points and prototypes in the embedding space. Experimental results on the ESC-50 dataset [12] show that our HPNs yield a superior performance over prototypical networks with no hierarchy knowledge and outperform other state-of-the-art models.

The contributions of our work are three-fold:

i) Several state-of-the-art few-shot learning algorithms are benchmarked for generic everyday sound recognition. Different experimental setups are carried out to investigate the impact of data splits on model evaluation.

ii) A hierarchical prototypical network is proposed and applied to leverage a priori knowledge of sound event taxonomy by taking samples' ancestor classes into consideration.

iii) The impact of data splits on overall performance is investigated. The code is also released to benchmark state-of-the-art few-shot algorithms and to set up an evaluation environment on the ESC-50 dataset.

The remainder of this paper is organised as follows. Section 2 briefly summarises work related to our research and Section 3 introduces our proposed hierarchical prototypical network and the implementation details. In the Section 4, experimental results are discussed to demonstrate the superior performance of our network compared with other few-shot methods. Discrepancy in performance is then discussed among different data splits. Section 5 concludes the work and points out directions for future work.

## 2. RELATED WORK

### 2.1. Few-shot learning for everyday sound recognition

Few shot learning aims to use a limited amount of labeled examples to train a model that can be generalised to unseen categories easily. Suppose $C_{base}$ and $C_{novel}$ are two non-overlapping label sets (or splits) drawn from the whole label set $C$. The task is to train a classifier $f$ with labelled samples of classes from $C_{base}$ and to evaluate $f$ on samples of classes belonging to $C_{novel}$. Transfer learning [13] and meta learning [14, 15] are two of the most frequently used techniques. On the one hand, transfer learning strategies train $f$
with abundant data from C<sub>base</sub> and then fine-tune *f* through some iterations using a limited amount of data from C<sub>novel</sub>. On the other hand, meta learning (or *episodic learning*) strategies [16, 15, 14] update model parameters through a series of independent tasks in the training process. A task herein is formed by drawing *N* classes from C<sub>base</sub> each of which contains *K* “training” data and *Q* “test” data. To differentiate the above “training” and “test” data with the conventional terms, we refer to them as *support* and *query* data instead. The model is then trained to predict the categories of query data out of *N* classes by offering support data. The underlying assumption of episodic learning is aligning the training process with the evaluation benefits a model’s ability to generalise to novel categories. However, there exist some works [13] stating that transfer learning can have a competitive performance as well. Therefore, it still remains an open problem to work out the best practice for few-shot learning.

One of the biggest challenges for everyday sound recognition nowadays is that it covers thousands of sound classes, while only hundreds of them are available in the existing large-scale datasets [6]. There are some attempts regarding how to apply few-shot learning in the everyday sound domain. Shi et al. [17] implemented several few-shot learning algorithms on a subset of the AudioSet dataset and demonstrated the advantage of meta-learning methods in the audio domain. Wang et al. [18] curated a synthesized audio dataset for few-shot audio recognition based on FSD50K [7] and compared the state-of-the-art by controlling annotated samples, polyphony levels, and signal-to-noise ratio (SNR). Heggan et al. [19] attempted to setup a benchmark for few-shot learning techniques in audio domains. In addition to generic everyday sound recognition, the Detection and Classification of Acoustic Scenes and Events (DCASE) challenge<fnref n="1" /> has been holding a task on fine-grained few-shot learning in the past two years which attracts increasing amount of attention. Although the previous works introduced few-shot learning paradigms to everyday sounds successfully, most of them ignored leveraging the relationship between labels in the audio taxonomy. Different from those works, this paper integrates audio taxonomy knowledge within existing few-shot algorithms.

## 2.2. Knowledge-based few-shot learning

Knowledge-based learning incorporates label taxonomy knowledge into the supervised learning [20]. There are some knowledge-based few-shot methods in few-shot scenarios [21, 22]. Peng et al. proposed a Knowledge Transfer Network (KTN) to incorporate visual features and semantic information for image recognition [21]. They used two independent classifiers to capture visual patterns and to conduct knowledge inference, followed by an integration network to merge them together. Due to the difference between sound and image, however, the definition of sound classes is more abstract and obscure for annotators compared with images. Garcia et al. designed hierarchical prototypical networks for music instrument recognition [22]. They aggregated classes according to a predefined instrument hierarchy and calculated the hierarchical loss by adding a weight decay to each level in the tree structure. Compared with music instrument classification, everyday sound recognition cannot be sorted into a tree structure by their physical properties merely. Some intermediate classes, such as “domestic sound” and “wild animal”, are connected with psycho-acoustics directly, which makes the classification task even more complicated. Inspired by [22],

this paper applies hierarchical prototypical networks to leverage the audio taxonomy knowledge derived from the dataset. Compared with their original implementation, our proposed models lower the limit on the number of prototypes for ancient prototypes generation, which encourages the model to use the audio taxonomy knowledge in more circumstances.

# 3. HIERARCHICAL PROTOTYPICAL NETWORKS

## 3.1. Prototypical networks

Prototypical networks were proposed in [16] to train a few-shot model using a series of independent tasks. Prototypical networks learn an embedding space where classification is performed by computing distances between each query sample and prototypes. Suppose a support set of $N \times K$ examples be $S = \{(x_1, y_2), \dots, (x_{N \times K}, y_{N \times K})\}$ where each $x_i \in \mathbb{R}^D$ is the $D$-dimensional feature vector of an example obtained from an encoder $f$ and $y_i \in \{1, \dots, K\}$ be the corresponding label. A prototype could be calculated by averaging the embeddings of support samples belonging to its class:

$$c_k = \frac{1}{|S_k|} \sum_{(x_i, y_i) \in S_k} f_{\phi}(x_i) \quad (1)$$

where $S_k$ denotes the set of examples whose ground truth is class $k$ and $\phi$ are the learnable parameters in the encoder $f$.

Prototypical networks then produce a distribution over classes for a query embedding $x$ using a softmax function over distances to the prototypes:

$$q_{\phi}(y = k|x) = \frac{\exp(-\operatorname{dist}(f_{\phi}(x), c_k))}{\sum_{m} \exp(-\operatorname{dist}(f_{\phi}(x), c_m))} \quad (2)$$

We applied the Euclidean distance as the distance function in our experiments. The probability distribution over classes is used to calculate cross-entropy loss.

$$L_{CE} = - \sum p(x) \log q(x) \quad (3)$$

where $p, q$ are distributions of ground truth and predictions.

## 3.2. Hierarchical prototypical networks

Based on prototypical networks in Section 3.1, we devise hierarchical prototypical networks (HPNs) to incorporate the audio taxonomy knowledge into the training process. As shown in Figure 1, each level of the tree structure is treated as an independent multi-class classification task in the training stage. We thus build a HPN to extract the acoustic features with a shared encoder $f$ and predict multiple labels corresponding to each level. Similar to (1), the acoustic features of support samples are used to calculate prototypes of the bottom level as:

$$c_k^{(0)} = \frac{1}{|S_k|} \sum_{(x_i, y_i) \in S_k} f_{\phi}(x_i) \quad (4)$$

To generate prototypical nodes of a higher level in the HPN, prototypes of the lower level are clustered together as per their parent level and aggregated to obtain the prototypes of a higher level:

$$c_j^{(h)} = \frac{1}{|C_k^{(h)}|} \sum_{c_j^{(h)} \in C_k^{(h)}} c_j^{(h-1)} \quad (5)$$

<footnote n="1">¹https://dcase.community/</footnote>
<figure><img src="image_62.png" /><figcaption>Figure 1: Illustration of training a hierarchical prototypical network in a multi-task scenario. The model learns to solve a 4-way 5-shot problem on the bottom level while trying to conduct a 2-way classification on a higher level.</figcaption></figure>

<table><caption>Table 1: The architecture of the designed encoder.</caption><thead><tr><td>Block name</td><td>Filter size</td><td>Output shape</td></tr></thead><tbody><tr><td>Conv block 1</td><td>3 × 3@64</td><td>(64, 32, 215)</td></tr><tr><td>Conv block 2</td><td>3 × 3@128</td><td>(128, 16, 107)</td></tr><tr><td>Conv block 3</td><td>3 × 3@256</td><td>(256, 8, 53)</td></tr><tr><td>Conv block 4</td><td>3 × 3@512</td><td>(512, )</td></tr><tr><td>Fully connected layer</td><td>256</td><td>(256,)</td></tr></tbody></table>

where $C_j^{(h)}$ is a set of prototypes belonging to the class j at the h-th level in the taxonomy. The HPN iterates the clustering and aggregation process until the prototypes of the highest level are calculated.

Different from the original hierarchical network [22] which ignores an ancestor node if the number of its child nodes is less than two, HPN takes this ancestor class into consideration to fully exploit the hierarchical information. We believe this can leverage the audio taxonomy knowledge even better. The Euclidean distance between the query embedding and prototypes in each level is then calculated and used for classification. The evaluation process is similar to the training except only the classification task of the bottom level in HPNs will be taken into consideration for a fair comparison.

Following the architecture of VGGNet [23], we design a convolutional neural network containing 8 convolutional layers as the backbone of our HPNs, as shown in Table 1. Each block consists of two identical convolutional filters. Except for the last block, a max pooling layer with strides equal to 2 is appended to the block. A global pooling operation is used in the last block. Finally, the network outputs audio embedding sized 256.

### 3.3. Structural loss

Let $h$ be the hierarchical level of a node in the tree structure, we can calculate the hierarchical loss [22] by using the cross-entropy loss function as follows:

$$L_{hierarchical} = \sum_{h=0}^{H} e^{\alpha h} L_{CE}^{(h)} \quad (6)$$

where $L_{CE}^{(h)}$ is the cross-entropy loss in the level $h$, $H$ is the height of the taxonomy, and $\alpha$ is a hyper-parameter to control the loss de-

cay of each level with respect to their height in the hierarchy. We set $\alpha$ equal to 1 in all experiments.

## 4. EVALUATION

### 4.1. Dataset

<table><caption>Table 2: Parent classes and examples of the child classes in ESC-50</caption><thead><tr><td>Parent class</td><td>Examples of child classes</td></tr></thead><tbody><tr><td>Animals</td><td>Dogs, Rooster, Pig, Cow, ...</td></tr><tr><td>Natural & water soundscapes</td><td>Rain, Sea waves, Crickets, ...</td></tr><tr><td>Human, non-speech sounds</td><td>Crying baby, Sneezing, ...</td></tr><tr><td>Interior/domestic sounds</td><td>Door knock, Clock tick, ...</td></tr><tr><td>Exterior/urban noises</td><td>Engine, Chainsaw, Siren, ...</td></tr></tbody></table>

The few-shot learning methods are evaluated on the ESC-50 dataset [12]. ESC-50² is a collection of sound events which consists of 2000 5-second recordings in total. These recordings are assigned one label out of 50 child classes. Table 2 shows the tree structure of ESC-50. It can be observed that these child classes are loosely arranged into 5 parent categories: “Animals”, “Natural & water soundscapes”, “Human, non-speech sounds”, “Human, non-speech sounds”, “Interior/domestic sounds”, “Exterior/urban noises”. It should be noted that neither the original nor our experimental setting uses the parent classes in the evaluation stage.

### 4.2. Comparative methods

In addition to prototypical networks, we also use a selection of few-shot algorithms for comparison [13, 14]. The transfer learning method in [13] trains an encoder from scratch using the base split. It then applies this trained model with the best validation performance for few-shot evaluation using the novel split. Another work in [14] used matching networks to calculate an attention matrix between query samples and support samples. The attention matrix is multiplied by matrix of support labels to obtain the logs of the query ones.

Except matching networks, we apply the identical encoder as described in Table 1. For the matching network we build a model following the implementation in [14]. As all of the comparative methods are metric-based, we apply the Euclidean distance to assess the similarity between two samples.

### 4.3. Evaluation metrics

As with Sect.6.5 in [24], this work uses accuracy to measure the ability of a classifier to make the correct decisions. The $F_1$ score is calculated to trade off between the ability to make correct decisions and to retrieve positive samples.

### 4.4. Experiment setup

Inputs for the few-shot methods are log Mel spectrograms. We set the sampling rate to 44100 Hz. The window length is 1024 sample points (roughly 20ms) with 50% overlap, and the number of Mel bank filters is 64. We finally get spectrograms sized 431 × 64. Before forwarding the extracted feature into network, frequency normalisation is applied along each Mel bin.

<footnote n="2">²Dataset available in https://github.com/karolpiczak/ESC-50</footnote>
Following [22], in addition to a typical 5-way 5-shot problem, all few-shot learning methods were evaluated in a 12-way classification as well. For both experiments, each task (or episode) contains 5 support samples and 5 query samples. i.e. $K = 5$ and $Q = 5$. There are 32 episodes for each epoch. For a fair comparison, all models are trained through 100 epochs using the Adam optimiser with learning rate equal to 0.0001. 5-fold cross-validation was used throughout experiments. We randomly split the label set into two non-overlapping datasets, train and evaluation sets, as per the ratio 7:3. Following [22] we ensure ratios of children classes under same parent classes are identical between the train and evaluation process. We refer to this data split method as *uniform split* hereafter. It should be noted that the split label sets are changed with respect to different folds in the cross-validation, but all methods in the same folds share the same sets of train and evaluation.

## 4.5. Experimental results

<table><caption>Table 3: The performance of 12-way 5-shot learning methods on the ESC-50 dataset. The best one is highlighted in **bold**.</caption><thead><tr><th></th><th>Accuracy</th><th>F<sub>1</sub></th></tr></thead><tbody><tr><td>Transfer Learning [13]</td><td>72.90%</td><td>72.87%</td></tr><tr><td>Proto [16]</td><td>77.70%</td><td>77.52%</td></tr><tr><td>Matching [14]</td><td>71.81%</td><td>71.75%</td></tr><tr><td>HPN (ours)</td><td><strong>78.65%</strong></td><td><strong>78.51%</strong></td></tr></tbody></table>

<table><caption>Table 4: The performance of three episodic learning methods for 5-way 5-shot problems on the ESC-50 dataset. The best one is highlighted in **bold**.</caption><thead><tr><th></th><th>Accuracy</th><th>F<sub>1</sub></th></tr></thead><tbody><tr><td>Proto [16]</td><td>88.18%</td><td>88.18%</td></tr><tr><td>Matching [14]</td><td>86.83%</td><td>86.83%</td></tr><tr><td>HPN (ours)</td><td><strong>88.90%</strong></td><td><strong>88.88%</strong></td></tr></tbody></table>

Table 3 compares four few-shot learning methods in terms of accuracy and F1-score. Our hierarchical prototypical network yields the best performance among the transfer learning method, prototypical network (Proto), and matching network. The hierarchical prototypical network outperforms the best baseline (i.e. the prototypical network) by 0.95% and 0.99% in terms of accuracy and F1-score, respectively. This indicates that audio taxonomy knowledge can help an encoder to learn a better embedding space. Table 4 compares three episodic learning methods for 5-way 5-shot problems. Our HPNs can still yield a superior performance than prototypical networks and matching networks.

## 4.6. Impact of data split

In order to investigate the impact of data splits on overall performance, we compare different data split methods using the same few-shot model. Table 5 shows results of baseline prototypical networks with three data split methods. The *random split* method herein is to select 15 classes as novel classes randomly, so that the ratios of children classes under the same parent ones are not fixed. The *parent split* method is to select all child classes under the same parent category and fill this selected class set with the classes belonging to the

<table><caption>Table 5: The performance of prototypical networks with different data splits on the ESC-50 dataset.</caption><thead><tr><th></th><th>acc</th><th>F<sub>1</sub></th></tr></thead><tbody><tr><td>Random</td><td>74.59%</td><td>74.59%</td></tr><tr><td>Parent</td><td>73.35%</td><td>73.35%</td></tr><tr><td>Uniform</td><td>77.70%</td><td>77.52%</td></tr></tbody></table>

rest parent categories. In this way, we make the number of classes in the selected set obtained by parent split method identical to the one by other methods. The descending order of the performance of three data split methods is “Uniform” > “Random” > “Parent”. This observation adheres to our intuition that the performance on evaluation drops as the gap between the distribution of a base split and a novel split gets bigger. It also demonstrates that the characteristics between child classes from the same ancestor class are more similar than those from different classes, suggesting the importance of audio taxonomy knowledge in classification tasks.

## 5. CONCLUSION AND FUTURE WORK

This work designs a hierarchical prototypical network for everyday sound recognition. The network extracts acoustic features and generates prototypical nodes corresponding to multiple levels in the tree structure. Distances between query samples and prototypical nodes in each level are then calculated and used for classification separately. Compared with prototypical networks with no hierarchy information, our model achieved a better performance in terms of accuracy and $F_1$-score.

Although hierarchical prototypical networks proves that it is promising to incorporate the audio taxonomy knowledge in few-shot everyday sound recognition, it still suffers from some limitations. First, HPNs cannot be applied to some datasets where an explicit taxonomy is not available. Second, some taxonomies are too complex to assign a hierarchical level to each label (e.g., a label having multiple paths to the root). In the future we plan to extend HPNs to a more complicated large-scale dataset. In addition, it is also intriguing to explore knowledge-based methods without an explicit taxonomy.

## 6. ACKNOWLEDGEMENT

This work was supported by the Engineering and Physical Sciences Research Council [grant number EP/T518086/1]. The research utilised Queen Mary’s Apocrita HPC facility, supported by QMUL Research-IT, http://doi.org/10.5281/zenodo.438045.

## 7. REFERENCES

[1] X. Fan, T. Sun, W. Chen, and Q. Fan, “Deep neural network based environment sound classification and its implementation on hearing aid app,” *Measurement*, vol. 159, p. 107790, July 2020. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0263224120303286

[2] T. Spadini, D. L. d. O. Silva, and R. Suyama, “Sound Event Recognition in a Smart City Surveillance Context,” *arXiv:1910.12369 [cs, eess, stat]*, Feb. 2020, arXiv:
1910.12369. [Online]. Available: http://arxiv.org/abs/1910.12369

[3] B. W. Schuller, D. M. Schuller, K. Qian, J. Liu, H. Zheng, and X. Li, "COVID-19 and Computer Audition: An Overview on What Speech & Sound Analysis Could Contribute in the SARS-CoV-2 Corona Crisis," *Frontiers in digital health*, vol. 3, p. 14, 2021, publisher: Frontiers.

[4] Y. Gong, Y.-A. Chung, and J. Glass, "AST: Audio Spectrogram Transformer," in *Proc. Interspeech 2021*, 2021, pp. 571–575.

[5] T. Zhang, J. Liang, and B. Ding, "Acoustic scene classification using deep CNN with fine-resolution feature," *Expert Systems with Applications*, vol. 143, p. 113067, Apr. 2020. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0957417419307845

[6] J. F. Gemmeke, D. P. W. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, "Audio Set: An ontology and human-labeled dataset for audio events," in *2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. New Orleans, LA: IEEE, Mar. 2017, pp. 776–780. [Online]. Available: http://ieeexplore.ieee.org/document/7952261/

[7] E. Fonseca, X. Favory, J. Pons, F. Font, and X. Serra, "FSD50K: An Open Dataset of Human-Labeled Sound Events," *IEEE/ACM Transactions on Audio, Speech, and Language Processing*, vol. 30, pp. 829–852, 2022, conference Name: IEEE/ACM Transactions on Audio, Speech, and Language Processing.

[8] B. Kim, S. Yang, J. Kim, and S. Chang, "QTI Submission to DCASE 2021: Residual Normalization for Device-Imbalanced Acoustic Scene Classification with Efficient Design," DCASE2021 Challenge, Tech. Rep., June 2021.

[9] Y. Xu, Q. Kong, W. Wang, and M. D. Plumbley, "Large-Scale Weakly Supervised Audio Classification Using Gated Convolutional Neural Network," in *2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. Calgary, AB: IEEE, Apr. 2018, pp. 121–125. [Online]. Available: https://ieeexplore.ieee.org/document/8461975/

[10] T. Iqbal, Y. Cao, A. Bailey, M. D. Plumbley, and W. Wang, "ARCA23K: An Audio Dataset for Investigating Open-Set Label Noise," in *Proceedings of the 6th Detection and Classification of Acoustic Scenes and Events 2021 Workshop (DCASE2021)*, Barcelona, Spain, Nov. 2021, pp. 201–205.

[11] X. Li, Z. Sun, J.-H. Xue, and Z. Ma, "A concise review of recent few-shot meta-learning methods," *Neurocomputing*, vol. 456, pp. 463–468, Oct. 2021. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0925231220316222

[12] K. J. Piczak, "ESC: Dataset for Environmental Sound Classification," in *Proceedings of the 23rd ACM international conference on Multimedia*. Brisbane Australia: ACM, Oct. 2015, pp. 1015–1018. [Online]. Available: https://dl.acm.org/doi/10.1145/2733373.2806390

[13] Y. Tian, Y. Wang, D. Krishnan, J. B. Tenenbaum, and P. Isola, "Rethinking Few-Shot Image Classification: A Good Embedding is All You Need?" in *Computer Vision – ECCV 2020*, ser.

Lecture Notes in Computer Science, A. Vedaldi, H. Bischof, T. Brox, and J.-M. Frahm, Eds. Cham: Springer International Publishing, 2020, pp. 266–282.

[14] O. Vinyals, C. Blundell, T. Lillicrap, k. kavukcuoglu, and D. Wierstra, "Matching Networks for One Shot Learning," in *Advances in Neural Information Processing Systems*, vol. 29. Curran Associates, Inc., 2016. [Online]. Available: https://proceedings.neurips.cc/paper/2016/hash/90e1357833654983612fb05e3ec9148c-Abstract.html

[15] C. Finn, P. Abbeel, and S. Levine, "Model-agnostic meta-learning for fast adaptation of deep networks," in *International Conference on Machine Learning*. PMLR, 2017, pp. 1126–1135.

[16] J. Snell, K. Swersky, and R. Zemel, "Prototypical Networks for Few-shot Learning," *Advances in Neural Information Processing Systems*, vol. 30, pp. 4077–4087, 2017.

[17] B. Shi, M. Sun, K. C. Puvvada, C.-C. Kao, S. Matsoukas, and C. Wang, "Few-Shot Acoustic Event Detection Via Meta Learning," in *ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, 2020, pp. 76–80, iSSN: 2379-190X.

[18] Y. Wang, N. J. Bryan, J. Salamon, M. Cartwright, and J. P. Bello, "Who Calls The Shots? Rethinking Few-Shot Learning for Audio," in *2021 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA)*. IEEE, 2021, pp. 36–40.

[19] C. Heggan, S. Budgett, T. Hospedales, and M. Yaghoobi, "MetaAudio: A Few-Shot Audio Classification Benchmark," *arXiv:2204.02121 [cs, eess]*, Apr. 2022, arXiv: 2204.02121. [Online]. Available: http://arxiv.org/abs/2204.02121

[20] H. Phan, L. Hertel, M. Maass, P. Koch, R. Mazur, and A. Mertins, "Improved Audio Scene Classification Based on Label-Tree Embeddings and Convolutional Neural Networks," *IEEE/ACM Transactions on Audio, Speech, and Language Processing*, vol. 25, no. 6, pp. 1278–1290, June 2017, conference Name: IEEE/ACM Transactions on Audio, Speech, and Language Processing.

[21] Z. Peng, Z. Li, J. Zhang, Y. Li, G.-J. Qi, and J. Tang, "Few-Shot Image Recognition With Knowledge Transfer," 2019, pp. 441–449. [Online]. Available: https://openaccess.thecvf.com/content_ICCV_2019/html/Peng_Few-Shot_Image_Recognition_With_Knowledge_Transfer_ICCV_2019_paper.html

[22] H. F. Garcia, A. Aguilar, E. Manilow, and B. Pardo, "Leveraging Hierarchical Structures for Few-Shot Musical Instrument Recognition," in *Proceedings of the 22nd International Society for Music Information Retrieval Conference, ISMIR 2021*, Online, November 7-12, 2021, J. H. Lee 0001, A. Lerch 0001, Z. Duan, J. Nam, P. Rao, P. v. Kranenburg, and A. Srini-vasamurthy, Eds., 2021, pp. 220–228. [Online]. Available: https://archives.ismir.net/ismir2021/paper/000027.pdf

[23] K. Simonyan and A. Zisserman, "Very Deep Convolutional Networks for Large-Scale Image Recognition," in *International Conference on Learning Representations*, 2015.

[24] T. Virtanen, M. D. Plumbley, and D. Ellis, Eds., *Computational Analysis of Sound Scenes and Events*. Cham: Springer International Publishing, 2018. [Online]. Available: http://link.springer.com/10.1007/978-3-319-63450-0
SEGMENT-LEVEL METRIC LEARNING FOR FEW-SHOT BIOACOUSTIC EVENT DETECTION

Haohe Liu<fnref n="1" />, Xubo Liu<fnref n="1" />, Xinhao Mei<fnref n="1" />, Qiuqiang Kong², Wenwu Wang<fnref n="1" />, Mark D. Plumbley<fnref n="1" />

¹ Centre for Vision, Speech, and Signal Processing (CVSSP), University of Surrey, UK

² Speech, Audio, and Music Intelligence (SAMI) Group, ByteDance, China

ABSTRACT

Few-shot bioacoustic event detection is a task that detects the oc-
currence time of a novel sound given a few examples. Previous
methods employ metric learning to build a latent space with the la-
beled part of different sound classes, also known as positive events.
In this study, we propose a segment-level few-shot learning frame-
work that utilizes both the positive and negative events during model
optimization. Training with negative events, which are larger in vol-
ume than positive events, can increase the generalization ability of
the model. In addition, we use transductive learning on the valida-
tion set during training for better adaptation to novel classes. We
conduct ablation studies on our proposed method with different se-
tups on input features, training data, and hyper-parameters. Our
final system achieves an F-measure of 62.73 on the DCASE 2022
challenge task 5 (DCASE2022-T5) validation set, outperforming
the performance of the baseline prototypical network 34.02 by a
large margin. Using the proposed method, our submitted system
ranks 2nd in DCASE2022-T5 with an F-measure of 48.2 on the
evaluation set. The code of this paper is open-sourced<fnref n="1" />.

***Index Terms***— few-shot learning, transductive learning, metric learning, audio event detection

# 1. INTRODUCTION

Few-shot learning (FSL) [1] is a machine learning problem that
makes predictions based on the training data that contains limited
information. Sound event detection (SED) [2] is a task that locates
the onset and offset of certain sound classes. By combining the idea
of FSL with SED [3], a system can detect a new type of sound with
only a few examples. Few-shot SED is useful for audio data label-
ing, especially when the user needs to detect a new type of sound.

Most prior studies use a prototypical network [4] as the main
architecture [5, 6, 7, 8]. Yang et al. [5] propose a mutual learn-
ing framework that employs transductive learning to iteratively im-
prove the feature extractor and classifier, where transductive learn-
ing means the model has access to the test set without labels during
the training process. A smoother manifold of embedding space can
help extend the decision boundary and reduce the noise in data rep-
resentation [9]. Tang et al. [6] propose to use embedding propaga-
tion [9] in few-shot SED to learn a smoother manifold by interpolat-
ing between the model output features based on a similarity graph.
Data augmentations such as spec-augment and mixup are used in
the method described in [7, 8]. There is also a spectrogram-cross-
correlation-based method called template matching [10], which per-
forms detection based on the normalized cross-correlation between
example sound event and unlabeled data.

<figure><img src="image_63.png" /><figcaption>**Figure 1:** Training and evaluation procedure of the N-way-M-shot segment-level metric learning. *M* denotes the number of segments or embeddings.</figcaption></figure>

Metric learning [11] refers to learning a distance function and feature space for a task. Previous metric-learning-based studies [5, 6] usually optimize the model with the labeled positive events, by grouping and separating the latent prototypes of the events with the same and different classes, respectively. The audio chunks that do not contain target events, which we refer to as negative events, are larger in volume but receive less attention. For example, in the DCASE 2022 task 5 development set [10], the duration of the negative events is 19.18 hours, accounting for 91.3% of the training data with a total duration of 21 hours.

In this paper, we propose a segment-level metric learning
method that achieves state-of-the-art results on the few-shot bio-
acoustic detection task. As shown in Figure 1, our system operates
on a segment level. Each sound event can contain multiple seg-
ments. We train a feature extraction network that maps the segments
into latent embeddings, which are averaged into prototypes to rep-
resent different sound classes. To learn a robust latent space, we use
a transductive learning scheme and propose to build contrastive loss
with negative events. We also improve our method by using feature
selection, data augmentation, and post-processing. We perform ab-
lation studies to measure the effectiveness of each component. Our
proposed method achieves an F-measure of 62.73 on the DCASE
task 5 validation set.

This paper will be organized as follows. Section 2 provides
an overview of our system. Section 3 introduces our methodology.
Section 4 discusses the experimental setup. Section 5 reports the
result and the ablation studies. Section 6 summarizes this work and
provides a conclusion.

<footnote n="1">¹https://github.com/haoheliu/DCASE_2022_Task_5</footnote>
<figure><img src="image_64.png" /><figcaption>**Figure 2:** This figure illustrates N-way-M-shot metric learning when N=3 and M=2. (a) Visualization of the previous method, which only uses positive classes. (b) The proposed metric learning with the negative segments we used in our system. Support embeddings are omitted for simplicity. $x_i$ and $\tilde{x}_i$ stand for the positive and negative prototypes of class i.</figcaption></figure>

## 2. SYSTEM OVERVIEW

We build our system using a prototypical network [4], which is widely used for metric-based few-shot learning. The training data $T = (\mathcal{S}_i, \mathcal{Y}_i)_{i=1}^{N_{\text{train}}}$ contains audio feature set $\mathcal{S}_i = \{s_i|y_i = 1\} \sqcup \{\tilde{s}_i|y_i = 0\}$ and its corresponding label set $\mathcal{Y}_i = \{y_i|y_i \in \{0, 1\}\}$, where $\{s_i\}$ and $\{\tilde{s}_i\}$ are the sets of positive and negative segments for class i, respectively, and $N_{\text{train}}$ is the total number of training classes. The evaluation dataset $\mathbb{E} = (\mathcal{S}'_i, \mathcal{Y}'_i)_{i=1}^{N_{\text{eval}}}$ also contains an audio feature set $\mathcal{S}'_i = \{s'_i\}$ and a label set $\mathcal{Y}'_i = \{y'_i\}$, where $N_{\text{eval}}$ is the number of classes in the evaluation set, $|\mathcal{S}'_i| = L_i$ and $|\mathcal{Y}'_i| = K$. Here we have $L_i \ge K$ because the evaluation set is partially labeled with only first K events. The validation set has the same structure as the evaluation set. The objective of our system is properly mapping different audio features into a latent embedding within a high-dimensional space, where similar audio features are closer together.

We use episodic training [12] to optimize our system in an N-way-M-shot way. As illustrated in Figure 2(a), N-way-M-shot means each training batch will select data from N classes. And for each classes i, the system will randomly select M segments $\{s_{ij}^q\}_{j=1...M}$ as support segments and another M segments $\{s_{ij}^s\}_{j=1...M}$ as query segments. All the segments in different classes have the same length. Then a feature extraction network (Section 3.1) will map these segments into fix-length embeddings, which are later averaged into query prototypes $x_i^q$ and supporting prototypes $x_i^s$. The system is optimized by minimizing the distance between the query and support prototypes with the same class. To build a robust latent space and generalize better to the new class, we propose to use the negative event in metric learning and the transductive learning scheme in Section 3.2 and 3.3.

During evaluations, the audio file will be segmented using a sliding window with an adaptive segment length (Section 3.4). The segments in the labeled parts will be used to build positive and negative prototypes, which are treated as the latent representation of the positive and negative events in an audio file. The segments in the unlabeled part are the query set, which can be classified by calculating and comparing the distance with the positive and negative prototype (Section 3.5). And if the probability of one query belonging to a positive prototype is greater than a threshold $h$, it will be classified as positive. Consecutive positive predictions will be

merged into one single event.

## 3. METHODOLOGY

### 3.1. Feature extraction network

Our feature extraction network $f_{\theta}$ is a convolutional neural network (CNN) based architecture that maps the audio feature $s$ into a latent embedding $x$. In a similar way to the architecture proposed by [13], the network $f_{\theta}$ consists of three convolutional blocks with hidden channels of sizes 64, 128, and 64. Each convolutional block consists of three two-dimensional CNN layers with batch normalization and leaky rectified linear unit activations [14]. As a common trick in CNN-based network [13, 15], we apply $2 \times 2$ max-pooling after each block for downsampling and enlarging the reception field. The input and output of each convolutional block have a residual connection processed by a downsampling CNN layer. In order to maintain the same output dimension with different input lengths, we apply an adaptive average pooling at the end of the network. The final output feature map after adaptive pooling is a $C \times T \times F$ size block, which is the final latent embedding of $s$.

### 3.2. Segment-level metric learning

We propose to utilize negative segments within negative events during model optimization to learn a more robust representation, as illustrated in Figure 2(b). In a similar way to [3], we first divide the audio features into segments with equal length for metric learning. Then $f_{\theta}$ maps all the segments into latent embeddings. During optimization, we will calculate the class probabilities distributions of the query prototype $x_i^q$, which involves the distance calculation with all the positive and negative support prototypes. In this case, the model can learn a larger amount of contrastive information from the negative events on building the latent space. Specifically, we first calculate a distance matrix $\mathbf{D} = [\mathbf{d}^{(1)}, \mathbf{d}^{(2)}, ..., \mathbf{d}^{(N)}]^T$ according to Equation 1,

$$ \mathbf{d}_{2j}^{(i)} = \|x_i^q - x_j^s\|_2, \quad \mathbf{d}_{2j+1}^{(i)} = \|x_i^q - \tilde{x}_j^s\|_2, \quad (1) $$

where $\mathbf{d}^{(i)} \in \mathbb{R}^{2N}$ stands for the distance between $x_i^q$ and $2N$ support prototypes, and $\tilde{x}_j^s$ denotes the support prototype for the negative events of class j. Then we optimize our model by maximizing the probability that $x_i^q$ is close to the positive support prototype of class i, $x_i^s$, given by

$$ \mathbf{d}^{(i)} = \log(\text{Softmax}(-\mathbf{d}^{(i)})), \quad l = \arg \max_{\theta} (\sum_{i=1}^{N} (\mathbf{d}_{2i}^{(i)})), \quad (2) $$

where $0 \le i, j \le N$, $i, j \in \mathbb{N}$, and $l$ is the objective function. Note that the learning process does not involve the query prototypes for negative events $\tilde{x}_i^q$, because $\tilde{x}_i^q$ and $\tilde{x}_i^s$ are not guaranteed to have the same type of sound.

Data balancing is important in this task because different sound classes have different total durations [10]. In order to balance between classes, we sample each class with equal probability during the episodic training. In this way, the model has equal probabilities to attend to each class and will be less prone to overfitting [16].

### 3.3. Transductive learning

We adopt a transductive learning [17] approach during training, which means our model will be optimized both on the fully-labeled training set and the partially labeled evaluation data. Each file in
evaluation data has first *K* labeled events for a particular type of sound. We treat these *K* events as positive events and the remaining *K* chunks of audio in the labeled part as negative events. In the evaluation set, although the sound class of each file is not available, files with the same sound class should be in the same subfolder, and we treat each subfolder of the evaluation set as a different sound class. Even though the files within each subfolder may not always contain the same target sound, our experiment shows transductive learning in this way can still help the model gain better adaptation to the evaluation set (Section 3).

### 3.4. Adaptive segment length

We use the same segment length among all classes during training for the convenience of batch processing. But during evaluation, using the same segment length is not ideal. For example, using a segment length that is too long or too short will tend to have a high false negative rate or false positive rate, respectively. In the evaluation set, different animal or bird species have drastically different lengths of vocalization, ranging from 30 milliseconds to 5 seconds. Thus we choose to use adaptive segment lengths during evaluation.

<table><caption>**Table 1:** The segment length we use on dividing the evaluation audio file for different values of $t_{max}$.</caption><thead><tr><th>t<sub>max</sub> (s)</th><th>[0,0.1]</th><th>(0.1,0.4]</th><th>(0.4,0.8]</th><th>(0.8,3.0]</th><th>(3.0, ∞)</th></tr></thead><tbody><tr><th>Length</th><td>8</td><td>t<sub>max</sub></td><td>t<sub>max</sub> / 2</td><td>t<sub>max</sub> / 4</td><td>t<sub>max</sub> / 8</td></tr></tbody></table>

As shown in Table 1, we set different segment length for each audio file based on the max length of the labeled events $t_{max} = \max(t_1, ..., t_K)$, where $t_1, ..., t_K$ denotes the duration of the *K* labeled positive events. We set the hop length as one-third of the window length. Note the parameters here are chosen by experience and not necessarily optimal.

### 3.5. Positive and negative prototypes

During the evaluation, we assume the first *K* labeled positive events do not contain too much variety, therefore we calculate the positive prototype by averaging the embeddings of the labeled positive segments. By comparison, building negative prototypes is more tricky because negative segments can contain many different kinds of sounds. So simply averaging all the negative embeddings would result in a sub-optimal representation of negative prototypes. To address these challenges, we choose to run our evaluation six times, each selecting 30 randomly selected negative segments within the labeled negative parts, and we average the predicted probabilities across time of six runs as the final prediction. The negative prototype in each run can have a chance to represent different sounds. This process is similar to the random subspace method [18], in which the ensemble of several estimators trained with a different subset of training data can outperform a single estimator optimized on full training data.

## 4. EXPERIMENTS

### 4.1. Dataset

DCASE2022-T5 The DCASE 2022 task 5 dataset<fnref n="2" /> contains a training set, a validation set, and an official evaluation set. The training

and validation set are both fully labeled. The official evaluation set has the labels of the first five positive events. Our result on the evaluation set is available on the DCASE 2022 Challenge result page<fnref n="3" />. The full label of the official evaluation set is not released at the time of writing, hence, we mainly report the result on the validation set in this paper. The validation during training is not meant to pick the best model. That's because we perform validation in a different way from evaluation. Similar to the training process, we calculate validation accuracy on a fix-length segment level without adaptive segment length. Therefore the best model on validation does not necessarily perform the best during evaluation. Nevertheless, we use the same validation process in our experiments, so the comparisons are fair in different settings. There are also similar ideas in [19, 20], which utilize the evaluation set for validations.

**AudioSet-Aminal-SL** AudioSet [21] is a large-scale dataset for audio research [13, 22]. Considering that the training set of DCASE2022-T5 only contains 47 different sound classes, we choose to use the strongly labeled part of the AudioSet dataset<fnref n="4" /> to augment training data with a wider variety of sounds. To alleviate the domain mismatch problem, we only use sound labels that are related to animal vocalizations and do not overlap with other non-animal sounds. After data cleaning, we have 1796 pieces of audio with 37 classes from AudioSet. However, even if the sounds have the same label in the AudioSet, they can still sound very different. To alleviate this problem, we treat each audio file in AudioSet as its own class, so we have 1796 classes in this dataset, which is named AudioSet-Aminal-SL, where SL means strongly labeled. To balance the 1796 classes and 47 classes in AudioSet-Aminal-SL and DCASE2022-T5, we choose half classes from each dataset during episodic training.

### 4.2. Evaluation metric

We use the F-measure score, the official evaluation metric provided by the organizers of DCASE task 5, as our main evaluation metric. We also report system performance with the Polyphonic Sound Detection Score (PSDS) [23], which is a robust intersection-based sound event detection evaluation metric. In PSDS, we set the detection tolerance criterion (DTC) and the ground truth intersection criterion (GTC) to 0.5, and the maximum effective false positive rate to 100.0. Other parameters like the cross-trigger tolerance criterion (CTTC) are not used because our task is not polyphonic detection.

### 4.3. Experimental setup

Following [5], all the audio data are resampled to a 22.5 kHz sampling rate. The input feature of our system is the stack of PCEN [24] and ΔMFCC [25] features. In the short-time Fourier transform, we set the window length as 1024 and the hop size as 256. We set the mel-frequency dimension as 128. The input length of our model during training is 0.2 seconds. If the sound event is less than 0.2 seconds, zero-padding will be applied. The size of the embedding mentioned in Section 3.1 is 2048, in which $C = 64, T = 4, F = 8$. All the experiments use an initial learning rate of 0.001 with 0.65 exponential decay every 10 epochs. We perform validation after every epoch. We perform validation in a 3-way-5-shot manner since there are only three classes (HB, ME, PB) in the validation set. We will stop model training if the validation accuracy does not improve

<footnote n="3">³https://dcase.community/challenge2022/
task-few-shot-bioacoustic-event-detection-results</footnote>

<footnote n="4">⁴https://research.google.com/audioset</footnote>

<footnote n="2">²https://zenodo.org/record/6482837</footnote>
<figure><img src="image_65.png" /><figcaption>**Figure 3:** The training and validation accuracy at different training steps, both with and without transductive learning, are shown in the left figure. The right figure is the PSD-ROC curve of our proposed system. HB, ME, and PB are three subsets in the validation set. The area under mean_TPR curve is PSDS, which indicates the system's overall performance. TPR and FPR stand for true positive rate and false positive rate, respectively.</figcaption></figure>

<figure><img src="image_66.png" /><figcaption>**Figure 4:** Ablation study on (a) input features; and (b) embedding dimension. We report the PSDS and F-measure on the DCASE2022-T5 validation dataset.</figcaption></figure>

for 10 consecutive epochs. And the model with the best validation accuracy is used for calculating metrics scores. To make full use of training data, we implement a dynamic data loader that generates training data with a random starting time on the fly. We assume the duration of one vocalization for a certain animal does not vary significantly. Therefore, we design the post-processing strategy for a sound class based on maximum length of positive event $t_{max} = \max(t_1, ..., t_K)$. We will remove a positive detection if its length is smaller than $\alpha * t_{max}$ or greater than $\beta * t_{max}$. We use different combinations of $\beta = 2.0, \alpha = [0.1, 0.2, ..., 0.9], h = [0.0, 0.05, ..., 0.95]$ to calculate data points [23], draw the PSD-ROC curve, and calculate PSDS. We choose the best F-measure among all $\beta, \alpha, h$ combinations as the final F-measure.

## 5. RESULT

<table><caption>**Table 2:** Comparisons with baseline template matching and prototypical network methods. Pre. and Rec. stand for precision and recall, respectively. The first two methods [10] did not report PSDS results. All the metrics are written in percentages.</caption><thead><tr><th>Method</th><th>Pre.</th><th>Rec.</th><th>F-measure</th><th>PSDS</th></tr></thead><tbody><tr><td>Template Matching [10]</td><td>2.42</td><td>18.32</td><td>4.28</td><td>N/A</td></tr><tr><td>ProtoNet (official) [10]</td><td>36.34</td><td>24.96</td><td>29.59</td><td>N/A</td></tr><tr><td>ProtoNet (our impl)</td><td>23.26</td><td>63.27</td><td>34.02</td><td>46.10</td></tr><tr><td>Proposed</td><td>69.30</td><td>57.30</td><td><b>62.73</b></td><td><b>57.52</b></td></tr></tbody></table>

The performance of our system on the validation set is reported in Table 2. The F-measure score of template matching and our re-

<table><caption>**Table 3:** Ablation study of the proposed method.</caption><thead><tr><th>Setting</th><th>F-measure (%)</th><th>PSDS (%)</th></tr></thead><tbody><tr><td>Proposed</td><td><b>62.73</b></td><td><b>57.52</b></td></tr><tr><td>w/o Negative contrast</td><td>55.25</td><td>54.95</td></tr><tr><td>w/o Transductive learning</td><td>56.37</td><td>54.50</td></tr><tr><td>w/o Post processing</td><td>57.27</td><td>55.90</td></tr></tbody></table>

<table><caption>**Table 4:** A study on using different training datasets. DCASE stands for the DCASE2022-T5 dataset.</caption><thead><tr><th>Training data</th><th>F-measure (%)</th><th>PSDS (%)</th></tr></thead><tbody><tr><td>DCASE</td><td><b>62.73</b></td><td>57.52</td></tr><tr><td>AudioSet-Aminal-SL</td><td>46.83</td><td>51.00</td></tr><tr><td>AudioSet-Aminal-SL & DCASE</td><td>58.48</td><td><b>58.77</b></td></tr></tbody></table>

implemented prototypical network baseline [10] is 4.28 and 34.02, respectively. Our system outperforms the baselines by a large margin with an F-measure score of 62.73 and a PSDS of 57.52.

As is shown in Figure 3, using transductive learning can significantly improve the validation accuracy. And the class-wise ROC indicates the HB class, which is mostly mosquito sounds, is the easiest one to detect. Class PB is the hardest class perhaps because it mainly consists of sparse bird calls with strong background noise. Class ME achieves an average performance in the validation set.

We perform a study on the effect of the input feature. As shown in Figure 4(a), the performance of F-measure and PSDS is not always consistent, and we use F-measure to guide our selection considering it is widely used in prior studies [10]. By comparing the F-measure score, PCEN+ΔMFCC appears to be a good feature combination on the validation set. We also compare different embedding dimension in Figure 4(b). We change the dimension by altering the dimension of F in the adaptive average pooling. We notice a dimension of 512 can considerably improve over 256, and 2048 has the best performance among all the settings.

We perform ablations on each of the components we proposed. As shown in Table 3, if we remove the negative segments, the performance drops considerably. The trend is the same with transductive learning and post-processing. We also study the effect of training data. In Table 4, we can see that the best F-measure score is achieved using the DCASE2022-T5 only. Using AudioSet-SL leads to an F-measure of 46.83 and a PSDS of 51.00. By combining two datasets we got an F-measure of 58.48 and a best PSDS of 58.77. We hypothesize that the degradation of F-measure using AudioSet is caused by domain mismatch on training data. However, combining two datasets yield the best PSDS, which means using AudioSet data can lead to a general improvement across all threshold and post-processing settings instead of getting a single best system with a high F-measure. This indicates that PSDS might be a suitable metric for the community to reference in this task.

## 6. CONCLUSIONS

This paper proposes a new framework for few-shot sound event detection. Our proposed metric learning with negative segments and the transductive learning scheme can significantly improve model performance. On the input feature, our experiment shows that PCEN with ΔMFCC yields the best performance in our settings. Our result also indicates that PSDS might be a useful metric to evaluate the model's overall performance by considering multiple thresholds and post-processing settings.
# 7. ACKNOWLEDGMENT

This research was partly supported by a Newton Institutional Links Award from the British Council (Grant number 623805725), BBC Research and Development, Engineering and Physical Sciences Research Council (EPSRC) Grant EP/T019751/1 "AI for Sound", and a PhD scholarship from the Centre for Vision, Speech and Signal Processing (CVSSP), Faculty of Engineering and Physical Science (FEPS), University of Surrey. For the purpose of open access, the authors have applied a Creative Commons Attribution (CC BY) license to any Author Accepted Manuscript version arising.

# 8. REFERENCES

[1] S. Ravi and H. Larochelle, "Optimization as a model for few-shot learning," *International Conference on Learning Representations*, 2017.

[2] A. Mesaros, T. Heittola, T. Virtanen, and M. D. Plumbley, "Sound event detection: A tutorial," *IEEE Signal Processing Magazine*, vol. 38, no. 5, pp. 67–83, 2021.

[3] Y. Wang, J. Salamon, N. J. Bryan, and J. P. Bello, "Few-shot sound event detection," in *IEEE International Conference on Acoustics, Speech and Signal Processing*, 2020, pp. 81–85.

[4] J. Snell, K. Swersky, and R. Zemel, "Prototypical networks for few-shot learning," *Advances in Neural Information Processing Systems*, vol. 30, 2017.

[5] D. Yang, H. Wang, Y. Zou, Z. Ye, and W. Wang, "A mutual learning framework for few-shot sound event detection," in *IEEE International Conference on Acoustics, Speech and Signal Processing*. IEEE, 2022, pp. 811–815.

[6] T. Tang, Y. Liang, and Y. Long, "Two improved architectures based on prototype network for few-shot bioacoustic event detection," DCASE2021 Challenge, Tech. Rep., 2021.

[7] Y. Zhang, J. Wang, D. Zhang, and F. Deng, "Few-shot bioacoustic event detection using prototypical network with background classes," DCASE2021 Challenge, Tech. Rep., 2021.

[8] M. Anderson and N. Harte, "Bioacoustic event detection with prototypical networks and data augmentation," *arXiv:2112.09006*, 2021.

[9] P. Rodríguez, I. Laradji, A. Drouin, and A. Lacoste, "Embedding propagation: Smoother manifold for few-shot classification," in *European Conference on Computer Vision*. Springer, 2020, pp. 121–138.

[10] V. Morfi, I. Nolasco, V. Lostanlen, S. Singh, A. Strandburg-Peshkin, L. F. Gill, H. Pamula, D. Benvent, and D. Stowell, "Few-shot bioacoustic event detection: A new task at the dcase 2021 challenge." in *DCASE*, 2021, pp. 145–149.

[11] B. Kulis et al., "Metric learning: A survey," *Foundations and Trends® in Machine Learning*, vol. 5, no. 4, pp. 287–364, 2013.

[12] D. Li, J. Zhang, Y. Yang, C. Liu, Y.-Z. Song, and T. M. Hospedales, "Episodic training for domain generalization," in *Proceedings of the IEEE/CVF International Conference on Computer Vision*, 2019, pp. 1446–1455.

[13] Q. Kong, Y. Cao, T. Iqbal, Y. Wang, W. Wang, and M. D. Plumbley, "PANNs: Large-scale pretrained audio neural networks for audio pattern recognition," *IEEE/ACM Transactions on Audio, Speech, and Language Processing*, vol. 28, pp. 2880–2894, 2020.

[14] B. Xu, N. Wang, T. Chen, and M. Li, "Empirical evaluation of rectified activations in convolutional network," *arXiv:1505.00853*, 2015.

[15] H. Liu, L. Xie, J. Wu, and G. Yang, "Channel-wise subband input for better voice and accompaniment separation on high resolution music," *Proc. Interspeech 2020*, pp. 1241–1245, 2020.

[16] Y. Gong, Y.-A. Chung, and J. Glass, "Psla: Improving audio tagging with pretraining, sampling, labeling, and aggregation," *IEEE/ACM Transactions on Audio, Speech, and Language Processing*, vol. 29, pp. 3292–3306, 2021.

[17] M. Boudiaf, H. Kervadec, Z. I. Masud, P. Piantanida, I. Ben Ayed, and J. Dolz, "Few-shot segmentation without meta-learning: A good transductive inference is all you need?" in *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, 2021, pp. 13979–13988.

[18] T. K. Ho, "The random subspace method for constructing decision forests," *IEEE Transactions on Pattern Analysis and Machine Intelligence*, vol. 20, no. 8, pp. 832–844, 1998.

[19] Q. Kong, Y. Cao, H. Liu, K. Choi, and Y. Wang, "Decoupling magnitude and phase estimation with deep resnet for music source separation," *arXiv:2109.05418*, 2021.

[20] Y. Gong, Y.-A. Chung, and J. Glass, "AST: Audio spectrogram transformer," *arXiv:2104.01778*, 2021.

[21] J. F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, "Audio set: An ontology and human-labeled dataset for audio events," in *IEEE International Conference on Acoustics, Speech and Signal Processing*, 2017, pp. 776–780.

[22] Q. Kong, H. Liu, X. Du, L. Chen, R. Xia, and Y. Wang, "Speech enhancement with weakly labelled data from audioset," *arXiv:2102.09971*, 2021.

[23] Ç. Bilen, G. Ferroni, F. Tuveri, J. Azcarreta, and S. Krstulović, "A framework for the robust evaluation of sound event detection," in *IEEE International Conference on Acoustics, Speech and Signal Processing*, 2020, pp. 61–65.

[24] Y. Wang, P. Getreuer, T. Hughes, R. F. Lyon, and R. A. Saurous, "Trainable frontend for robust and far-field keyword spotting," in *IEEE International Conference on Acoustics, Speech and Signal Processing*, 2017, pp. 5670–5674.

[25] M. A. Hossan, S. Memon, and M. A. Gregory, "A novel approach for mfcc feature extraction," in *2010 4th International Conference on Signal Processing and Communication Systems*, 2010, pp. 946–953.
# EXPLAINING THE DECISIONS OF ANOMALOUS SOUND DETECTORS

Kimberly T. Mai<sup>1,3</sup>, Toby Davies<sup>1,3</sup>, Lewis D. Griffin<sup>1</sup>, Emmanouil Benetos<sup>2,3</sup>

{kimberly.mai, toby.davies, l.griffin}@ucl.ac.uk emmanouil.benetos@qmul.ac.uk

## ABSTRACT

Deciding whether a sound is anomalous is accomplished by comparing it to a learnt distribution of inliers. Therefore, learning a distribution close to the true population of inliers is vital for anomalous sound detection (ASD). Data engineering is a common strategy to aid training and improve generalisation. However, in the context of ASD, it is debatable whether data engineering indeed facilitates generalisation or whether it obscures characteristics that distinguish anomalies from inliers. We conduct an exploratory investigation into this by focusing on frequency-related data engineering. We adapt local model explanations to anomaly detectors and show that models rely on higher frequencies to distinguish anomalies from inliers. We verify this by filtering the input data's frequencies and observing the change in ASD performance. Our results indicate that sifting out low frequencies by applying high-pass filters aids downstream performance, and this could serve as a simple pre-processing step for improving anomaly detectors.

**Index Terms**— anomaly detection, DCASE challenge, data engineering, interpretability

## 1. INTRODUCTION

Anomalous sound detection (ASD) is the task of deciding whether a sound produced from an object is normal or anomalous. This is conducted by comparing the sound to a learnt distribution of inliers. ASD is useful for machine condition monitoring and can result in more timely repairs after unusual sounds are identified. In practice, this task is difficult because only inliers are available for training. In addition, it is hard to anticipate the types of anomalies that may appear (for example, if they manifest in certain frequency bands or at specific times). The task has received more attention in recent years, featuring within the Detection and Classification of Acoustic Scenes and Events (DCASE) challenges for three consecutive years to date [1]. In 2021, the ASD challenge received a total of 75 submissions [2]. Analysing the top submissions, we note they share many similarities. The majority utilise ensembles and use metadata to pre-train their anomaly detection models. However, there is more variability in the usage of data engineering. Notably, the top three entrants do not apply any data engineering through either data augmentations or pre-processing [3, 4].

Data engineering typically modifies the training data using pre-defined rules to improve model generalisation. Whether data engineering helps or harms anomaly detection is debatable within

the machine learning community. However, existing studies have focused on the computer vision domain [5, 6]. In this paper, we seek to address this question for ASD. There are many types of data engineering steps that could be applied to audio such as speed perturbations, adding noise, and frequency masking [7]. We choose to focus on frequency-related data engineering, as [8] proposes the *high-frequency hypothesis*: anomaly detectors use higher frequencies to identify anomalies. If this is the case, methods such as random frequency masking could remove discriminative features between inliers and anomalies. This hypothesis could partly explain why the top three submissions outperform the others.

However, [8] does not explicitly substantiate their claim. We conduct a series of experiments in this paper to verify the high-frequency hypothesis. We firstly adapt Sound LIME (SLIME) [9] to a one-class anomaly detector. SLIME can provide an insight into how modifications at an individual sound clip level can affect model predictions through approximating changes in output using a linear model, hence even providing explanations for models with complex global behaviour. We find the majority of the datasets in the DCASE ASD challenge rely on higher-frequency information to make the correct decisions. Moreover, we quantitatively verify SLIME's result by re-training and re-testing our anomaly detectors by applying low and high pass filters to input data in a sequential manner. Our results corroborate that focusing on higher frequencies helps ASD, and simple frequency filtering instead of complicated data engineering is a more measured approach to improving performance.

## 2. APPROACH

### 2.1. Dataset

We use the DCASE 2021 Task 2 dataset for our experiments. This dataset combines subsets of the ToyADMOS2 [10] (*ToyCar*, *ToyTrain*) and the MIMII DUE [11] (*Fan*, *Gearbox*, *Pump*, *Slider*, *Valve*) datasets. All recordings are single channel, 10 seconds in duration and downsampled to 16 kHz. We refer to the subsets as different machines.

Each machine is subdivided into six sections, which correspond to different properties. Anomaly detectors are evaluated at the section level, using area under the curve (AUC) as the performance metric. Furthermore, each section contains data from two domains: the source domain (for which there is an abundant number of audio clips available) and the target domain representing a domain shift (where only a few clips are available).

The dataset is curated in a one-class fashion: the training split only contains inliers, whereas the test split contains both inliers and anomalies. The training split contains about 1,000 inliers for each of the six sections and is further subdivided into a development set

<footnote n="1"><sup>1</sup> University College London, UK</footnote>

<footnote n="2"><sup>2</sup> Queen Mary University of London, UK</footnote>

<footnote n="3"><sup>3</sup> The Alan Turing Institute, UK</footnote>

<footnote>K. T. Mai, T. Davies and E. Benetos are supported by The Alan Turing Institute under grant EP/N510129/1. K. T. Mai is also supported by EPSRC under grant EP/R513143/1.</footnote>
<figure><img src="image_67.png" /><figcaption>Figure 1: Schematic of the experimental workflow. The different coloured backgrounds correspond to the various stages. For fine-tuning, the AST is tasked with classifying metadata (the correct subset and section). For anomaly detection, the metadata and corresponding classification layer are disregarded. Instead, the embedding at the pre-logits layer of the fine-tuned model are extracted and used to train a *k*-NN. For SLIME, the input spectrograms are perturbed by segmenting via the frequency axis (denoted by the dotted lines in the spectrogram). The predicted distances produced by the trained *k*-NN are converted to a classification decision by using the equal error rate as a threshold. SLIME then approximates the classifier locally with an interpretable linear model to produce a heatmap explanation.</figcaption></figure>

containing the first three sections and an evaluation set consisting of the latter three sections. The test split contains the same number of samples for both domains: about 100 inliers and 100 anomalies per section.

## 2.2. Anomaly detection model

<table><caption>Table 1: Mean AUC scores (%) across all six subsections (source and target) by anomaly detection model. **Bold** denotes the best result.</caption><thead><tr><th></th><th>Fan</th><th>Gearbox</th><th>Pump</th><th>Slider</th><th>ToyCar</th><th>ToyTrain</th><th>Valve</th></tr></thead><tbody><tr><td>Autoencoder (Baseline)</td><td>64.0</td><td>66.8</td><td>63.7</td><td>69.2</td><td>63.2</td><td>63.0</td><td>53.7</td></tr><tr><td>MobileNetV2 (Baseline)</td><td>64.7</td><td>68.2</td><td>64.2</td><td>62.6</td><td>60.0</td><td>59.2</td><td>57.1</td></tr><tr><td>AST with Mel spectrogram</td><td><strong>74.5</strong></td><td>73.9</td><td>69.3</td><td><strong>76.5</strong></td><td>75.9</td><td>64.9</td><td><strong>76.4</strong></td></tr><tr><td>AST with STFT spectrogram</td><td>71.6</td><td><strong>81.0</strong></td><td><strong>73.2</strong></td><td>75.5</td><td><strong>78.9</strong></td><td><strong>65.0</strong></td><td>75.1</td></tr></tbody></table>

We use the same anomaly detection method for our experiments. Namely, we extract features from fine-tuned Audio Spectrogram Transformers (ASTs, 87m parameters) [12] and feed these to a shallow anomaly detection model. We choose to use ASTs as Transformers have demonstrated good anomaly detection performance in other domains [13, 14, 15]. Based on the success of using metadata in the DCASE 2021 submissions for learning representations, we fine-tune ASTs to classify both the machine and section (resulting in a 42-dimensional classification layer) using a cross-entropy loss. We choose to include all seven machines in the fine-tuning stage instead of fine-tuning an AST per machine, as we found that this improved anomaly detection performance. We choose cross-entropy instead of angular losses to reduce the number of hyperparameters that need to be tuned. Starting from ASTs pre-trained on AudioSet [16], we trained our models for a maximum of 10 epochs using the Adam optimiser and a learning rate of $1e-5$. Our fine-tuned model achieved a classification accuracy of 94%.

As an initial experiment to examine the importance of high frequencies, we use both STFT spectrograms and Mel spectrograms (128 bands) as input. We use Torchaudio [17] to compute the input features, using a frame length of 25ms and a hop size of 10ms. In line with the original AST implementation, we also normalise the spectrograms across both the time and frequency axes so that the dataset mean and standard deviation are 0 and 0.5 respectively. Af-

ter fine-tuning, we extract training features at the pre-logits layer, resulting in a 768-dimensional representation. We use the features to train a *k*-NN. For inference, we extract test features in the same manner and use the mean distance from a test datum to its nearest neighbours as the anomaly score. We set $k = 1$ after validation. We tried other methods such as maximum softmax probability [18], Mahalanobis distance and isolation forest [19] and found the nearest neighbour approach worked best.

Table 1 illustrates our initial results. Excluding *Fan*, the AST trained with STFT spectrograms either matches or exceeds the AST trained with Mel spectrograms. As Mel weighting loses spectral resolution at high frequencies, this substantiates the high-frequency hypothesis for the majority of machines.

## 2.3. Preliminaries: Is it appropriate to analyse frequencies?

<figure><img src="image_68.png" /><figcaption>Figure 2: Mean spectral centroids (with standard deviations) across all inlier training samples, calculated using Librosa [20] with $n resc = 2048$ and $hop length = 512$.</figcaption></figure>

To verify whether it is appropriate to examine how frequencies affect ASD, we checked whether the frequency distributions remain stable across different audio clips and times. We measured stability
<figure><img src="image_69.png" /><figcaption>Figure 3: Top: change in classification accuracy after applying frequency filters on Mel spectrograms. Bottom: changes in mean AUC (across all six sections) after applying frequency filters. Applying low-pass filters (left) significantly affects fine-tuning classification accuracy and downstream ASD performance. Conversely, referring to Table 1, classification performance and ASD is remains stable after applying high-pass filtering (right). This indicates low-frequencies potentially obscure discriminative features. The same trends are observed when using STFT spectrograms as input.</figcaption></figure>

by computing the average spectral centroids per frame in the training split for each machine. The spectral centroid is stable across time for most datasets (Figure 2), suggesting it is appropriate to examine the aggregate frequency features across time. We observe some variations to this in *ToyCar* and *ToyTrain*. For these machines, the spectral centroids are stable for most of the clips but dip at the start and end. We speculate this may be due to the 10-second clips being collected in *ToyADMOS2* in a different systematic way.

## 2.4. Adapting local explanations to one-class detectors

<figure><img src="image_70.png" /><figcaption>Figure 4: Median heatmaps generated by SLIME on Source Section 0, segmented by subset and condition {inlier, anomaly}. Yellow regions indicate frequency ranges that are more important for correct decisions while purple regions do not contribute to the ASD decision.</figcaption></figure>

The differences in ASD performance using STFT spectrograms instead of Mel spectrograms in subsection 2.2 indicates that generally, maintaining spectral resolution at high frequencies is helpful. However, these AUC scores only give us a global insight into a detector's behaviour and do not account for differences at a section level. Sound local interpretable model-agnostic explanations (SLIME) [9] can provide a localised understanding. SLIME gen-

erates synthetic samples by firstly segmenting input spectrograms into fixed components in the time or frequency axis. It then randomly occludes input components using the segments and observes the changes in a model's behaviour by approximating the output using a linear model. The changes in a model's decision are visualised in a heatmap. In particular, the heatmaps depict regions contributing or detracting from the prediction. However, SLIME is typically used to interpret classifiers with distinct decision boundaries. Such boundaries are not necessarily present in anomaly detectors. Instead, a detector's output often measures the distance from the inlier training distribution. Examples of these types of anomaly detectors include Mahalanobis distance, k-nearest neighbours and one-class support vector machines.

To adapt SLIME to work for distance-based ASD, we convert the distances to labels $L = \{0 = \text{inlier}, 1 = \text{anomaly}\}$ using the equal error rate (EER) as the threshold. EER indicates the distance where the false-positive rate equals the false-negative rate. We choose EER as the threshold because it summarises overall ASD performance. Given $\lambda$ as the value at which the EER occurs, and $s_i$ as the score output by the anomaly detector for a test datum $x_i$, this decision can be expressed as follows:

$$L = 1[s_i \geq \lambda] \qquad (1)$$

The SLIME region of Figure 1 (denoted by green dashes) illustrates our workflow. We run adapted SLIME to conduct a post-hoc evaluation on the test splits and by section. We segment the spectrograms by frequency as section 2.3 indicates the frequency distributions are relatively stable across time for the subsets. The time domain is not in scope for our experiments. As SLIME's outputs are sensitive to its training hyperparameters [9, 21], we divide the input spectrograms into eight fixed interpretable components and set the number of synthesised samples to 1,000. Figure 4 shows median heatmaps for *Gearbox* and *ToyCar*, split by condi-