second system on the clean challenge dataset seems to give worse
results (ATAE-EP-F: mAP@10 = 0.127) in comparison with simply
training only with the challenge dataset (ATAE). Fourth, our sys-
tem first pretrained with noisy data only and then fine-tuned on the
challenge dataset (ATAE-NP-F: mAP@10 = 0.139) improves on the
performance of the first experiment but only slightly. Finally, all of
our submitted systems surpass the challenge baseline in each metric
by a comfortable margin but are inferior to the best systems in the
challenge.

Since the metrics of our best system (ATAE-NP-F) lie within
the confidence intervals of our next best system (ATAE) and vice
versa, we conclude that no significant difference is measurable be-
tween them. These results suggest that no apparent advantage exists
for our method in utilising additional noisy training data. However,
when comparing the two systems (ATAE & ATAE-NP-F) on the
challenge test set the advantage of pretraining with external data is
more noticeable. A possible explanation for this might be that the
model pretrained with additional external data has better generali-
sation capabilities and is less affected by a shift in data distribution.

5. ABLATION STUDY

Our approach is similar to the systems of the two top-ranked teams ([6, 7]) in the DCASE2022 challenge, yet we fail to reach the same level of retrieval performance. For example, analogous to us, both teams employ a two-tower architecture and shallow neural networks as adapter layers. Their choice of pretrained models (e.g., PANNs & RoBERTa) is also similar to ours. The most striking differences between our and their submissions are that they decided to: (i) use NT-Xent as a loss function, (ii) fine-tune the encoder models, and (iii) use the AudioCaps dataset [10] in pretraining. In view of this resemblance, we conduct additional experiments to investigate why a large gap in performance exists between our submission and the top-ranked systems.

We test five additive changes in training configuration. The re-
sults for each of the configurations are computed from five train-
ing runs. First, we employ the NT-Xent loss instead of the con-
trastive loss. Second, we assess the impact of pair selection for the
loss function on the retrieval metrics. Our submission systems were
trained considering not only text-audio pairs but also text-text and
audio-audio pairs in the loss calculation. Since samples from differ-
ent training instances (i.e., with different labels) will be considered
dissimilar but could contain semantically similar content (e.g., two
different recordings of birds), this could harm the training process.
Therefore, we compare using only text-audio pairs in the loss cal-
culation with using all possible pairs. Third, we want to test if our
approach is restricted by the fixed encoder models and can benefit
if they are fine-tuned in the training process. To limit the compu-
tational cost, we adopt the idea to only fine-tune the text encoder
from a work in computer vision that showed that only fine-tuning
the text model can help to train competitive text-to-image alignment
models [24]. Fourth, we investigate the potential of pretraining with
additional data. As we saw from the results in Section 4, pretrain-
ing with extra (noisy) data might help the model generalise better
to unseen data. Also, both leading teams adopt pretraining in their
training process. This is why we test if adding a pretraining stage
relying on the entire *dev* split of the FSDK50 dataset can enhance
our system's performance. Finally, we evaluate the benefits of fine-
tuning both encoder models instead of only the text encoder similar
to the approach in [7].

Figure 2 compares all ablation experiment configurations by

<figure><img src="image_120.png" /><figcaption>Figure 2: Comparison of the average retrieval results measured in mAP@10 on the development test set for different training configuration settings. The error bars show the standard deviation.</figcaption></figure>

the average mAP@10 achieved on the development test set. What
can be clearly seen in this figure is the accumulative increase in
mAP@10 with every added change. We find that replacing the con-
trastive loss with the NT-Xent loss (see ‘all pairs’ in Fig. 2) already
gives improved results in comparison with our challenge submis-
sion (mAP@10 = 0.193 compared to ATAE: mAP@10 = 0.136).
Only considering text-audio pairs in the NT-Xent loss, however,
further improves the retrieval performance to mAP@10 = 0.209.
Furthermore, fine-tuning the text encoder model and including a
pretraining stage adds to the improvement (mAP@10 = 0.224 and
mAP@10 = 0.228, respectively). As the last change, fine-tuning
both encoder models results in the best score on average (mAP@10
= 0.233). This comparison points to the conclusion that fine-tuning
the encoder models and a pretraining stage are essential to achieve
a high retrieval performance with our method. However, with the
small sample size, the results must be interpreted with caution as the
difference between the last three settings might not be significant.

6. CONCLUSION

We presented an analysis of our submission for the *Language-based Audio Retrieval* subtask of the DCASE2022 challenge. Our approach consists of extracting embeddings for the text and the audio through pretrained encoder models and mapping these embeddings to a shared space with a cross-modal alignment procedure. The best system in our submission is a model that is first pretrained with noisy text-audio data collected from Freesound and later fine-tuned on the challenge dataset. Even though our approach is similar to those of other teams we fall behind in the competition. Through an ablation study, we show that a large part of the performance gap can be attributed to our choice of the loss function and the fact that we keep encoders fixed instead of fine-tuning them. Moreover, we note promising results when pretraining our models with noisy data. Future work should further investigate the use of large quantities of noisy data for pretraining.
7. REFERENCES

[1] H. Xie, S. Lipping, and T. Virtanen, “DCASE 2022 Challenge Task 6B: Language-Based Audio Retrieval,” 2022. [Online]. Available: https://arxiv.org/abs/2206.06108

[2] B. Weck, M. Pérez Fernández, H. Kirchhoff, and X. Serra, “Aligning Audio and Text Embeddings for the Language-Based Audio Retrieval Task of the DCASE Challenge 2022,” DCASE2022 Challenge, Tech. Rep., July 2022.

[3] K. Mahmut and B. Hasan, “Deep metric learning: A survey,” Symmetry, vol. 11, no. 9, p. 1066, Aug 2019. [Online]. Available: http://dx.doi.org/10.3390/sym11091066

[4] S. Chopra, R. Hadsell, and Y. LeCun, “Learning a similarity metric discriminatively, with application to face verification,” in 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), vol. 1, 2005, pp. 539–546 vol. 1.

[5] K. Sohn, “Improved deep metric learning with multi-class n-pair loss objective,” in Advances in Neural Information Processing Systems, D. Lee, M. Sugiyama, U. Luxburg, et al., Eds., vol. 29. Curran Associates, Inc., 2016.

[6] X. Xu, Z. Xie, M. Wu, and K. Yu, “The SJTU System for DCASE2022 Challenge Task 6: Audio Captioning with Audio-Text Retrieval Pre-training,” DCASE2022 Challenge, Tech. Rep., July 2022.

[7] X. Mei, X. Liu, H. Liu, et al., “Language-Based Audio Retrieval with Pre-trained Models,” DCASE2022 Challenge, Tech. Rep., July 2022.

[8] K. Drossos, S. Lipping, and T. Virtanen, “Clotho: an Audio Captioning Dataset,” in ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Barcelona, Spain: IEEE, May 2020, pp. 736–740. [Online]. Available: https://ieeexplore.ieee.org/document/9052990/

[9] I. Martín-Morató and A. Mesaros, “Diversity and Bias in Audio Captioning Datasets,” in Proceedings of the 6th Workshop on Detection and Classification of Acoustic Scenes and Events 2021 (DCASE 2021), Online, November 15-19, 2021, F. Font, A. Mesaros, D. P. W. Ellis, et al., Eds., 2021, pp. 90–94. [Online]. Available: https://dcase.community/documents/workshop2021/proceedings/DCASE2021Workshop_Martin_34.pdf

[10] C. D. Kim, B. Kim, H. Lee, and G. Kim, “AudioCaps: Generating Captions for Audios in The Wild,” in NAACL-HLT, 2019.

[11] F. Font, G. Roma, and X. Serra, “Freesound Technical Demo,” in Proceedings of the 21st ACM International Conference on Multimedia, ser. MM ’13. New York, NY, USA: Association for Computing Machinery, 2013, pp. 411–412, event-place: Barcelona, Spain. [Online]. Available: https://doi.org/10.1145/2502081.2502245

[12] Q. Han, W. Yuan, D. Liu, et al., “Automated Audio Captioning with Weakly Supervised Pre-Training and Word Selection Methods,” in Proceedings of the 6th Workshop on Detection and Classification of Acoustic Scenes and Events 2021 (DCASE 2021), Online, November 15-19, 2021, F. Font, A. Mesaros, D. P. W. Ellis, et al., Eds., 2021, pp. 6–10. [Online]. Available: https://dcase.community/documents/workshop2021/proceedings/DCASE2021Workshop_Han_9.pdf

[13] E. Fonseca, X. Favory, J. Pons, et al., “FSD50K: an open dataset of human-labeled sound events,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 30, pp. 829–852, 2022, publisher: IEEE.

[14] C. D. Manning, P. Raghavan, and H. Schütze, Introduction to information retrieval. New York: Cambridge University Press, 2008.

[15] A. Paszke, S. Gross, F. Massa, et al., “PyTorch: An Imperative Style, High-Performance Deep Learning Library,” in Advances in Neural Information Processing Systems 32, H. Wallach, H. Larochelle, A. Beygelzimer, et al., Eds. Curran Associates, Inc., 2019, pp. 8024–8035. [Online]. Available: http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf

[16] K. Musgrave, S. Belongie, and S.-N. Lim, “PyTorch Metric Learning,” 2020. [Online]. Available: https://arxiv.org/abs/2008.09164

[17] T. Wolf, L. Debut, V. Sanh, et al., “Transformers: State-of-the-Art Natural Language Processing,” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. Online: Association for Computational Linguistics, Oct. 2020, pp. 38–45. [Online]. Available: https://aclanthology.org/2020.emnlp-demos.6

[18] Y. Liu, M. Ott, N. Goyal, et al., “RoBERTa: A Robustly Optimized BERT Pretraining Approach,” 2019. [Online]. Available: https://arxiv.org/abs/1907.11692

[19] V. Sanh, L. Debut, J. Chaumond, and T. Wolf, “DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter,” 2019. [Online]. Available: https://arxiv.org/abs/1910.01108

[20] B. Weck, X. Favory, K. Drossos, and X. Serra, “Evaluating Off-the-Shelf Machine Listening and Natural Language Models for Automated Audio Captioning,” in Proceedings of the Detection and Classification of Acoustic Scenes and Events 2021 Workshop (DCASE2021), Barcelona, Spain, Nov. 2021, pp. 60–64.

[21] Q. Kong, Y. Cao, T. Iqbal, et al., “PANNs: Large-Scale Pre-trained Audio Neural Networks for Audio Pattern Recognition,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 28, pp. 2880–2894, 2020.

[22] D. P. Kingma and J. Ba, “Adam: A Method for Stochastic Optimization,” in 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, Y. Bengio and Y. LeCun, Eds., 2015. [Online]. Available: http://arxiv.org/abs/1412.6980

[23] A. Mesaros, A. Diment, B. Elizalde, et al., “Sound Event Detection in the DCASE 2017 Challenge,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, pp. 992–1006, 2019.

[24] X. Zhai, X. Wang, B. Mustafa, et al., “LiT: Zero-Shot Transfer with Locked-image Text Tuning,” in IEEE / CVF Computer Vision and Pattern Recognition Conference (CVPR), 2022.
# CONTINUAL LEARNING FOR ON-DEVICE ENVIRONMENTAL SOUND CLASSIFICATION

Yang Xiao<fnref n="1" />,*, Xubo Liu²,*, James King², Arshdeep Singh², Eng Siong Chng<fnref n="1" />,
Mark D. Plumbley², Wenwu Wang²

¹ School of Computer Science and Engineering, Nanyang Technological University, Singapore

² Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey, UK

## ABSTRACT

Continuously learning new classes without catastrophic forgetting is a challenging problem for on-device environmental sound classification given the restrictions on computation resources (e.g., model size, running memory). To address this issue, we propose a simple and efficient continual learning method. Our method selects the historical data for the training by measuring the per-sample classification uncertainty. Specifically, we measure the uncertainty by observing how the classification probability of data fluctuates against the parallel perturbations added to the classifier embedding. In this way, the computation cost can be significantly reduced compared with adding perturbation to the raw data. Experimental results on the DCASE 2019 Task 1 and ESC-50 dataset show that our proposed method outperforms baseline continual learning methods on classification accuracy and computational efficiency, indicating our method can efficiently and incrementally learn new classes without the catastrophic forgetting problem for on-device environmental sound classification.

**Index Terms—** Continual learning, environmental sound classification, on-device, convolutional neural networks

## 1. INTRODUCTION

Environmental sound classification aims to categorize audio recordings into pre-defined environmental sound classes [1]. Recently, on-device environmental sound classification [2, 3, 4] has attracted increasing research interest, as shown in Task 1 of Detection and Classification of Acoustic Scenes and Events (DCASE) 2022 Challenge: "Low-Complexity Acoustic Scene Classification" [5]. Such a sound classification system with low computation-complexity can be deployed on mobile and embedded platform for many real-world audio applications, such as acoustic surveillance [6], bio-acoustic monitoring [7] and multimedia indexing [8].

Most existing environment sound classification models [1, 3, 4, 9, 10] are trained with limited sound classes, which cannot directly adapt to new sound classes. When model developers want to expand the categories of environmental sounds to be classified, one way to do this is to fine-tune the model with new classes of data [11, 12]. However, this method may discard previously learned knowledge during the fine-tuning process: this is also known as the catastrophic forgetting problem [13]. Another possible solution is to re-train sound classification models with a mixture of historical and new data. However, this method is resource- and time-consuming in real-world on-device scenarios. As the solution based on re-training is computationally expensive, it is important to design efficient and effective methods to adapt the trained on-device sound classification model to new sound classes.

Continual learning (CL) [14, 15, 16] aims to continuously learn new knowledge over time while retaining and reusing previously learned knowledge. Existing CL methods can be generally divided into two categories: regularization-based methods [17, 18] and replay-based methods [19, 20]. Regularization-based methods use a regularization loss to preserve previously learned model parameters when learning new knowledge. Replay-based methods use a memory update algorithm (MUA) [20, 21, 22] to sample a few informative examples from historical data. The selected examples are used to preserve information about old classes when training new classes. Recently, replay-based CL methods have shown promising results outperforming regularization-based methods in audio tasks such as keywords spotting [23, 24] and sound event detection [25]. However, CL in on-device applications, such as on-device environmental sound classification, has received less attention in the literature, which is the focus in this paper. The on-device scenarios are often associated with restrictions in storage and memory space [3], which can pose challenges to replay-based CL which relied on external memory to restore historical data. As a result, the sound classification models that can be operated on the device may be limited in their capacities, thus prone to forgetting old knowledge when continuously learning new sound classes.

In this work, we investigate the replay-based CL (RCL) methods for on-device environmental sound classification. We first study the performance of existing memory update algorithm (MUA) methods such as Reservoir [21], Prototype [20] and Uncertainty [22] (as described in Section 2.1) on RCL for on-device environmental sound classification. We empirically demonstrate that Uncertainty [22] method performs best in our scenario. Furthermore, we propose Uncertainty++, a simple yet efficient MUA method based on Uncertainty method. Different to the Uncertainty method, our proposed Uncertainty++ introduces the perturbations to the embedding layer of the classifier. As a result, the computation cost (e.g., running memory and time) can be significantly reduced when measuring the data uncertainty. We evaluate the performance of our method on the DCASE 2019 Task1 [26] and the ESC-50 [27] datasets with on-device model BC-ResNet-Mod (~86k parameters) [28, 29]. Experimental results show that uncertainty++ outperforms the existing MUA methods on classification accuracy, indicating its potential in real-world on-device audio applications. Our proposed method is model-independent and simple to apply. Our code is made available at the GitHub<fnref n="1" />.

The remainder of this paper is organized as follows. Section 2 introduces the continual learning method we proposed for on-device environmental sound classification. Section 3 and Section 4 present the experimental settings and the evaluation results. Conclusions and future directions are given in Section 5.

<footnote>*The first two authors contributed equally to this work.</footnote>

<footnote n="1">¹https://github.com/swagshaw/ASC-CL</footnote>
## 2. METHOD

This section first describes replay-based continual learning and four memory update algorithms, and then introduces the proposed uncertainty++ algorithm.

### 2.1. Replay-based continual learning

Following the continual learning setting [14, 18, 25] of environmental sound classification, we assume that the model M should identify all classes in a series of tasks $T = \{\tau_0, ..., \tau_t\}$ without catastrophic forgetting. For each task $\tau \in T$, we have input pairs $(x, y)$ and classes $C$, where $x$ denotes audio waveforms and $y$ are classes $c \in C$. We aim to minimize a cross-entropy loss of all classes $C$ present in the current task $\tau$ formulated as:

$$L_{CE}(\tau) = \sum_{c \in C} y_c \log \frac{\exp(M(x)_c)}{\sum_{c \in C} \exp(M(x)_c)}, \quad (1)$$

Where $M(x)$ denotes the output of the model $M$ for input $x$.

The parameters learned from the previous task are potentially overwritten after learning the new class, also known as catastrophic forgetting. To mitigate this issue, we introduce replay-based methods. The replay-based methods utilize a region of the memory which is called 'replay buffer' to temporarily store the historical training samples to maintain the performance.

Re-training sound classification models with the mixture of the whole historical and new data is resource- and time-consuming in real-world on-device scenarios. To mitigate this issue, the replay-based methods access only a subset of the historical data to save the storage space. In this case, how to select the part of samples to the replay buffer by the memory update algorithm is the key.

Specifically, in the training of task $\tau_t$, the replay buffer stores the selected training samples from the previous $t-1$ learned task(s) $\{\tau_0, \tau_1, ..., \tau_{t-1}\}$, and builds the training data buffer $\hat{D}_t$ for task $\tau_t$ formulated as:

$$\hat{D}_t = g(\hat{D}_{t-1}) \cup D_t, \quad (2)$$

where $g$ is the memory update algorithm [24], $\hat{D}_{t-1}$ is the training data buffer for task $\tau_{t-1}$, and $D_t$ is the incoming data for the new task.

#### 2.1.1. Memory update algorithm (MUA)

We introduce four memory update algorithms in the literature. Generally, we assume that the memory update should select $L$ samples from the training data $\hat{D}_{t-1}$ of the previous task $\tau_{t-1}$ for the training of the task $\tau_t$.

**Random** [30] memory update algorithm selects $L$ new samples $\{(x_1, y_1), (x_2, y_2), ..., (x_L, y_L)\}$ for the next task randomly from the candidates $\hat{D}_{t-1}$ into replay buffer.

**Reservoir** [21] memory update algorithm conducts uniform sampling from $\hat{D}_{t-1}$. Specifically, the reservoir algorithm initializes the replay buffer indexed from 1 to $L$, containing the first $L$ items $\{(x_1, y_1), (x_2, y_2), ..., (x_L, y_L)\}$ of the candidates. When updating replay buffer from the candidates, for each sample, the reservoir algorithm generates a random number $m$ uniformly in $\{1, ..., len(\hat{D}_{t-1})\}$. If $m \in \{1, ..., L\}$, then the sample with the index $m$ in the replay buffer is replaced with the sample $\hat{D}_{t-1}[m]$.

<figure><img src="image_121.png" /><figcaption>Figure 1: Block diagram of the native uncertainty approach and our proposed approach. Specifically, the naive approach adds perturbations to $x$ by waveform and generates multiple waveform as $\hat{x}$. Our approach inputs the embedding $e$ and generates perturbed embedding $\hat{e}$ which means we only save the embedding. The output of the backbone of the model is calculated only once. “Compute $u(x)$” is to compute $u(x)$ by Eq. (3). The $K$ refers to the number of the perturbations generated by perturb methods.</figcaption></figure>

**Prototype** [20] memory update algorithm selects the samples from $\hat{D}_{t-1}$ where the embedding of the classifier is close to the embedding mean of its own class. Specifically, the algorithm first groups the $\hat{D}_{t-1}$ into subsets as $D_c, c = 1...N^t$ by unique classes, where $N^t$ denotes the total numbers of unique classes in the $\hat{D}_{t-1}$ set. Then the algorithm uses the current model to extract the embedding of the candidates for each $D_c$ and calculates the class mean by the embedding as the average feature vector. For each class, the algorithm selects the samples of the candidates so that the average feature vector over the replay buffer provides best approximate to the average feature vector over all the samples of the corresponding class.

**Uncertainty** [22] memory update algorithm selects the sample by the uncertainty of the sample through the inference by the classification model. Specifically, the first step groups the $\hat{D}_{t-1}$ in the same way as the prototype algorithm introduced above. The second step estimates the uncertainty of each sample $x$ in $D_c$. Predictive likelihood captures how well a model fits the data, with larger values indicating better model fit. Uncertainty score can be determined from predictive likelihood [31]. Following the derivation from [31], the predictive likelihood of a sample given by the model can be approximated by the Monte-Carlo (MC) integration [32] method with the model outputs of perturbed samples [24], which is defined as follows:

$$P(y = c | x) = \int p(y = c | \hat{x})p(\hat{x} | x)d\hat{x}, \quad (3)$$

where $x, \hat{x}, y$ denote an audio utterance of one class, the perturbed samples of $x$, and the label of $x$. Therefore, the uncertainty of the audio utterance $x$ is formulated as $u(x)$:

$$u(x) \approx 1 - \frac{1}{K} \sum_{k=1}^{K} P(y = c | \hat{x}_k), \quad (4)$$

where $K$ presents the number of the perturbations generated by perturb methods such as Audio Shift [33], Audio PitchShift [33] and Audio Colored Noise [34, 35]. A larger $u(x)$ indicates a smaller confidence of the model in predicting the perturbed samples. The third step selects $L$ examples from $D_c$ through descending the uncertainty $u(x)$ with the step size of $len(D_c) * C/L$, where $L$ is the size of the replay buffer.
Previous research [24] demonstrated that the uncertainty memory update algorithm performs better than the other three algorithms on speech tasks such as keyword spotting. However, the computation cost of *Uncertainty* increases linearly with the number of perturbation operations.

## 2.2. Proposed MUA method (*Uncertainty++*)

As illustrated in Figure 1, the native uncertainty memory update algorithm requires to employ perturbation methods offline for the waveform of each sample to generate the perturbed samples first. In our proposed method, noisy perturbations are added to the pre-classifier embedding of the sample, and not to the waveform, so the output of the backbone of the model is calculated only once. Specifically, we propose a vector-wise perturbation method that adds noise with different intensities according to the variance of classifier’s embedding. We denote the perturbed version of the classifier’s embedding *e* as *ê*, which is computed as follows:

$$ \hat{e} = e + U(-\frac{\lambda}{2}, \frac{\lambda}{2}) * \text{std}(e), \quad (5) $$

where *std(·)* stands for standard deviation, the function *U(a, b)* represents the noise distributed uniformly from *a* to *b*, *U(a, b)* is a vector with the same shape as *e*, and *λ* is a hyperparameter that controls the relative noise intensity.

By the vector-wise perturbation method, we generate the perturbed embedding *ê* of the embedding *e*. Finally, we input *ê* to the final classification layer of the model and output *P(y = c | ê)* which is used to compute the uncertainty as in Eq. (3). After the uncertainty is estimated, we select examples for replay as native approach. This method saves time by calculating the output of the backbone of the model only once. We also save the memory usage by replacing the perturbed raw data with the classifier’s embedding which is of much smaller size as compared with the raw data.

# 3. EXPERIMENTS

## 3.1. Environmental sound classification model

For the on-device environmental sound classification model, we use BC-ResNet-Mod [29] which is an adaptation of the BC-ResNet [28] that achieves improved results on acoustic scene classification. The BC-ResNet paradigm works via repeatedly extracting spectral and then temporal features in series. Because these spectral features are of a lower dimension than the input, this model has fewer parameters than one that processes the waveform directly. Feature extraction is channel-wise, and both parameter reductions have negligible impact on performance [28]. For our experiments, we use BC-ResNet-Mod-4, which increases the input channel dimension to 80 before extracting spectral and temporal features.

## 3.2. Datasets

ESC-50 consists of 2000 five-second environmental audio recordings [27]. Data are balanced between 50 classes, with 40 examples per class, covering animal sounds, natural soundscapes, human sounds (non-speech), and ambient noises. The dataset has been prearranged into five folds for cross-validation.

DCASE 2019 Task 1 is an acoustic scene classification task, with a development set [26] consisting of 10-second audio segments from 10 acoustic scenes: airport, indoor shopping mall, metro station, pedestrian street, public square, the street with a medium level of

<table><caption>Table 1: Accuracy (ACC) and Backward Transfer (BWT) in a comparative study of the proposed memory update algorithm.</caption><thead><tr><th rowspan="2">Method</th><th colspan="2">DCASE 2019 Task 1</th><th colspan="2">ESC-50</th></tr><tr><th>ACC ↑</th><th>BWT ↑</th><th>ACC ↑</th><th>BWT ↑</th></tr></thead><tbody><tr><td>Finetune</td><td>0.205</td><td>-0.276</td><td>0.181</td><td>-0.307</td></tr><tr><td>Random</td><td>0.473</td><td>-0.115</td><td>0.225</td><td>-0.231</td></tr><tr><td>Reservoir</td><td>0.568</td><td>-0.096</td><td>0.430</td><td>-0.121</td></tr><tr><td>Prototype</td><td>0.559</td><td>-0.089</td><td>0.482</td><td><strong>-0.104</strong></td></tr><tr><td>Uncertainty</td><td>0.578</td><td><strong>-0.079</strong></td><td>0.477</td><td>-0.111</td></tr><tr><td>Uncertainty++</td><td><strong>0.581</strong></td><td><strong>-0.079</strong></td><td><strong>0.500</strong></td><td>-0.121</td></tr></tbody></table>

traffic, traveling by tram, traveling by bus, traveling by an underground metro and urban park. In the development set, there are 9185 and 4185 audio clips for training and validation, respectively.

## 3.3. Experimental setup

**Task setting** To evaluate the performance of the proposed approach, we split the data into five tasks. Each task includes 2 new unique classes in DCASE 19 Task 1 and 10 new unique classes in ESC-50, which is unseen in previous tasks. To simulate the condition of edge devices, we set the buffer size *L* of examples as 500, 100 samples in DCASE 19 Task 1 and ESC-50 due to the memory limitation.

**Implementation details** The original audio clip is converted to 64-dimensional log Mel-spectrogram by using the short-time Fourier transform with a frame size of 1024 samples, a hop size of 320 samples, and a Hanning window. The classification network is optimized by the Adam [36] algorithm with the learning rate 1 × 10⁻³. The batch size is set to 32 and the number of epochs is 50.

## 3.4. Evaluation metrics

We report performances in terms of the accuracy and forgetting metric. Specifically, the *Accuracy* (ACC) reports an accuracy averaged on learned classes after the entire training ends. The *Backward Transfer* (BWT) [37] evaluates accuracy changes on all previous tasks after learning a new task, indicating the forgetting degree. For measuring BWT, we first construct the matrix $R \in \mathbb{R}^{T \times T}$, where $R_{i,j}$ is the test classification accuracy of the model on task $\tau_j$ after observing the last sample from task $\tau_i$. After the model finished learning about each task $\tau_i$, we evaluate its BWT on all *T* tasks, which is formulated as:

$$ BWT = \frac{1}{T-1} \sum_{i=1}^{T-1} R_{T,i} - R_{i,i}. \qquad (6) $$

There exists negative BWT when learning about some task decreases the performance on some preceding task. A smaller value of BWT indicates a higher catastrophic forgetting.

## 3.5. Reference baselines

We built five baselines for comparisons. The *Finetune* training strategy adapts the BC-ResNet-Mod model for each new task without any continual learning strategies, as the lower-bound baseline. The four prior memory update algorithms of replay-based continual learning (i.e., *Random*, *Reservoir*, *Prototype*, *Uncertainty*) are introduced in Section 2.1. Specifically, at the perturbation stage of the uncertainty, we use two perturbation methods, namely, ‘uncertainty-shift’, which
<table><caption>Table 2: Accuracy (ACC) and Backward Transfer (BWT) in a comparative study of the proposed perturbation method. The K refers to the number of the perturbations generated by perturbation methods.</caption><thead><tr><th rowspan="2">Method</th><th rowspan="2">K</th><th colspan="2">DCASE 2019 Task 1</th><th colspan="2">ESC-50</th></tr><tr><th>ACC ↑</th><th>BWT ↑</th><th>ACC ↑</th><th>BWT ↑</th></tr></thead><tbody><tr><td rowspan="3">Uncertainty-Shift</td><td>2</td><td>0.557</td><td>-0.101</td><td>0.461</td><td><strong>-0.111</strong></td></tr><tr><td>4</td><td>0.575</td><td>-0.103</td><td>0.476</td><td>-0.118</td></tr><tr><td>6</td><td>0.567</td><td>-0.079</td><td>0.477</td><td>-0.118</td></tr><tr><td rowspan="3">Uncertainty-Noise</td><td>2</td><td>0.560</td><td>-0.100</td><td>0.465</td><td>-0.118</td></tr><tr><td>4</td><td>0.535</td><td>-0.104</td><td>0.473</td><td>-0.118</td></tr><tr><td>6</td><td>0.578</td><td>-0.079</td><td>0.458</td><td>-0.120</td></tr><tr><td rowspan="3">Uncertainty++</td><td>2</td><td>0.571</td><td>-0.102</td><td><strong>0.500</strong></td><td>-0.121</td></tr><tr><td>4</td><td>0.548</td><td>-0.103</td><td>0.481</td><td>-0.114</td></tr><tr><td>6</td><td><strong>0.581</strong></td><td><strong>-0.079</strong></td><td>0.484</td><td>-0.119</td></tr></tbody></table>

includes Audio Shift and Audio PitchShift, and ‘uncertainty-noise’ which refers to the Audio Colored Noise perturbation method.

# 4. RESULTS

## 4.1. Experiments on MUA methods

Table 1 presents the results on DCASE 2019 Task 1 and ESC-50 test set in terms of ACC and BWT. We compare the proposed *Uncertainty++* MUA method with five baselines. We observe that the *uncertainty* MUA method achieves better performance than the five baselines. Comparing with the best baseline *uncertainty*, we observe that the proposed *uncertainty++* method obtains 58.1% on classification accuracy which outperforms the existing MUA methods. In addition, we observe that the *Finetune* method achieves the worst ACC and BWT performance compared with other baselines, which indicates the issue of catastrophic forgetting.

We further analyze and summarize the performances of the proposed *uncertainty++* method compared with the *uncertainty* MUA method with different numbers of the perturbation methods in terms of ACC and BWT as shown in Table 2. The K refers to the number of the perturbations generated by perturb methods. Even with only two perturbation methods, our proposed method still outperforms other two baselines. We also observe that our method under two perturbations obtains the best performance on the ESC-50 test set. Such performance might be due to the small size of the ESC-50, therefore it is more sensitive to perturbations.

## 4.2. Comparative experiments on computation time for Uncertainty and Uncertainty++

We further report the Average Time for the proposed method when there is an increasing number of perturbations. The Average Time measures a relative time increase compared to training time in each task. As shown in Table 3, even with 6 perturbations, the Average Time of the *uncertainty++* is still less than 60s. This can be explained by the fact that our proposed method can limit the growth of the additional training time. We also observe that our proposed method outperforms other baselines in any number of perturbations, which indicates our proposed method is computationally more efficient. In addition, the average time of *uncertainty-shift* is much longer than others. Because the Audio Shift and Audio PitchShift perturbations takes more time than simply adding noise.

<table><caption>Table 3: Average Time (s) in a comparative study of the proposed *uncertainty++* method. The K refers to the number of the perturbations generated by perturbation methods.</caption><thead><tr><th>Method</th><th>K</th><th>Average Time (s) ↓</th></tr></thead><tbody><tr><td rowspan="3">Uncertainty-Shift</td><td>2</td><td>1221.7</td></tr><tr><td>4</td><td>2205.1</td></tr><tr><td>6</td><td>2926.1</td></tr><tr><td rowspan="3">Uncertainty-Noise</td><td>2</td><td>246.2</td></tr><tr><td>4</td><td>390.8</td></tr><tr><td>6</td><td>506.3</td></tr><tr><td rowspan="3">Uncertainty++</td><td>2</td><td>44.0</td></tr><tr><td>4</td><td>48.5</td></tr><tr><td>6</td><td>55.1</td></tr></tbody></table>

# 5. CONCLUSIONS

In this work, we have presented *uncertainty++*, an efficient replay-based continual learning method for on-device environmental sound classification. Our method selects the historical data for the training by measuring the per-sample classification uncertainty on the embedding layer of the classifier. Experimental results on the DCASE 2019 Task 1 and ESC-50 datasets show that our proposed method outperforms the baseline continual learning methods on classification accuracy and computational efficiency. In future work, we plan to apply and adapt our approach to other on-device audio classification tasks such as audio tagging and sound event detection.

# 6. ACKNOWLEDGMENT

This work is partly supported by UK Engineering and Physical Sciences Research Council (EPSRC) Grant EP/T019751/1 “AI for Sound”, a Newton Institutional Links Award from the British Council, titled “Automated Captioning of Image and Audio for Visually and Hearing Impaired” (Grant number 623805725), a Research Scholarship from the China Scholarship Council (CSC), and a PhD studentship from the Engineering and Physical Sciences Research Council (EPSRC) Doctoral Training Partnership EP/T518050/1. For the purpose of open access, the authors have applied a Creative Commons Attribution (CC BY) licence to any Author Accepted Manuscript version arising.

# 7. REFERENCES

[1] K. J. Piczak, “Environmental sound classification with convolutional neural networks,” in Proc. IEEE 25th International Workshop on Machine Learning for Signal Processing (MLSP), 2015, pp. 1–6.

[2] A. Singh, J. A King, X. Liu, W. Wang, and M. D. Plumbley, “Low-complexity CNNs for acoustic scene classification,” DCASE2022 Challenge, Tech. Rep., June 2022.

[3] A. Singh and M. D. Plumbley, “A passive similarity based cnn filter pruning for efficient acoustic scene classification,” arXiv preprint:2203.15751, 2022.

[4] K. Choi, M. Kersner, J. Morton, and B. Chang, “Temporal knowledge distillation for on-device audio classification,” in Proc. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022, pp. 486–490.

[5] I. Martín-Morató, F. Paissan, A. Ancilotto, T. Heittola,
A. Mesaros, E. Farella, A. Brutti, and T. Virtanen, "Low-complexity acoustic scene classification in DCASE 2022 Challenge," *arXiv preprint:2206.03835*, 2022.

[6] R. Radhakrishnan, A. Divakaran, and A. Smaragdis, "Audio analysis for surveillance applications," in *Proc. IEEE Workshop on Applications of Signal Processing to Audio and Acoustics, 2005.*, 2005, pp. 158–161.

[7] H. Liu, X. Liu, X. Mei, Q. Kong, W. Wang, and M. D. Plumbley, "Surrey system for DCASE 2022 task 5: Few-shot bioacoustic event detection with segment-level metric learning technical report," DCASE2022 Challenge, Tech. Rep., June 2022.

[8] S. Kiranyaz, A. F. Qureshi, and M. Gabbouj, "A generic audio classification and segmentation approach for multimedia indexing and retrieval," *IEEE Transactions on Audio, Speech, and Language Processing*, vol. 14, no. 3, pp. 1062–1081, 2006.

[9] J. Salamon and J. P. Bello, "Deep convolutional neural networks and data augmentation for environmental sound classification," *IEEE Signal processing letters*, vol. 24, no. 3, pp. 279–283, 2017.

[10] J. Sun, X. Liu, X. Mei, J. Zhao, M. D. Plumbley, V. Kılç, and W. Wang, "Deep neural decision forest for acoustic scene classification," *arXiv preprint:2203.03436*, 2022.

[11] N. Hou, C. Xu, E. S. Chng, and H. Li, "Domain adversarial training for speech enhancement," in *Proc. Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC)*. IEEE, 2019, pp. 667–672.

[12] N. Hou, C. Xu, J. T. Zhou, E. S. Chng, and H. Li, "Multi-task learning for end-to-end noise-robust bandwidth extension," in *Proc. Interspeech*, 2020, pp. 4069–4073.

[13] M. McCloskey and N. J. Cohen, "Catastrophic interference in connectionist networks: The sequential learning problem," in *Psychology of Learning and Motivation*, 1989, vol. 24, pp. 109–165.

[14] A. Awasthi and S. Sarawagi, "Continual learning with neural networks: A review," in *Proc. the ACM India Joint International Conference on Data Science and Management of Data*, 2019, pp. 362–365.

[15] H. Zhang, M. Shen, Y. Huang, Y. Wen, Y. Luo, G. Gao, and K. Guan, "A serverless cloud-fog platform for DNN-based video analytics with incremental learning," *arXiv preprint:2102.03012*, 2021.

[16] Y. Huang, H. Zhang, Y. Wen, P. Sun, and N. B. D. Ta, "Modelcice: Enabling continual learning in deep learning serving systems," *arXiv preprint:2106.03122*, 2021.

[17] J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, et al., "Overcoming catastrophic forgetting in neural networks," *the National Academy of Sciences*, vol. 114, no. 13, pp. 3521–3526, 2017.

[18] F. Zenke, B. Poole, and S. Ganguli, "Continual learning through synaptic intelligence," in *Proc. International Conference on Machine Learning*, 2017, pp. 3987–3995.

[19] Y.-C. Hsu, Y.-C. Liu, A. Ramasamy, and Z. Kira, "Re-evaluating continual learning scenarios: A categorization and case for strong baselines," *arXiv preprint:1810.12488*, 2018.

[20] S.-A. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert, "iCaRL: Incremental classifier and representation learning," in *Proc. the IEEE Conference on Computer Vision and Pattern Recognition*, 2017, pp. 2001–2010.

[21] J. S. Vitter, "Random sampling with a reservoir," *ACM Transactions on Mathematical Software (TOMS)*, vol. 11, no. 1, pp. 37–57, 1985.

[22] J. Bang, H. Kim, Y. Yoo, J.-W. Ha, and J. Choi, "Rainbow memory: Continual learning with a memory of diverse samples," in *Proc. the IEEE Conference on Computer Vision and Pattern Recognition*, 2021, pp. 8218–8227.

[23] Y. Huang, N. Hou, and N. F. Chen, "Progressive continual learning for spoken keyword spotting," in *Proc. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, 2022, pp. 7552–7556.

[24] Y. Xiao, N. Hou, and E. S. Chng, "Rainbow keywords: Efficient incremental learning for online spoken keyword spotting," *arXiv preprint:2203.16361*, 2022.

[25] Z. Wang, C. Subakan, E. Tzinis, P. Smaragdis, and L. Charlin, "Continual learning of new sound classes using generative replay," in *Proc. IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA)*, 2019, pp. 308–312.

[26] T. Heittola, A. Mesaros, and T. Virtanen, "TAU Urban Acoustic Scenes 2019, Development dataset," Mar. 2019. [Online]. Available: https://doi.org/10.5281/zenodo.2589280

[27] K. J. Piczak, "Esc: Dataset for environmental sound classification," in *Proc. the ACM international conference on Multimedia*, 2015, pp. 1015–1018.

[28] B. Kim, S. Chang, J. Lee, and D. Sung, "Broadcasted residual learning for efficient keyword spotting," *arXiv preprint:2106.04140*, 2021.

[29] B. Kim, S. Yang, J. Kim, and S. Chang, "QTI submission to DCASE 2021: Residual normalization for device-imbalanced acoustic scene classification with efficient design," DCASE2021 Challenge, Tech. Rep., June 2021.

[30] Y.-C. Hsu, Y.-C. Liu, A. Ramasamy, and Z. Kira, "Re-evaluating continual learning scenarios: A categorization and case for strong baselines," *arXiv preprint:1810.12488*, 2018.

[31] Y. Gal and Z. Ghahramani, "Dropout as a Bayesian approximation: Representing model uncertainty in deep learning," 2016.

[32] T. Kloek and H. K. Van Dijk, "Bayesian estimates of equation system parameters: An application of integration by monte carlo," *Econometrica: Journal of the Econometric Society*, pp. 1–19, 1978.

[33] T. Ko, V. Peddinti, D. Povey, and S. Khudanpur, "Audio augmentation for speech recognition," in *Proc. Interspeech*, 2015.

[34] I. Jordal, S. ES, H. BREDIN, K. Nishi, F. Lata, H. C. Blum, P. Manuel, akash raj, K. Choi, FrenchKrab, P. Želasko, amiasato, M. L. Quatra, and E. Schmidbauer, "asteroid-team/torch-audiomentations: v0.11.0," June 2022. [Online]. Available: https://doi.org/10.5281/zenodo.6778064

[35] S. Kamath and P. Loizou, "A multi-band spectral subtraction method for enhancing speech corrupted by colored noise," in *Proc. IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP)*, 2002.

[36] D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization," *arXiv preprint:1412.6980*, 2014.

[37] D. Lopez-Paz and M. Ranzato, "Gradient episodic memory for continual learning," in *Proc. Advances in neural information processing systems*, vol. 30, 2017.
# LANGUAGE-BASED AUDIO RETRIEVAL TASK IN DCASE 2022 CHALLENGE

Huang Xie, Samuel Lipping, Tuomas Virtanen

Audio Research Group, Tampere University, Tampere, Finland

## ABSTRACT

Language-based audio retrieval is a task, where natural language textual captions are used as queries to retrieve audio signals from a dataset. It has been first introduced into DCASE 2022 Challenge as Subtask 6B of task 6, which aims at developing computational systems to model relationships between audio signals and free-form textual descriptions. Compared with audio captioning (Subtask 6A), which is about generating audio captions for audio signals, language-based audio retrieval (Subtask 6B) focuses on ranking audio signals according to their relevance to natural language textual captions. In DCASE 2022 Challenge, the provided baseline system for Subtask 6B was significantly outperformed, with top performance being 0.276 in mAP@10. This paper presents the outcome of Subtask 6B in terms of submitted systems' performance and analysis.

**Index Terms—** Language-based audio retrieval, DCASE 2022 Challenge, Clotho.

## 1. INTRODUCTION

With the growth of multimedia content in recent decades, there is a need for retrieval methods that can efficiently organize the data based on its content, and retrieve relevant items when doing searches to datasets. Natural language provides an efficient way to represent complex information about multimedia. It can represent high level information about data that goes beyond any fixed taxonomies. For audio signals, natural language can represent information related to temporal relationships between sound sources, and attributes of sounds and their environment.

Language-based multimedia retrieval has received increasing attention in recent years. The majority of recent works has focused heavily on the visual domain [1, 2]. For example, there are plenty of approaches [3] tackling content-based image retrieval with free-form textual descriptions. In contrast, only a few studies have been conducted on language-based audio retrieval in the existing literature. Early works [4, 5] deal with language-based audio retrieval using multi-word text queries consisting of audio tags or class labels, rather than sentence-like textual descriptions, e.g., captions. Recent studies [6, 7] prompt research in this field by exploring human written captions as queries. In DCASE 2022 Challenge, language-based audio retrieval is introduced into as Subtask 6B, which aims to inspire further research into audio retrieval with unconstrained textual descriptions.

In this paper, we present the task setup and submissions for Subtask 6B of task 6 in DCASE 2022 Challenge. We introduce the datasets for system development and evaluation, describe the baseline system for Subtask 6B, and present the challenge submissions. Evaluation and analysis of submitted systems includes general statistics on systems and performance and system characteristics.

The remainder of this paper is organized as follows. In Section 2, we describe the task setup, including task description, task datasets, and evaluation metrics. Then, we introduce the task baseline system in Section 3. We present the evaluation results and analysis of challenge submissions in Section 4. Finally, we conclude this paper in Section 5.

## 2. TASK SETUP

In this section, we introduce the task description, datasets for system development and evaluation, and evaluation metrics.

### 2.1. Task Description

Language-based audio retrieval is concerned with retrieving audio signals using their sound content textual descriptions (i.e., audio captions). With this task, the goal is to evaluate audio retrieval methods, where a retrieval system takes an audio caption as a text query and ranks audio signals in a fixed dataset according to their relevance to the caption. In DCASE 2022 Challenge, human-written natural language audio captions are used as text queries. For each query, the retrieval task is to retrieve 10 audio files from a given evaluation dataset and sort them according to their relevance to the query.

### 2.2. Development Dataset

The Clotho v2 [8] is provided as the task development dataset, which consists of audio samples of 15 to 30 seconds duration, with each audio sample having five captions of eight to 20 words length. There are 6,974 audio samples with 34,870 captions in total. All audio samples are sourced from the Freesound platform [9], and captions are crowd-sourced using a three-step framework [8].

The Clotho v2 [8] is divided into a training split of 3,839 audio clips with 19,195 captions, a validation split of 1,045 audio clips with 5,225 captions, and a testing split of 1,045 audio clips with 5,225 captions. These splits are created by first constructing the sets of unique words of the captions of each audio clip. These sets of words are combined to form the bag of words of the whole dataset, from which the frequency of a given word can be derived. With the unique words of audio files as classes, multi-label stratification is applied. The data collecting procedure is explained in detail in [8].

### 2.3. Clotho Retrieval Evaluation Dataset

The task evaluation dataset consists of 1,000 audio samples sourced from the Freesound platform [9], and one human written caption is provided for each audio sample. The audio samples are collected following the procedure described in [8], by optimizing the tag distribution of the selected samples. The samples were selected from the set of files not used in Clotho v2. Captions were gathered using
<figure><img src="image_122.png" /><figcaption>Figure 1: The baseline system for the language-based audio retrieval subtask.</figcaption></figure>

<table><caption>Table 1: Statistics of the development and evaluation datasets.</caption><thead><tr><th>Dataset</th><th>Split</th><th>#Audio</th><th>#Captions</th></tr></thead><tbody><tr><td rowspan="3">Development</td><td>Training</td><td>3839</td><td>19195</td></tr><tr><td>Validation</td><td>1045</td><td>5225</td></tr><tr><td>Testing</td><td>1045</td><td>5225</td></tr><tr><td colspan="2">Evaluation</td><td>1000</td><td>1000</td></tr></tbody></table>

the first crowd-sourcing step of Clotho v2 manually screened for typographical errors and speech transcription. Table 1 summarizes the information about the development and evaluation datasets.

## 2.4. Evaluation Metrics

In the evaluation, the ground truth relevance of audio samples are considered binary (i.e., only the audio samples belong to the caption query are considered relevant, and all the others not relevant). The submissions for this task will be evaluated using mean average precision at top-10 (mAP@10) as the main metric, and recall at $k$ (R@k with $k \in \{1, 5, 10\}$) as the secondary metrics.

The mAP metric has been widely used for evaluating the performance of cross-modal retrieval algorithms [3]. It is a rank-aware metric, which measures the mean of average precisions (AP) over all the queries. The AP for a query is calculated by averaging the precisions at positions, where relevant items are in the retrieved rank list. The more relevant items in the top rank list, the higher mAP value it has. The R@k metric is another standard, rank-unaware retrieval metric [6], which is defined as the proportion of relevant items among the top-K retrieved results to all the relevant items in the evaluation dataset, averaged across all the caption queries. The challenge submissions will be ranked by the mAP@10 metric.

## 3. BASELINE SYSTEM

In this section, we describe the task baseline system, as illustrated in Figure 1. The baseline system is a simplified version of the audio-text aligning framework presented in [7], which calculates relevant scores between encoded textual descriptions (i.e., encoded captions) and encoded audio signals. It consists of two input encoders: one for audio, and the other for text, as illustrated in Figure 1. These two modality-specific encoders generate vector representations (i.e., audio embeddings and caption embeddings) for audio clips and textual

descriptions. Then, the relevance score between an audio clip and a textual description is calculated by the dot product of their vector representations. The baseline system is optimized with a sampling-based triplet loss [19] at the training stage, and then applied to retrieve audio for caption queries at the testing stage.

### 3.1. Audio Encoder

A convolutional recurrent neural network (CRNN) [20] is used as the audio encoder, which extracts frame-wise acoustic embeddings from audio signals. It consists of five convolution blocks, followed by a bidirectional gated recurrent unit (BiGRU). Each convolution block includes an initial batch normalization, a convolutional layer with padded 3 × 3 convolutions, and a LeakyReLU activation with a slope of −0.1. After the first, third, and fifth convolution blocks, one L4-Norm subsampling layer is used to reduce the temporal dimension of each block's output by a factor of four. A dropout layer with a rate of 0.3 is placed between the last L4-Norm layer and the BiGRU. Lastly, an up-sampling operation is applied to ensure the final output has the same temporal dimension as the CRNN input.

The CRNN audio encoder takes 64-dimensional log mel-band energies as input. Each audio clip is split into 40 ms Hanning-windowed frames with a hop length of 20 ms. Then, 64 log mel-band coefficients are extracted from each frame. A sequence of 300-dimensional frame-wise acoustic embeddings are generated for each audio clip. The final audio embedding is calculated by averaging the frame-wise acoustic embeddings.

### 3.2. Text Encoder

Word2Vec (Skip-gram model) [21] is utilized as the text encoder to convert textual descriptions into sequences of word embeddings. It is a two-layer fully-connected neural network, which learns word embeddings that are good at predicting surrounding words in a sentence or a document. For the sake of simplicity, we adopt a publicly available pre-trained Word2Vec [22], which is trained on Google News dataset. It consists of 300-dimensional word embeddings for roughly three million case-sensitive English words and phrases. The Word2Vec text encoder converts textual descriptions into sequences of semantic word embeddings word by word. The final caption embedding is computed by averaging the word embeddings.
<table><caption>Table 2: Evaluation results with 95% confidence intervals for the top system of each team.</caption><thead><tr><th>Rank</th><th>Team</th><th>mAP@10 with 95% CI</th><th>R@1 with 95% CI</th><th>R@5 with 95% CI</th><th>R@10 with 95% CI</th></tr></thead><tbody><tr><td>1</td><td>Xu et al. [10]</td><td>0.276 [0.254, 0.299]</td><td>0.176 [0.152, 0.200]</td><td>0.416 [0.385, 0.447]</td><td>0.536 [0.505, 0.567]</td></tr><tr><td>2</td><td>Mei et al. [11]</td><td>0.251 [0.229, 0.273]</td><td>0.153 [0.131, 0.175]</td><td>0.387 [0.357, 0.417]</td><td>0.504 [0.473, 0.535]</td></tr><tr><td>3</td><td>Lamort et al. [12]</td><td>0.221 [0.200, 0.242]</td><td>0.131 [0.110, 0.152]</td><td>0.343 [0.314, 0.372]</td><td>0.466 [0.435, 0.497]</td></tr><tr><td>4</td><td>Pellegrini [13]</td><td>0.216 [0.195, 0.237]</td><td>0.127 [0.106, 0.148]</td><td>0.321 [0.292, 0.350]</td><td>0.463 [0.432, 0.494]</td></tr><tr><td>5</td><td>Lai et al. [14]</td><td>0.215 [0.194, 0.235]</td><td>0.122 [0.102, 0.142]</td><td>0.328 [0.299, 0.357]</td><td>0.478 [0.447, 0.509]</td></tr><tr><td>6</td><td>Wu et al. [15]</td><td>0.188 [0.168, 0.207]</td><td>0.107 [0.088, 0.126]</td><td>0.303 [0.275, 0.331]</td><td>0.413 [0.382, 0.444]</td></tr><tr><td>7</td><td>Weck et al. [16]</td><td>0.128 [0.111, 0.145]</td><td>0.077 [0.060, 0.094]</td><td>0.188 [0.164, 0.212]</td><td>0.284 [0.256, 0.312]</td></tr><tr><td>8</td><td>Xiao et al. [17]</td><td>0.097 [0.083, 0.111]</td><td>0.043 [0.030, 0.056]</td><td>0.162 [0.139, 0.185]</td><td>0.267 [0.240, 0.294]</td></tr><tr><td>9</td><td>Park et al. [18]</td><td>0.075 [0.063, 0.088]</td><td>0.033 [0.022, 0.044]</td><td>0.127 [0.106, 0.148]</td><td>0.208 [0.183, 0.233]</td></tr><tr><td>10</td><td>Baseline</td><td>0.061 [0.049, 0.072]</td><td>0.026 [0.016, 0.036]</td><td>0.102 [0.083, 0.121]</td><td>0.176 [0.152, 0.200]</td></tr></tbody></table>

### 3.3. Training Objective

The baseline system is trained by optimizing a ranking-based criterion [19], such that audio clips and captions that belong together are more similar in the embedding space than mismatched audio-caption pairs. Specifically, across a batch of N audio-caption pairs $\{(x_n, y_n)\}_{n=1}^N$, where $y_n$ is the caption pertaining to an audio clip $x_n$, we randomly select an imposter clip $\hat{x}_n$ and an imposter caption $\hat{y}_n$ for each audio-caption pair $(x_n, y_n)$. Then, the widely used sampling-based triplet loss [7, 23] is calculated by

$$ \text{loss} = \frac{1}{N} \sum_{n=1}^{N} [\max(0, S(x_n, \hat{y}_n) - S(x_n, y_n) + 1) + \max(0, S(\hat{x}_n, y_n) - S(x_n, y_n) + 1)], \quad (1) $$

where S is the audio-caption relevance score.

### 3.4. Baseline Results

The baseline system is trained with batches of 32 audio-caption pairs in the training split for at most 150 epochs, while monitoring the loss (1) on the validation split during the training process. An Adam optimizer with an initial learning rate of 0.001 is adopted to optimize the training process. The learning rate is reduced by a factor of ten once the validation loss does not improve for five epochs. Training is terminated by early stopping with ten epochs.

As shown in Table 3, the baseline system achieves similar performance in terms of mAP@10 and recall scores on the testing split and the evaluation dataset. Specifically, with the evaluation dataset, the theoretical chance levels are $1/1000 = 0.001$ for R@1, $1/200 = 0.005$ for R@5, and $1/100 = 0.01$ for R@10, respectively. In contrast to the theoretical chance levels, the baseline system obtains better recall scores, with an R@1 / R@5 / R@10 of $0.026 / 0.102 / 0.176$. The experimental results show that the baseline system can retrieve audio with their corresponding captions, i.e., perform language-based audio retrieval. On the other hand, since the baseline system employs a simple pipeline (e.g., averaging frame-wise acoustic embeddings and word embeddings), the retrieval performance remains limited.

## 4. CHALLENGE SUBMISSIONS

In this section, we present the evaluation results and analysis of the submissions for language-based audio retrieval.

<table><caption>Table 3: Baseline results on the testing split and the evaluation dataset.</caption><thead><tr><th>Dataset</th><th>mAP@10</th><th>R@1</th><th>R@5</th><th>R@10</th></tr></thead><tbody><tr><td>Testing split</td><td>0.068</td><td>0.032</td><td>0.109</td><td>0.188</td></tr><tr><td>Evaluation</td><td>0.061</td><td>0.026</td><td>0.102</td><td>0.176</td></tr></tbody></table>

### 4.1. Evaluation Results

The task received a total number of 31 submissions from nine teams, with maximum four submissions per team allowed. Table 2 shows the results of evaluation metrics (e.g., mAP@10) with 95% confidence intervals (CIs) for the top system of each team, comparing with the baseline system. The 95% confidence intervals for evaluation metrics are calculated using the Jackknife estimate [24].

All the submitted systems outperformed the baseline system in terms of mAP@10 and R@k with $k \in \{1, 5, 10\}$. Particularly, Xu et al. [10] achieved the best performance, with a mAP@10 of 0.276 and a 95% CI between 0.254 and 0.299. Mei et al. [11] ranked second, with their best system having a mAP@10 of 0.251 (95% CI [0.229, 0.273]). The top two teams obtained mAP@10 over 0.250 and R@10 over 0.500, in contrast to those of teams ranked third - fifth, having mAP@10 around 0.220 and R@10 around 0.470. Out of the nine teams, six teams achieved mAP@10 over 0.180, R@5 over 0.300, and R@10 over 0.410. The baseline system ranked last with a significantly lower performance.

### 4.2. Analysis of Submissions

The following analysis is based on the information reported by participating teams.

**System summary.** All the 31 submitted systems and the baseline system adopted a bi-encoder architecture, which consisted of an audio encoder and a caption encoder, to associate audio with captions. For audio encoders, pre-trained convolutional neural networks (e.g., PANNs [25]) were the most common choice (27 systems from eight teams) among the nine participating teams. For caption encoders, pre-trained Transformer-based language embedding models (e.g., BERT [26] and Sentence-BERT [34]) were frequently employed (28 systems from seven teams). Then, audio and captions were encoded into a common embedding space, where their relevance was scored with cosine similarity (24 systems from six teams) or dot product (seven systems from three teams) of their embeddings. Most of the submitted systems (23 systems from seven
<table><caption>Table 4: Summary of systems with best performance from all teams.</caption><thead><tr><th>Rank</th><th>Team</th><th>Audio Modelling</th><th>Caption Modelling</th><th>Loss Function</th></tr></thead><tbody><tr><td>1</td><td>Xu et al. [10]</td><td>PANNs [25]</td><td>BERT [26], RoBERTa [27]</td><td>InfoNCE loss [28]</td></tr><tr><td>2</td><td>Mei et al. [11]</td><td>PANNs [25]</td><td>BERT [26]</td><td>NT-Xent loss [29]</td></tr><tr><td>3</td><td>Lamort et al. [12]</td><td>PANNs [25], OpenL3 [30], VGGSound [31], VGGish [32], BART [33]</td><td>Sentence-BERT [34]</td><td>Triplet loss [6, 7]</td></tr><tr><td>4</td><td>Pellegrini [13]</td><td>PaSST [35]</td><td>Sentence-BERT [34]</td><td>Triplet loss [6, 7]</td></tr><tr><td>5</td><td>Lai et al. [14]</td><td>ESResNet [36]</td><td>Transformer [37]</td><td>CE loss [37]</td></tr><tr><td>6</td><td>Wu et al. [15]</td><td>PANNs [25], HTS-AT [38]</td><td>Transformer [37]</td><td>CE loss [37]</td></tr><tr><td>7</td><td>Weck et al. [16]</td><td>PANNs [25]</td><td>RoBERTa [27]</td><td>Contrastive loss [39]</td></tr><tr><td>8</td><td>Xiao et al. [17]</td><td>PANNs [25]</td><td>Word2Vec [21]</td><td>Triplet loss [6, 7]</td></tr><tr><td>9</td><td>Park et al. [18]</td><td>CRNN [20]</td><td>Word2Vec [21]</td><td>Triplet loss [6, 7]</td></tr><tr><td>10</td><td>Baseline</td><td>CRNN [20]</td><td>Word2Vec [21]</td><td>Triplet loss [6, 7]</td></tr></tbody></table>

teams and the baseline) were trained by optimizing some contrastive losses (e.g., InfoNCE loss [28] and triplet loss [6, 7]).

**Audio encoders.** Among the submitted systems, PANNs [25] (e.g., CNN14, Wavegram-Logmel-CNN14 and CNN10) were the most common choice for encoding audio data. There were in total 21 systems from six teams utilizing PANNs as their audio encoders, including the top three teams [10, 11, 12]. Particularly, the CNN14 model was most preferred (20 systems). Xu et al. [10] experimented with multiple pre-trained audio expert models, including CNN14 and Wavegram-Logmel-CNN14. With ensemble of different models, they achieved the best performance. Mei et al. [11] adopted CNN14 as their audio encoder without model ensembles, having their best system ranked second. Tied on third rank, Lamort et al. [12] experimented with aggregating the most number of pre-trained audio expert models (i.e., five models in total, as shown in Table 4). The audio encoders utilized in other systems were PaSST [35], ESResNet [36], HTS-AT [38], and CRNN [18].

Log-mel energies were frequently taken as input among the submitted systems. A total number of 28 systems from eight teams utilized log-mel energies as standalone audio features or in combination with other features (e.g., raw waveform [10]). Other audio features used included log-magnitude spectrogram [12] and log-power spectrogram [14].

**Caption encoders.** Most of the participating teams (i.e., seven out of nine) preferred Transformer-based language embedding models (e.g., BERT [26]) over word embedding models (e.g., Word2Vec [21]), to learn caption embeddings. As shown in Table 4, pre-trained Transformer-based language embedding models, including BERT [26], RoBERTa [27], and Sentence-BERT [34], were utilized as caption encoders by seven teams (28 systems in total). The other two teams and the baseline converted captions into embeddings with pre-trained Word2Vec word vectors [21]. All teams with Transformer-based models ranked higher than those with pre-trained Word2Vec [21], which showed that Transformer-based models learned caption embeddings more efficiently.

**Loss functions.** The loss functions utilized in the submissions can be categorized into three groups: contrastive loss (one team), triplet loss (four teams), and N-pair loss (four teams). Contrastive loss [40] takes pairs of samples as input and measures the similarity between two inputs. Triplet loss [6, 7] takes as input triplets of samples, consisting of one anchor sample along with one positive and one negative sample. Compared to contrastive loss, triplet loss focuses on the difference of (dis-)similarities between positive and

negative samples to the anchor sample. N-pair loss [40] shares a similar spirit to triplet loss, but extends to multiple negative samples, i.e., one positive and multiple negative samples for an anchor sample. In practice, several versions of N-pair loss were adopted, including InfoNCE loss [28], NT-Xent loss [29], and symmetric cross-entropy (CE) loss [37].

**Data augmentation and external data.** Data augmentation was adopted by only three participating teams (i.e., Mei et al. [11] on rank two, Lamort et al. [12] on rank three, and Wu et al. [15] on rank six) during their systems' training. In contrast, external audio and textual data was leveraged for optimizing system performance by most participating teams (i.e., seven out of nine). The submitted systems were usually pre-trained with a large amount of external audio and textual data, and then fine-tuned on the task development dataset. For example, the top two teams (i.e., Xu et al. [10] and Mei et al. [11]) pre-trained their systems with AudioCaps [41]. Additionally, Wu et al. [15] trained their systems by involving various external data, including audio samples and text (e.g., labels, file names and captions) from seven external datasets in addition to Clotho v2 [8].

## 5. CONCLUSIONS

The DCASE 2022 Challenge has introduced the language-based audio retrieval task, which is about using natural language textual captions as queries to retrieve audio signals from a dataset. This paper describes setups of the task, including task description, datasets for system development and evaluation, evaluation metrics, and a task baseline system. Moreover, this paper reports the final evaluation results of the task submissions and their analysis. The evaluation results show that all submitted systems outperformed the baseline in terms of evaluation metrics, with top performance being 0.276 in mAP@10 and 0.536 in R@10.

One immediate observation about the submitted systems is that all of them adopted a bi-encoder architecture consisting of an audio encoder and a caption encoder. Pre-trained convolutional neural networks (e.g., PANNs [25]) were commonly employed as audio encoders, and pre-trained Transformer-based language embedding models (e.g., BERT [26]) were the preferred caption encoders. In addition to the task development dataset, external audio and textual data (e.g., labels and captions) was also frequently leveraged during system training.
## 6. REFERENCES

[1] J. Dong, X. Li, C. Xu, X. Yang, G. Yang, X. Wang, and M. Wang, "Dual encoding for video retrieval by text," *IEEE Trans. Pattern Anal. Mach. Intell.*, pp. 1–10, 2021.

[2] L. Qu, M. Liu, J. Wu, Z. Gao, and L. Nie, "Dynamic modality interaction modeling for image-text retrieval," in *Proc. ACM SIGIR Int. Conf. Res. Dev. Inf. Retr.*, 2021, pp. 1104–1113.

[3] P. Kaur, H. S. Pannu, and A. K. Malhi, "Comparative analysis on cross-modal information retrieval: A review," *Comput. Sci. Rev.*, p. 100336, 2021.

[4] G. Chechik, E. Ie, M. Rehn, S. Bengio, and D. Lyon, "Large-scale content-based audio retrieval from text queries," in *Proc. Int. Conf. Multimed. Inf. Retr.*, 2008, pp. 105–112.

[5] B. Elizalde, S. Zarar, and B. Raj, "Cross modal audio search and retrieval with joint embeddings based on text and audio," in *Proc. IEEE Int. Conf. Acoustic., Speech and Signal Process. (ICASSP)*, 2019, pp. 4095–4099.

[6] A. M. Oncescu, A. Sophia Koepke, J. F. Henriques, Z. Akata, and S. Albanie, "Audio retrieval with natural language queries," in *Proc. Annu. Conf. Int. Speech Commun. Assoc. (INTERSPEECH)*, 2021, pp. 2411–2415.

[7] H. Xie, O. Räsänen, K. Drossos, and T. Virtanen, "Unsupervised audio-caption aligning learns correspondences between individual sound events and textual phrases," in *Proc. IEEE Int. Conf. Acoustic., Speech and Signal Process. (ICASSP)*, 2022, pp. 8867–8871.

[8] K. Drossos, S. Lipping, and T. Virtanen, "Clotho: an audio captioning dataset," in *Proc. IEEE Int. Conf. Acoustic., Speech and Signal Process. (ICASSP)*, 2020, pp. 736–740.

[9] F. Font, G. Roma, and X. Serra, "Freesound technical demo," in *Proc. Int. Conf. Multimed.*, 2013, pp. 411–412.

[10] X. Xu, Z. Xie, M. Wu, and K. Yu, "The SJTU system for DCASE2022 challenge task 6: Audio captioning with audio-text retrieval pre-training," DCASE2022 Challenge, Tech. Rep., 2022.

[11] X. Mei, X. Liu, H. Liu, J. Sun, M. D. Plumbley, and W. Wang, "Language-based audio retrieval with pre-trained models," DCASE2022 Challenge, Tech. Rep., 2022.

[12] T. Lamort de Gail and D. Kicinski, "Take it easy: Relaxing contrastive ranking loss with CIDER," DCASE2022 Challenge, Tech. Rep., 2022.

[13] T. Pellegrini, "IRIT-UPS DCASE 2022 language-based audio retrieval system," DCASE2022 Challenge, Tech. Rep., 2022.

[14] Y. Lai, J. Pan, and B. Chen, "A ResNet-Based clip text-to-audio retrieval system for DCASE challenge 2022 task," DCASE2022 Challenge, Tech. Rep., 2022.

[15] Y. Wu, T. Zhang, and K. Chen, "Text-to-audio retrieval via large-scale contrastive training," DCASE2022 Challenge, Tech. Rep., 2022.

[16] B. Weck, M. Pérez Fernández, H. Kirchhoff, and X. Serra, "Aligning audio and text embeddings for the language-based audio retrieval task of the DCASE challenge 2022," DCASE2022 Challenge, Tech. Rep., 2022.

[17] F. Xiao, J. Guan, H. Lan, Q. Zhu, and W. Wang, "Language-based audio retrieval with pretrained CNN and graph attention," DCASE2022 Challenge, Tech. Rep., 2022.

[18] J. Park, C. Hwang, I.-Y. Kwak, and C. Lim, "CAU submission to DCASE 2022 task6b: Language-based audio retrieval using transfer learning," DCASE2022 Challenge, Tech. Rep., 2022.

[19] J. Bromley, I. Guyon, Y. LeCun, E. Säckinger, and R. Shah, "Signature verification using a "siamese" time delay neural network," in *Proc. Int. Conf. Neural Inf. Process. Syst. (NIPS)*, 1993, pp. 737–744.

[20] X. Xu, H. Dinkel, M. Wu, and K. Yu, "A crnn-gru based reinforcement learning approach to audio captioning," in *Proc. Detect. Classif. Acoust. Scenes Events Work. (DCASE)*, 2019, pp. 225–229.

[21] T. Mikolov, K. Chen, G. Corrado, and J. Dean, "Efficient estimation of word representations in vector space," in *Proc. Int. Conf. Learn. Representations (ICLR)*, 2013.

[22] "Word2Vec," https://code.google.com/archive/p/word2vec, accessed: 2021-09-27.

[23] D. Harwath, A. Torralba, and J. R. Glass, "Unsupervised learning of spoken language with visual context," in *Proc. Int. Conf. Neural Inf. Process. Syst. (NIPS)*, 2016, pp. 1866–1874.

[24] A. Mesaros, A. Diment, B. Elizalde, T. Heittola, E. Vincent, B. Raj, and T. Virtanen, "Sound event detection in the DCASE 2017 challenge," *IEEE/ACM Trans. Audio Speech Lang. Process.*, pp. 992–1006, 2019.

[25] Q. Kong, Y. Cao, T. Iqbal, Y. Wang, W. Wang, and M. D. Plumbley, "PANNs: Large-scale pretrained audio neural networks for audio pattern recognition," *IEEE/ACM Trans. Audio Speech Lang. Process.*, pp. 2880–2894, 2020.

[26] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, "BERT: Pre-training of deep bidirectional transformers for language understanding," in *Proc. Conf. North Am. Chapter Assoc. Comput. Linguist. Hum. Lang. Technol. (NAACL-HLT)*, 2019, pp. 4171–4186.

[27] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, "RoBERTa: A robustly optimized BERT pretraining approach," 2019. [Online]. Available: https://arxiv.org/abs/1907.11692

[28] A. van den Oord, Y. Li, and O. Vinyals, "Representation learning with contrastive predictive coding," 2018. [Online]. Available: https://arxiv.org/abs/1807.03748

[29] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, "A simple framework for contrastive learning of visual representations," in *Proc. Int. Conf. Mach. Learn. (ICML)*, 2020.

[30] R. Arandjelovic and A. Zisserman, "Look, listen and learn," in *Proc. Int. Conf. Comput. Vis. (ICCV)*, 2017.

[31] H. Chen, W. Xie, A. Vedaldi, and A. Zisserman, "VGGsound: A large-scale audio-visual dataset," in *Proc. IEEE Int. Conf. Acoustic., Speech and Signal Process. (ICASSP)*, 2020, pp. 721–725.

[32] S. Hershey, S. Chaudhuri, D. P. W. Ellis, J. F. Gemmeke, A. Jansen, R. C. Moore, M. Plakal, D. Platt, R. A. Saurous, B. Seybold, M. Slaney, R. J. Weiss, and K. Wilson, "CNN architectures for large-scale audio classification," in *Proc. IEEE Int. Conf. Acoustic., Speech and Signal Process. (ICASSP)*, 2017, pp. 131–135.

[33] M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer, "BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension," in *Proc. Annu. Meet. Assoc. Comput. Linguist. (ACL)*, 2020, pp. 7871–7880.

[34] N. Reimers and I. Gurevych, "Sentence-BERT: Sentence embeddings using Siamese BERT-networks," in *Proc. Emphirical Methods Nat. Lang. Process. (EMNLP)*, 2019, pp. 3982–3992.

[35] K. Koutini, J. Schlüter, H. Eghbal-zadeh, and G. Widmer, "Efficient training of audio transformers with patchout," 2021. [Online]. Available: https://arxiv.org/abs/2110.05069

[36] A. Guzhov, F. Raue, J. Hees, and A. Dengel, "ESResNet: Environmental sound classification based on visual domain models," in *Proc. Int. Conf. Pattern Recognit. (ICPR)*, 2021, pp. 4933–4940.

[37] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, "Learning transferable visual models from natural language supervision," in *Proc. Int. Conf. Mach. Learn. (ICML)*, 2021, pp. 8748–8763.

[38] K. Chen, X. Du, B. Zhu, Z. Ma, T. Berg-Kirkpatrick, and S. Dubnov, "HTS-AT: A hierarchical token-semantic audio transformer for sound classification and detection," in *Proc. IEEE Int. Conf. Acoustic., Speech and Signal Process. (ICASSP)*, 2022, pp. 646–650.

[39] P. Khosla, P. Teterwak, C. Wang, A. Sarna, Y. Tian, P. Isola, A. Maschinot, C. Liu, and D. Krishnan, "Supervised contrastive learning," in *Proc. Int. Conf. Neural Inf. Process. Syst. (NIPS)*, 2020, pp. 18661–18673.

[40] K. Sohn, "Improved deep metric learning with multi-class n-pair loss objective," in *Proc. Int. Conf. Neural Inf. Process. Syst. (NIPS)*, 2016, pp. 1857–1865.

[41] C. D. Kim, B. Kim, H. Lee, and G. Kim, "AudioCaps: Generating captions for audios in the wild," in *Proc. Conf. North Am. Chapter Assoc. Comput. Linguist. Hum. Lang. Technol. (HLT-NAACL)*, 2019, pp. 119–132.
# A MIXED SUPERVISED LEARNING FRAMEWORK FOR TARGET SOUND DETECTION

Dongchao Yang¹, Helin Wang¹, Wenwu Wang², Yuexian Zou¹*

¹ADSPLAB, School of ECE, Peking University, Shenzhen, China
²Center for Vision, Speech and Signal Processing, University of Surrey, UK

## ABSTRACT

Target sound detection (TSD) aims to detect the target sound from mixture audio given the reference information. Previous works have shown that TSD models can be trained on fully-annotated (frame-level label) or weakly-annotated (clip-level label) data. However, there are some clear evidences show that the performance of the model trained on weakly-annotated data is worse than that trained on fully-annotated data. To fill this gap, we provide a mixed supervision perspective, in which learning novel categories (target domain) using weak annotations with the help of full annotations of existing base categories (source domain). To realize this, a mixed supervised learning framework is proposed, which contains two mutually-helping student models (*f* *student* and *w* *student*) that learn from fully-annotated and weakly-annotated data, respectively. The motivation is that *f* *student* learned from fully-annotated data has a better ability to capture detailed information than *w* *student*. Thus, we first let *f* *student* guide *w* *student* to learn the ability to capture details, so *w* *student* can perform better in the target domain. Then we let *w* *student* guide *f* *student* to fine-tune on the target domain. The process can be repeated several times so that the two students perform very well in the target domain. To evaluate our method, we built three TSD datasets based on UrbanSound and Audioset. Experimental results show that our methods offer about 8% improvement in event-based F-score as compared with a recent baseline.

**Index Terms—** Target sound detection, audioset, weakly supervised, mixed supervised learning

## 1. INTRODUCTION

In target sound detection (TSD) [1], one aims to recognize and localize the target sound source within a mixture audio given a reference audio, e.g. detecting the dog bark sound within a street. TSD can be applied to numerous potential fields [2–4], such as species migration monitoring and large-scale multimedia indexing. Sound event detection (SED) [5] is a similar task with TSD, and a lot of works have been done for SED [6–11]. However, SED aims to classify and localize all pre-defined events (e.g., dog barking, man speaking) within an audio clip, which significantly limits the flexibility to detect unseen classes. Different with SED, TSD only focuses on detecting the event that we interest. TSD does not require pre-defined set of categories, therefore, it can be easily extended to sound detection from an open set. In a recent work, a target

<figure><img src="image_123.png" /><figcaption>Figure 1: (a) shows the predicted results generated by weakly (WS) and strongly supervised (SS) learning. (b) shows an example of altering the true label to false label. (c) shows the influence of the error rate of the frame-level label when we train TSDNet model [1] on the URBAN-TSD dataset.</figcaption></figure>

sound detection network (TSDNet) [1] is presented, where a conditional network is used to generate sound-discriminative embedding which is then used as the reference information to guide a detection network for the detection of the target sound from the mixture audio. TSDNet provides a good detection performance when training data is fully-annotated, e.g. the onset and offset time of the target sound are provided in the annotations. However, collecting large-scale fully-annotated data is time-consuming and labor-intensive. Weakly supervised TSD is an effective method to reduce the reliance on fully-annotated data, but the performance tends to degrade significantly [1].

In this paper, we consider TSD with mixed supervision, which learns novel sound categories (target domain) using weak annotations with the help of full annotations of the existing base sound categories (source domain). Under this setting, we can use a small-scale fully-annotated dataset (e.g. URBAN-SED [12]) to complement a large-scale weakly-annotated dataset (e.g. Audioset [13]). To achieve this, we propose a novel mixed supervised learning framework, which includes two mutually-helping student models (*f* *student* and *w* *student*), which are trained by fully- and weakly-annotated data, respectively. The proposed method involves three novel aspects. Firstly, the *f* *student* learned from fully-annotated data has better ability in capturing detailed information than the *w* *student*. As Figure 1 (a) shows, the model trained on weakly-annotated data fails to locate the event boundary: it only focuses on the most distinct part and misses the boundary information. Thus, we propose a frame-level knowledge distillation (KD) strategy to transfer the knowledge from *f* *student* to *w* *student*, which makes *w* *student* able to capture more details, e.g. boundary information.

<footnote>* Corresponding Author: zouyx@pku.edu.cn</footnote>

<footnote>This paper was partially supported by Shenzhen Science & Technology Research Program (No: GXWD20201231165807007-20200814115301001; No: JSGG20191129105421211) and NSFC (No: 62176008).</footnote>
However, it is hard to transfer all of the knowledge from *f* to *w- student*. Thus, we propose to directly apply *f- student* to the target domain with the guidance of *w- student*. Specifically, *w- student* is used to produce frame-level pseudo labels for *f- student*. This strategy inspired by an interesting phenomenon that even if there are some errors in the frame-level labels, the detection performance remains stable or decreases only slightly (when the error rate lower than 0.35), as Figure 1 (c) shows. The process of mutually helping can be repeated several times so that the two students perform very well in the target domain. Lastly, we found that the mismatch between source and target data distribution tends to affect significantly the performance of transfer learning, e.g. URBAN-SED [12] and Audioset [14]. Thus, we propose an adversarial training strategy to solve the domain mismatch problem. To evaluate our method, we built two small-scale fully-annotated datasets and a large-scale weakly-annotated dataset based on URBAN-SED and Audioset. Experimental results show that two small-scale fully annotated datasets could significantly improve the performance on large-scale weakly-annotated dataset.

## 2. RELATED WORK

Many methods [15–22] have been proposed to utilize both fully- and weakly-annotated data to train the SED model. However, previous methods assume that fully- and weakly- annotated data belong to the same set of pre-defined categories. Our method aims to use the existing base categories with full annotations to facilitate the recognition of novel categories with weak labels. This setting is more realistic for the reason that a small-scale fully-annotated dataset with few categories is, in practice, easier to create, as compared with a large-scale fully-annotated dataset of many categories.

## 3. TSD DATASETS

We built two TSD datasets based on Audioset [14], which includes 94126 training clips and 16118 test clips, from 456 different classes. One is the large-scaled TSD dataset, named as L-TSD dataset, by choosing 192 different classes from Audioset. The other is the small-scaled fully-annotated TSD dataset (S-TSD), by choosing 51 different classes from Audioset. L-TSD includes two types of annotated samples, that is, fully-annotated (i.e. L-TSD-strong) and weakly-annotated (i.e. L-TSD-weak) samples. There is no common class between L-TSD and S-TSD datasets. The process of building the datasets is similar to the one described in [1]. The mixture audio signals come from Audioset. If there are *N* sound events in the mixture audio, we can generate *N* positive samples. We also generate *N*/2 negative samples, which do not contain the target sound. To prepare reference audio, we select the audio clips for each category directly from the Audioset training set i.e. those that do not contain interference from other events. In total, L-TSD includes 490,336 training and 83,334 test clips, while S-TSD contains 26,247 training and 5113 test clips. In addition, we choose the URBAN-TSD dataset [1] as another small-scale fully-annotated dataset.

## 4. PROPOSED METHOD

Figure 2 shows our proposed mixed supervised learning framework. The core idea of the framework is that the two students can teach each other iteratively. One of the student models is trained on fully-annotated data, we name it as *f- student*. The other model is trained

<figure><img src="image_124.png" /><figcaption>Figure 2: The architecture of the mixed supervised learning framework. CL denotes classification layers, which includes two fully-connected layers and one softmax function. D denotes the discriminator, which consists of three convolutional layers and one fully-connected layer.</figcaption></figure>

on weakly-annotated data, and we name it as *w- student*. The two students have the same structure, while the only difference is that *w- student* has a linear softmax pooling layer [10].

### 4.1. Network Structure

**Conditional network.** The conditional network aims to extract a sound-discriminative embedding vector from the reference audio. Similar to the previous work [1], we adopt a VGG-like convolutional neural network (CNN) model [23] for the conditional network.

**Detection network.** Similar to the previous work [1], the network is composed of 5 convolutional layers, 1 Bi-GRU layer, and 2 fully-connected layers. Given the mel-spectrogram of the mixture audio $x \in \mathbb{R}^{T \times F}$, where *T* and *F* denote the number of frames and the dimension of frame. The detection network aims to predict frame-level probabilities

$$ \hat{p}_i = \mathbb{P}(Y = k | X = x_i, e; \phi) \quad (1) $$

where $\phi$ denotes the trainable parameters of the detection network, $e$ denotes the embedding obtained from the conditional network and $x_i$ denotes the *i*-th frame of the mixture audio $x$. For *f- student*, given the ground-truth label $p_i \in \{0, 1\}$ for each frame, which can be optimized by minimizing the binary cross entropy (BCE) loss:

$$ \mathcal{L}_s = \sum_{i=1}^{t} \left( -p_i \log \hat{p}_i - (1 - p_i) \log(1 - \hat{p}_i) \right) \quad (2) $$

where *t* indicates the number of frames. The difference between *f- student* and *w- student* is that the latter needs a pooling layer to get the clip-level prediction. Thus, a LinSoft pooling layer [10] is added after the last layer of the *f- student*. *w- student* aims to predict a clip-level probability $\hat{P} = f_{LSF}(\hat{p}_1, \hat{p}_2, ..., \hat{p}_t)$ where $f_{LSF}(\cdot)$ denotes the LinSoft pooling function. Given the clip-level ground-truth label $P \in \{0, 1\}$, the BCE loss is applied as the loss function:

$$ \mathcal{L}_w = -P \log \hat{P} - (1 - P) \log(1 - \hat{P}) \quad (3) $$

### 4.2. Two-student Learning

In this part, we introduce the details of how to enable *w- student* and *f- student* to help each other.

**Frame-level knowledge distillation.** According to formula (3), we
can see that w\_student makes a decision on the whole audio clip. Compared to f\_student, w\_student is limited in capturing the detailed information of the sound events. To address this issue, we pro- pose to first train f\_student on a small-scale fully-annotated dataset (i.e. source domain), and then transfer its knowledge to w\_student, so that w\_student can get better performance on weakly-annotated dataset (i.e. target domain). Specifically, we first train the f\_student model on the source dataset with strong labels. After that, we train the w\_student model on the source data with weak labels. We then treat the trained f\_student model as a teacher, to generate a frame- level feature representation. As a result, w\_student may capture more detailed information, due to the frame-level class-agnostic knowledge distillation. More specifically, we can train w\_student with the following objective function,

$$
\mathcal{L}_{wkd} = \mathcal{L}_w + \mathcal{L}_{kd} \qquad (4)
$$

$$
\mathcal{L}_{kd} = \|\mathbf{F}_s \cdot \mathbf{W}_s - \mathbf{F}_w \cdot \mathbf{W}_w\|_2 \quad (5)
$$

where **F**<sub>s</sub> and **F**<sub>w</sub> denote the feature map of the GRU layer of the two models, and **W**<sub>s</sub> and **W**<sub>w</sub> denote the transformation matrix.

**Pseudo Supervised Training.** The idea of pseudo supervised training strategy is motivated by an interesting observation, i.e. even if there are some errors in the frame-level labels, the detection performance remains stable or decreases only slightly. This means that we could use the noisy frame-level labels as our training target. In this paper, we propose to use w\_student to produce noisy frame-level labels (i.e. pseudo labels), and then use the pseudo labels to re-train f\_student, as follows

$$
\hat{p}_{i}^{w} = \mathbb{P}(Y = k | X = x_i, e; \phi_w), \hat{p}_{i}^{s} = \mathbb{P}(Y = k | X = x_i, e; \phi_s) \quad (6)
$$

$$
\mathcal{L}_{\text{re-s}} = \sum_{i=1}^{t} \left( -\hat{p}_{i}^{w} \log \hat{p}_{i}^{s} - (1 - \hat{p}_{i}^{w}) \log(1 - \hat{p}_{i}^{s}) \right) \quad (7)
$$

**Adversarial Training.** In our experiments, we found that the mismatch between source and target data distribution could significantly degrade the performance of the mixed supervised learning framework. For example, if we choose the URBAN-TSD as the source dataset, and the L-TSD-weak as the target dataset, the performance will decrease substantially. This is because there is domain mismatch between URBAN-TSD and L-TSD-weak datasets [24, 25], i.e. *f*\_student and *w*\_student are first trained on the URBAN-TSD dataset but tested on L-TSD-weak dataset. To solve the domain mismatch problem, we propose a domain adversarial training strategy that aims to learn a common subspace shared by both the source and target domains, which enables all domains to have the same data distribution in the feature space. Specifically, inspired by GAN [26] and DANN [27], we make use of the adversarial relationship between modules Feature extractor (F) and Discriminator (D) to learn domain-invariant features in the feature space. To achieve this, we add an adversarial loss when we train *f*\_student and *w*\_student, as follows

$$
\mathcal{L}_d = \|D(\boldsymbol{z}) - \boldsymbol{d}\|_2^2 \qquad (8)
$$

$$
\mathcal{L}_{d,tsd} = \mathcal{L}_{tsd} - \lambda_d * \mathcal{L}_d \qquad (9)
$$

where z denotes the intermediate feature produced by F, and d denotes the domain label. The domain label is defined as d = [1, 0]<sup>T</sup> (which stands for the source domain) or d = [0, 1]<sup>T</sup> (target domain). L<sub>d</sub> denotes the domain classification loss of the discriminator, L<sub>tsd</sub> denotes the detection loss. L<sub>d_tsd</sub> denotes the training objective function, which minimizes the detection loss and means while maximizes the domain classification loss. The parameter λ<sub>d</sub>

controls the trade-off between L<sub>tsd</sub> and L<sub>d</sub>. In our experiments, λ<sub>d</sub> is set to 0.2 empirically based on the validation set.

**Iterative Training Strategy.** According to the previous description, we can use frame-level KD to transfer the knowledge from *f*\_student to *w*\_student, and use the pseudo supervised training to transfer the knowledge from *w*\_student to *f*\_student. Intuitively, the process can be repeated several times. Thus, an iterative training strategy is proposed. The whole algorithm is summarized in Algorithm 1.

**Algorithm 1 Two-Student Learning**

Input:

The source dataset D<sub>s</sub> and the target dataset D<sub>t</sub>

Output: *f*\_student and *w*\_student model

1: Training *f*\_student on *D*<sub>s</sub> using formula (2) and (9)

2: Training *w*\_student on *D*<sub>s</sub> using formula (4) and (9)

3: Retraining *w*\_student on *D*<sub>t</sub> using formula (3) and (9)

4: Retraining *f*\_student on *D*<sub>t</sub> using formula (7) and (9)

5: While True:

Retraining *w*\_student on *D*<sub>t</sub>, using formula (4)

Retraining *f*\_student on *D*<sub>t</sub> using formula (7)

If no improvement: break;

6: return *f*\_student and *w*\_student;

5. EXPERIMENTS

5.1. Datasets

In this section, we introduce the source dataset and target dataset used in our experiments. The source (resp., target) dataset is fully-annotated (resp., weakly-annotated).

**Source Dataset.** We first use the S-TSD dataset as the source dataset, which is a small-scale fully-annotated dataset based on Au-dioset [14]. In addition, we choose 10-category URBAN-TSD [1] as another source dataset, which includes two similar categories as in the L-TSD dataset: *dog_bark* and *gun shot*.

**Target Dataset.** We take the L-TSD-weak dataset as the target dataset. The details were given in Section 3.

5.2. Experimental Setups

**Conditional Network.** We use the pre-trained PANNs [23] model to initialize the conditional network, and then fix it in the training process.

**Detection network.** All the raw audios are down-sampled to 22.05kHz and then Short Time Fourier Transform (STFT) with a window size of 2048 samples are applied, followed by a Mel-scaled filter bank on perceptually weighted spectrogram. This results in 64 Mel frequency bins and around 50 frames per second. When training *f*\_student and *w*\_student for the first time, the Adam optimizer [28] is used for 100 epochs, with an initial learning rate of 1 × 10<sup>-3</sup>. When they are re-trained, the learning rate is set as 1 × 10<sup>-4</sup>.

**Metrics.** We use the segment-based F-score and event-based F-score [29] as the evaluation metrics, which are the most commonly used metrics for detection task.
<table><caption>Table 1: F-score comparison with different supervision strategy on L-TSD test set. SS, WS and MS represent strong, weak and mixed supervision, respectively. F-scores are macro-averaged.</caption><thead><tr><th>Method</th><th>Source dataset</th><th>Segment-F score</th><th>Event-F score</th></tr></thead><tbody><tr><td>SS [1]</td><td>-</td><td>58.57</td><td>50.4</td></tr><tr><td>WS [1]</td><td>-</td><td>49.39</td><td>39.07</td></tr><tr><td rowspan="2">MS (ours)</td><td>S-TSD</td><td>50.95</td><td>47.19</td></tr><tr><td>URBAN-TSD</td><td>51.31</td><td>47.56</td></tr></tbody></table>

<table><caption>Table 2: Ablation studies on different strategies on L-TSD test set.</caption><thead><tr><th>Model</th><th>KD</th><th>PS</th><th>AD</th><th>Segment-F score</th><th>Event-F score</th></tr></thead><tbody><tr><td rowspan="4">w student</td><td>X</td><td>X</td><td>X</td><td>49.39</td><td>39.07</td></tr><tr><td>X</td><td>X</td><td>✓</td><td>49.34</td><td>39.11</td></tr><tr><td>✓</td><td>X</td><td>X</td><td>37.55</td><td>39.74</td></tr><tr><td>✓</td><td>X</td><td>✓</td><td>50.34</td><td>41.35</td></tr><tr><td rowspan="3">f student</td><td>X</td><td>✓</td><td>✓</td><td>48.09</td><td>43.47</td></tr><tr><td>✓</td><td>✓</td><td>X</td><td>37.67</td><td>45.42</td></tr><tr><td>✓</td><td>✓</td><td>✓</td><td><b>51.31</b></td><td><b>47.56</b></td></tr></tbody></table>

## 5.3. Experiments on S-TSD and L-TSD Datasets

In this section, we use S-TSD as the source dataset and L-TSD-weak as the target dataset. We compare our method with strongly supervised (SS) and weakly supervised (WS) methods. For SS-TSD, we directly train *f student* on the L-TSD-strong dataset with the strong labels. For WS-TSD, we directly train *w student* on the L-TSD-weak dataset with the weak labels. Note that for mixed supervised (MS) method, we only report the results obtained by *f student* for the reason that *f student* performs better than *w student*. The experimental results are given in Table 1. From this table, we can see that our proposed MS method performs significantly better than the WS method, and performs similarly to the SS method.

## 5.4. Experiments on URBAN-TSD and L-TSD Datasets

We also conduct experiments by using URBAN-TSD as the source dataset and L-TSD-weak as the target dataset. Table 1 shows the experimental results, and we can see that using URBAN-TSD as the source dataset significantly improves the performance compared with the WS method. Furthermore, by comparing rows 3 and 4, we can find that using URBAN-TSD as the source dataset obtains better performance than using S-TSD as the source dataset. One of the reasons is the categories of URBAN-TSD and L-TSD-weak datasets have overlaps.

## 5.5. Ablation Studies

By taking URBAN-TSD as the source dataset, we conduct ablation studies to investigate the effectiveness of knowledge distillation (KD), pseudo supervised training (PS) and adversarial (AD) training, with the results shown in Table 2. For the *w student* model: (1) The first row shows the results for directly training *w student* on L-TSD-weak (without using any strategy). (2) The second row shows the results of directly training *w student* on L-TSD-weak while using the AD strategy. We can see that only using the AD strategy does not give improvements, because it only aims to align the data distribution. (3) By comparing rows 2 and 4, we can see that the

<table><caption>Table 3: Ablation study on the effect of the number of iterations on iterative training strategy.</caption><thead><tr><th>Iterations</th><th>Model</th><th>Segment-F score</th><th>Event-F score</th></tr></thead><tbody><tr><td rowspan="2">1</td><td>w student</td><td>50.39</td><td>40.88</td></tr><tr><td>f student</td><td>50.21</td><td>46.99</td></tr><tr><td rowspan="2">2</td><td>w student</td><td>50.14</td><td>41.60</td></tr><tr><td>f student</td><td>51.10</td><td>46.96</td></tr><tr><td rowspan="2">3</td><td>w student</td><td>51.33</td><td>44.70</td></tr><tr><td>f student</td><td>50.95</td><td>47.19</td></tr></tbody></table>

KD strategy can improve the *w student*'s ability in capturing detailed information. (4) By comparing rows 3 and 4, we can see the effectiveness of the AD strategy when there is mismatch between the source and target datasets.

For the *f student* model: (1) By comparing rows 5 and 7, we can see that the KD strategy can improve the performance of *f student* for the reason that it can improve the performance of *w student*, and the better *w student* leads to better *f student*. (2) By comparing rows 6 and 7, we can find the improvements given by the AD strategy. Lastly, we can see that *f student* has better performance than *w student*. A reason could be that the ability of *f student* in capturing the detailed information is only partially transferred to *w student* through the KD strategy.

**Influence of the number of iterations.** By taking S-TSD as the source dataset, we conduct ablation studies to investigate the influence of the number of iterations on the iterative training strategy, and the results are shown in Table 3. In this work, considering the cost of training time, we only tested three stages. For *f student*, the segment- and event-based F-scores are increased from 50.21% and 46.99% to 50.95% and 47.19%, respectively. Furthermore, the event-based F-score of *w student* is increased from 40.88% to 44.7%. It means that the two students can help each other.

**Why does pseudo supervised training work?** To explain why the pseudo supervised training strategy is effective, we calculate the error rate between the true label and pseudo label (produced by *w student*) on the L-TSD-strong training set. We found that the average error rate is 27.03%, over the 490,336 audio clips. This is consistent with our empirical results as shown earlier in Figure 1: if the error rate is smaller than 0.35, the performance is similar to that of the case where the true label is taken as the target.

# 6. CONCLUSIONS

In this paper, we have presented a novel mixed supervised learning framework, which effectively improves the performance of novel categories with the help of a small-scale fully-annotated base categories dataset. In the future, we will apply our method to other tasks, such as SED and object detection. The source code and dataset of this work have been released<fnref n="1" />.

# 7. REFERENCES

[1] D. Yang, H. Wang, Y. Zou, and C. Weng, “Detect what you want: Target sound detection,” *arXiv preprint arXiv:2112.10153*, 2021.

[2] J. P. Bello, C. Silva, O. Nov, R. DuBois, A. Arora, J. Salamon, C. Mydlarz, and H. Doraiswamy, “Sonyc: A system for the

<footnote n="1">¹https://github.com/yangdongchao/weakly-target-sound-detection</footnote>
monitoring, analysis and mitigation of urban noise pollution,” arXiv preprint arXiv:1805.00889, 2018.

[3] D. Stowell and D. Clayton, “Acoustic event detection for multiple overlapping similar sources,” in *IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA)*. IEEE, 2015, pp. 1–5.

[4] S. Hershey, S. Chaudhuri, D. Ellis, J. Gemmeke, A. Jansen, R. C. Moore, M. Plakal, D. Platt, R. A. Saurous, B. Seybold, et al., “CNN architectures for large-scale audio classification,” in *IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. IEEE, 2017, pp. 131–135.

[5] Q. Kong, Y. Xu, I. Sobieraj, W. Wang, and M. D. Plumbley, “Sound event detection and time-frequency segmentation from weakly labelled data,” *IEEE/ACM Transactions on Audio, Speech, and Language Processing*, vol. 27, pp. 777–787, 2019.

[6] H. Dinkel, M. Wu, and K. Yu, “Towards duration robust weakly supervised sound event detection,” *IEEE/ACM Transactions on Audio, Speech, and Language Processing*, vol. 29, pp. 887–900, 2021.

[7] L. Lin, X. Wang, H. Liu, and Y. Qian, “Specialized decision surface and disentangled feature for weakly-supervised polyphonic sound event detection,” *IEEE/ACM Transactions on Audio, Speech, and Language Processing*, vol. 28, pp. 1466–1478, 2020.

[8] Q. Kong, Y. Xu, W. Wang, and M. D. Plumbley, “Sound event detection of weakly labelled data with cnn-transformer and automatic threshold optimization,” *IEEE/ACM Transactions on Audio, Speech, and Language Processing*, vol. 28, pp. 2450–2460, 2020.

[9] A. Mesaros, T. Heittola, T. Virtanen, and M. D. Plumbley, “Sound event detection: A tutorial,” *IEEE Signal Processing Magazine*, vol. 38, no. 5, pp. 67–83, 2021.

[10] Y. Wang, J. Li, and F. Metze, “A comparison of five multiple instance learning pooling functions for sound event detection with weak labeling,” in *IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. IEEE, 2019, pp. 31–35.

[11] I. Martín-Morató, A. Mesaros, T. Heittola, T. Virtanen, M. Cobos, and F. Ferri, “Sound event envelope estimation in polyphonic mixtures,” in *IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. IEEE, 2019, pp. 935–939.

[12] J. Salamon, D. MacConnell, M. Cartwright, P. Li, and J. P. Bello, “Scaper: A library for soundscape synthesis and augmentation,” in *IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA)*. IEEE, 2017, pp. 344–348.

[13] J. Gemmeke, D. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, “Audio set: An ontology and human-labeled dataset for audio events,” in *IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. IEEE, 2017, pp. 776–780.

[14] S. Hershey, D. P. Ellis, E. Fonseca, A. Jansen, C. Liu, R. C. Moore, and M. Plakal, “The benefit of temporally-strong labels in audio event classification,” in *IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. IEEE, 2021, pp. 366–370.

[15] A. Kumar and B. Raj, “Audio event and scene recognition: A unified approach using strongly and weakly labeled data,” in *International Joint Conference on Neural Networks (IJCNN)*. IEEE, 2017, pp. 3475–3482.

[16] Y. Liang, Y. Long, Y. Li, and J. Liang, “Joint weakly supervised AT and AED using deep feature distillation and adaptive focal loss,” *arXiv preprint arXiv:2103.12388*, 2021.

[17] L. Lin, X. Wang, H. Liu, and Y. Qian, “Guided learning for weakly-labeled semi-supervised sound event detection,” in *IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. IEEE, 2020, pp. 626–630.

[18] Z. Shi, L. Liu, H. Lin, R. Liu, and A. Shi, “Hodgepodge: Sound event detection based on ensemble of semi-supervised learning methods,” *arXiv preprint arXiv:1907.07398*, 2019.

[19] T. K. Chan, C. S. Chin, and Y. Li, “Non-negative matrix factorization-convolutional neural network (nmf-cnn) for sound event detection,” *arXiv preprint arXiv:2001.07874*, 2020.

[20] X. Zheng, Y. Song, I. McLoughlin, L. Liu, and L.-R. Dai, “An improved mean teacher based method for large scale weakly labeled semi-supervised sound event detection,” in *IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. IEEE, 2021, pp. 356–360.

[21] L. Cances and T. Pellegrini, “Comparison of deep co-training and mean-teacher approaches for semi-supervised audio tagging,” in *IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. IEEE, 2021, pp. 361–365.

[22] Y. Guan, J. Xue, G. Zheng, and J. Han, “Sparse self-attention for semi-supervised sound event detection,” in *IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. IEEE, 2022, pp. 821–825.

[23] Q. Kong, Y. Cao, T. Iqbal, Y. Wang, W. Wang, and M. Plumbley, “Panns: Large-scale pretrained audio neural networks for audio pattern recognition,” *IEEE/ACM Transactions on Audio, Speech, and Language Processing*, vol. 28, pp. 2880–2894, 2020.

[24] S. J. Pan and Q. Yang, “A survey on transfer learning,” *IEEE Transactions on Knowledge and Data Engineering*, vol. 22, no. 10, pp. 1345–1359, 2009.

[25] S. Ben-David, J. Blitzer, K. Crammer, and F. Pereira, “Analysis of representations for domain adaptation,” *Advances in Neural Information Processing Systems*, vol. 19, 2006.

[26] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial networks,” *arXiv preprint arXiv:1406.2661*, 2014.

[27] R. Wang, M. Wang, X.-L. Zhang, and S. Rahardja, “Domain adaptation neural network for acoustic scene classification in mismatched conditions,” in *Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC)*. IEEE, 2019, pp. 1501–1505.

[28] D. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in *ICLR*, 2015.

[29] A. Mesaros, T. Heittola, and T. Virtanen, “Metrics for polyphonic sound event detection,” *Applied Sciences*, vol. 6, no. 6, p. 162, 2016.