<table><caption>Table 2: F-score results per team (best scoring system) on evaluation and validation sets, and summary of system characteristics. Systems are ordered by higher scoring rank on the evaluation set. These results and technical reports for the submitted systems can be found on task 5 results page [17].</caption><thead><tr><th>Team code</th><th>Code</th><th>Eval set:<br>F-score % (95% CI)</th><th>Val set<br>F-score %</th><th>Main characteristics</th></tr></thead><tbody><tr><td>Du_NERCSLIP</td><td>(A)</td><td>60.22 (59.66-60.70)</td><td>74.4</td><td>CNN+ProtoNet; Frame-level embeddings; PCEN;</td></tr><tr><td>Liu_Surrey</td><td>(B)</td><td>48.52 (48.18-48.85)</td><td>50.03</td><td>CNN+ProtoNet; extra data; PCEN+ΔMFCC; various post-process.</td></tr><tr><td>Martinsson_RISE</td><td>(C)</td><td>47.97 (47.48-48.40)</td><td>60</td><td>ResNet+ProtoNet; Ensemble(15) based input size; logMel+PCEN</td></tr><tr><td>Hertkorn_ZF</td><td>(D)</td><td>44.98 (44.44-45.42)</td><td>61.76</td><td>CNN; Frequency resolution preserving pooling; various post-process</td></tr><tr><td>Liu_BIT-SRCB</td><td>(E)</td><td>44.26 (43.85-44.62)</td><td>64.77</td><td>CNN+ProtoNet; Transductive inference</td></tr><tr><td>Wu_SHNU</td><td>(F)</td><td>40.93 (40.48-41.30)</td><td>53.88</td><td>ResNet+ProtoNet; Continual-learning; spectrogram</td></tr><tr><td>Zgorzynski_SRPOL</td><td>(G)</td><td>33.24 (32.69-33.69)</td><td>57.2</td><td>CNN+Siamese Networks; Ensemble (3) average event-length;</td></tr><tr><td>Mariajohn_DSPC</td><td>(H)</td><td>25.66 (25.40-25.91)</td><td>43.89</td><td>CNN+ProtoNet; logMel; augmentation with time-shifting and mirroring</td></tr><tr><td>Wilbo_RISE</td><td>(I)</td><td>21.67 (21.32-21.97)</td><td>47.94</td><td>ResNet+ProtoNet; Semi-supervised; Melspect+PCEN; various post-process</td></tr><tr><td>Zou_PKU</td><td>(J)</td><td>19.20 (18.88-19.51)</td><td>51.99</td><td>CNN+protoNet; mutual information loss; time frequency masking + mixup</td></tr><tr><td>Huang_SCUT</td><td>(K)</td><td>18.29 (18.01-18.56)</td><td>54.63</td><td>Transductive inference + Adapted central difference convolution</td></tr><tr><td>Tan_WHU</td><td>(L)</td><td>17.22 (16.82-17.55)</td><td>54.53</td><td>CNN+ProtoNet pretrained; transductive inference; task adaptive features</td></tr><tr><td>Li_QMUL</td><td>(M)</td><td>15.49 (15.16-15.77)</td><td>47.88</td><td>CNN+protoNet; PCEN; time, frequency masking + time warping</td></tr><tr><td>baseline-TempMatch</td><td>[5]</td><td>12.35 (11.52-12.75)</td><td>3.37</td><td>Spectrogram Cross correlation</td></tr><tr><td>baseline-ProtoNet</td><td>[5]</td><td>5.3 (5.1-5.2)</td><td>28.45</td><td>ResNet+ProtoNet</td></tr><tr><td>Zhang_CQU</td><td>(N)</td><td>4.34 (3.74-4.56)</td><td>44.17</td><td>CNN+protoNet; Fine tuning with MIMI; PCEN</td></tr><tr><td>Kang_ET</td><td>(O)</td><td>2.82 (2.76-2.87)</td><td>-</td><td>CNN+ProtoNet; pretrained ECAPA-TDNN; Fine-tuning; Specaugment</td></tr></tbody></table>

<figure><img src="image_86.png" /><figcaption>Figure 1: F-Score results by dataset. Systems are ordered by highest scoring rank on the evaluation set.</figcaption></figure>

FN - ground truth events that are not predicted. Matches to UNK events are ignored from these counts as to not negatively impact the systems that predict these events. Finally, the F-score metric is computed per dataset in the evaluation set and the harmonic mean over all is reported.

## 4. RESULTS

For the 2022 edition, 15 teams participated submitting a total of 46 systems. The results for the highest scoring submission for each team are presented in Table 2, together with the reported F-scores on the validation set and summary of the system characteristics. Fig. 1 presents the F-scores obtained by each team on each subset of the evaluation data. The majority of systems adopted a prototypical network approach. Similar to last year's results, simple improvements over the baselines were achieved by applying data augmentation techniques and intelligent post-processing.

Better ways to construct the negative prototype were also explored by some teams who report improved results (B, C, F, I). Transductive inference, the method used by the past edition's winning team, was also applied here by several participants (B, M, L, J). The highest scoring system implements a frame-level embedding learning approach which confers to the system a high time resolution capability (A). The system was particularly effective on the QU and MGE dataset (Fig.1). This confirms that good time precision is fundamental, particularly for classes with events of very short duration as the ones in these datasets. The system ranked in second place implements a novel approach designed to optimise the contrast between positive events and negative prototypes (B). This, together with an adaptive segment length dependent on each target class, works well across all the evaluation sets. The problem of very different lengths of events across target classes was also directly addressed by other submissions. Both (C) and (G) implemented an ensemble approach where each individual model focuses on a different input size range. In (E) this is explored through a multi-scale ResNet, and in (I) with a wide ResNet containing many channels. Finally, it is worth mentioning the system in (D). Their few-shot adaptation was based on fine-tuning alone. The innovation here is related to simple modifications to a CNN-based architecture in order to optimise the use of information, particularly in the frequency axis. Furthermore, by allowing the network to overfit (up to a degree) to the 5 shots, the system achieves surprisingly good performance across all the datasets of the evaluation set.

Overall, this edition saw some novel ideas being implemented that tried to address previously identified challenges related to this task: how to deal with very different event lengths; how to construct a negative class when no explicit labels are given for this; and how to bridge the gap between classification and detection for few-shot sound event detection. We believe these remain relevant questions for our goal and for SED in general, and that the collective work developed here helped pushing few-shot bioacoustic sound event detection into DCASE central stage.
## 5. REFERENCES

[1] D. Stowell, "Computational bioacoustics with deep learning: a review and roadmap," *PeerJ*, vol. 10, 2022.

[2] W.-P. Vellinga and R. Planqué, "The xeno-canto collection and its relation to sound recognition and classification," in *CLEF (Working Notes)*, 2015.

[3] A. E. Méndez Méndez, M. Cartwright, and J. P. Bello, "Machine-crowd-expert model for increasing user engagement and annotation quality," in *Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems*, 2019, pp. 1-6.

[4] J. Snell, K. Swersky, and R. Zemel, "Prototypical networks for few-shot learning," *Advances in neural information processing systems*, vol. 30, 2017.

[5] V. Morfi, I. Nolasco, V. Lostanlen, S. Singh, A. Strandburg-Peshkin, L. F. Gill, H. Pamula, D. Benvent, and D. Stowell, "Few-shot bioacoustic event detection: A new task at the dcase 2021 challenge." in *DCASE*, 2021, pp. 145-149.

[6] Y.-X. Wang, R. B. Girshick, M. Hebert, and B. Hariharan, "Low-shot learning from imaginary data," *2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 7278-7286, 2018.

[7] A. Nichol, J. Achiam, and J. Schulman, "On first-order meta-learning algorithms," *ArXiv*, vol. abs/1803.02999, 2018.

[8] Y. Wang, J. Salamon, M. Cartwright, N. J. Bryan, and J. P. Bello, "Few-shot drum transcription in polyphonic music," *CoRR*, vol. abs/2008.02791, 2020.

[9] Y. Shiu, K. Palmer, M. A. Roch, E. Fleishman, X. Liu, E.-M. Nosal, T. Helble, D. Cholewiak, D. Gillespie, and H. Klinck, "Deep neural networks for automated detection of marine mammal species," *Scientific reports*, vol. 10, no. 1, pp. 1-12, 2020.

[10] P. Wolters, C. Daw, B. Hutchinson, and L. Phillips, "Proposal-based few-shot sound event detection for speech and environmental sounds with perceivers," *arXiv preprint arXiv:2107.13616*, 2021.

[11] D. Yang, H. Wang, Z. Ye, and Y. Zou, "Few-shot bioacoustic event detection = a good transductive inference is all you need," *DCASE2021 Challenge, Tech. Rep.*, June 2021.

[12] K. D. S. Lehmann, "Communication and cooperation in silico and nature," Ph.D. dissertation, Michigan State University, 2020.

[13] J. Gómez-Gómez, E. Vidaña-Vila, and X. Sevillano, "Western mediterranean wetlands bird species classification: evaluating small-footprint deep learning approaches on a new annotated dataset," 2022. [Online]. Available: https://arxiv.org/abs/2207.05393

[14] I. Kiskin, D. Zilli, Y. Li, M. Sinka, K. Willis, and S. Roberts, "Bioacoustic detection with wavelet-conditioned convolutional neural networks," *Neural Computing and Applications*, vol. 32, no. 4, pp. 915-927, 2020.

[15] I. Kiskin, M. Sinka, A. D. Cobb, W. Rafique, L. Wang, D. Zilli, B. Gutteridge, R. Dam, T. Marinos, Y. Li, et al., "Humbugdb: a large-scale acoustic mosquito dataset," *arXiv preprint arXiv:2110.07607*, 2021.

[16] Y. Li, I. Kiskin, M. Sinka, D. Zilli, H. Chan, E. Herreros-Moya, T. Chareonviriyaphap, R. Tisgratog, K. Willis, and S. Roberts, "Fast mosquito acoustic detection with field cup recordings: an initial investigation." in *DCASE*, 2018, pp. 153-157.

[17] "DCASE challenge 2022 Few-shot bioacoustic event detection task - results page," accessed: 2022-09-27. [Online]. Available: https://dcase.community/challenge2022/task-few-shot-bioacoustic-event-detection-results

[18] J. E. Hopcroft and R. M. Karp, "An n5/2 algorithm for maximum matchings in bipartite graphs," *SIAM J. Comput.*, vol. 2, pp. 225-231, 1973.
# EXPLORING ECO-ACOUSTIC DATA WITH K-DETERMINANTAL POINT PROCESSES

Mohamed Outidrarine¹, Pierre Baudet¹, Vincent Lostanlen¹*,

Mathieu Lagrange¹, Juan Sebastián Ulloa²

¹ Nantes Université, École Centrale Nantes, CNRS, LS2N, UMR 6004, F-44000 Nantes, France

² Instituto de Investigación de Recursos Biológicos Alexander von Humboldt, Bogotá, Colombia vincent.lostanlen@ls2n.fr

## ABSTRACT

The deployment of acoustic sensor networks in a natural environment contributes to the understanding and the conservation of biodiversity. Yet, the sheer size of audio data which result from these recordings prevents listening them in full. In order to skim through an eco-acoustic corpus, one may typically draw K snippets uniformly at random. In this article, we present an alternative method, based on K-determinantal point processes (K-DPP). This method weights the sampling of K-tuples according to a two-fold criterion of relevance and diversity. To study the eco-acoustics of a tropical dry forest in Colombia, we define relevance in terms of time-frequency second derivative (TFSD) and diversity in terms of scattering transform. Hence, we show that K-DPP offers a better tradeoff than K-means clustering. Furthermore, we estimate the species richness of the K selected snippets by means of the BirdNET birdsong classifier, which is based on a deep neural network. We find that, for K > 10, K-DPP and K-means tend to produce a species checklist that is richer than sampling K snippets independently without replacement.

**Index Terms—** Acoustic sensor networks, Bird species classification, Scattering transform, Time-frequency second derivative

## 1. INTRODUCTION

The science of eco-acoustics aims to analyze outdoor acoustic scenes so as to characterize certain ecological processes, such as population dynamics, species assemblages, and the emergence of a sonic landscape or “soundscape” [1]. This young field of research contributes to the overarching goal of biodiversity conservation at the planetary scale. Yet, a supervised classifier of eco-acoustic events may only be deployed once defined a research hypothesis, a taxonomy of sounds of interest, and a training set with expert or crowdsourced annotation.

Hence, the mass collection of eco-acoustic signals raises a problem in exploratory data analysis: how to quickly skim through a given corpus of N audio snippets without listening to it in full? In this context, the naive approach consists in drawing a subcorpus X of K ≪ N equiprobable samples. One may refine this method by weighting the probability of drawing each signal xᵢ in the corpus by a “relevance” prior qᵢ. Intuitively, the audio feature qᵢ is designed so as to quantify the saliency of spectrotemporal modulations in the time-frequency domain, such as animal vocalizations. However, this reweighted approach incurs a form of selection bias in terms of species richness: the top-K values of the saliency measure qᵢ typically belong to much fewer than K distinct species. This is because

few species will exhibit calls with high saliency while most species will only exhibit calls with low-to-moderate saliency. The former tend to be over-represented in X, at the detriment of the latter.

In this article, we propose a method for sampling audio signals according to a probabilistic tradeoff between relevance and diversity. The key idea is to represent each xᵢ by a vector Φᵢ of norm $\sqrt{q_i}$ and such that the angles $\angle(\Phiᵢ, \Phiⱼ)$ approximate the auditory dissimilarity of the pair (xᵢ, xⱼ). Thus, we draw the subcorpus X with a probability which is proportional to the determinant of the indexed family Φᵢ of vectors xᵢ ∈ X. Because of this proportionality, the proposed method is known as K-determinantal point process, or K-DPP for short.

There is a growing body of literature on the topic of applying (K-DPP to various machine learning problems, such as image search and graph threading in a document collection: we refer to [2] for an introduction. However, no application of K-DPP to machine listening has been published until today. Hence, the novel contribution of our paper is to offer a first proof of concept which demonstrates its interest for diverse sampling in eco-acoustics. A recent article [3] has employed a K-DPP in order to extract a subcorpus of urban acoustic scenes as part of a human annotation campaign; but the authors used a constant relevance term qᵢ = 1 and did not evaluate their approach.

As a first illustration, Figure 1 illustrates different sampling methods with K = 3 for eco-acoustics signals from a dry tropical forest. Qualitatively speaking, we notice that uniform independent draws (row 1) tend to lack in relevance, with some samples containing only high-frequency insect stridulations. Relevance weighting (row 2) mitigates this problem at the cost of a reduction in diversity: the most salient species tend to recur disproportionately. Clustering with K-means (row 3) restores diversity at the detriment of relevance. Lastly, K-DPP (row 4) seems to offer an interesting tradeoff between relevance and diversity.

## 2. DETERMINANTAL POINT PROCESS

### 2.1. Time-frequency second derivative (TFSD)

We decompose each signal xᵢ, of same duration T, by means of a constant-Q wavelet filters $\psi_{\lambda_1}$, with $\lambda_1 \in \Lambda$. We apply pointwise complex modulus and denote by $U_1x_i(t, \lambda_1) = |x_i * \psi_{\lambda_1}|(t)$ the resulting scalogram. representation. Then, we compute the first-order difference of $U_1x_i$ over both variables t and $\lambda_1$. Hence:

$$ \text{TFSD}(x_i)(t, \lambda_1) = U_1x_i(t + \tau, \lambda_1 + \delta) + U_1x_i(t, \lambda_1) \\ - U_1x_i(t + \tau, \lambda_1) - U_1x_i(t, \lambda_1 + \delta), \quad (1) $$

<footnote>*This work is supported by an Atlanstic 2020 award on “Trainable Acoustic Sensors” (TrAcS).</footnote>
<figure><img src="image_87.png" /><figcaption>Figure 1: Short-term Fourier spectrograms of three samples from the dataset under study, drawn from four different random sampling methods: uniform, relevance-based, K-means, and K-DPP.</figcaption></figure>

where the acronym TFSD stands for Time–Frequency Second Derivative. In practice, we set the hop size $\tau$ to 23 ms, the quality factor to $Q = 3$, the relative frequency interval $\delta$ to one third of an octave. We define a region of interest $\Lambda' \subset \Lambda$, corresponding to the third-octave bands $\lambda_1$ in $U\mathbf{x}_i$ which range between 2 and 8 kHz. This region corresponds to the vocal range of most singing species of birds. Lastly, we sum the absolute values of the TFSD($\mathbf{x}_i$) matrix over both regions $\Lambda$ and $\Lambda'$. Their ratio provides a measure of relevance

$$q_i = \frac{\int_0^T \int_{\Lambda'} |\text{TFSD}(\mathbf{x}_i)(t, \lambda_1)| \, dt \, d\lambda_1}{\int_0^T \int_{\Lambda} |\text{TFSD}(\mathbf{x}_i)(t, \lambda_1)| \, dt \, d\lambda_1} \quad (2)$$

between zero and one. A recent study in an urban environment [4] has shown that the descriptor $q_i$ (defined with a slight variation in spectrotemporal parameters) significantly correlates with the perceived time of presence of bird vocalizations.

## 2.2. Time scattering transform

The time scattering transform, also known as deep scattering spectrum [5], is a nonlinear convolutional operator in the time–frequency domain whose role is to extract amplitude modulations at various

time scales in each wavelet subband. First, we integrate the scalogram $U_1\mathbf{x}_i$ along time $t$, which gives the averaged scalogram:

$$S_1\mathbf{x}_i(\lambda_1) = \int_0^T U_1\mathbf{x}_i(t, \lambda_1) \, dt, \quad (3)$$

also known as first-order scattering. The transformation from $U_1$ to $S_1$ guarantees a property of global invariance, which comes at the cost of a loss in discriminability: $S_1\mathbf{x}_i(\lambda_1)$ ignores the amplitude modulations in each wavelet subband $U_1\mathbf{x}_i(t, \lambda_1)$ around its average value, and so for every fixed $\lambda_1$. The key idea behind the scattering transform is to recover these amplitude modulations by means of a second filter bank of constant-$Q$ wavelet filters $\psi_{\lambda_2}$, except with a quality factor of $Q = 1$ now. We obtain the so-called amplitude modulation spectrum of $\mathbf{x}_i$:

$$U_2\mathbf{x}_i(t, \lambda_1, \lambda_2) = |U_1\mathbf{x}_i * \psi_{\lambda_2}|(t, \lambda_1), \quad (4)$$

where the convolution between scalogram and second-order wavelet is performed over time $t$, and broadcasted across all frequencies $\lambda_1$. Symmetrically to $U_1$ et $S_1$, we integrate $U_2$ along time to obtain the matrix:

$$S_2\mathbf{x}_i(\lambda_1, \lambda_2) = \int_0^T U_2\mathbf{x}_i(t, \lambda_1, \lambda_2) \, dt. \quad (5)$$

We concatenate the first-order coefficients $S_1\mathbf{x}_i$ with the flattened matrix $S_2\mathbf{x}_i$ so as to obtain a high-dimensional vector $S\mathbf{x}_i$, which is generically indexed by the “scattering path” multiindex $p$, i.e., either the singleton $\lambda_1$ or to the pair $(\lambda_1, \lambda_2)$ depending on order.

Time scattering approximately verifies a property of energy conservation, similarly to the Parseval identity for the Fourier transform. Therefore, dividing the vector $S\mathbf{x}_i$ by its $\ell^2$ norm is tantamount to normalizing the waveform $\mathbf{x}_i$ itself.

## 2.3. Likelihood kernel

We define $\phi_i = S\mathbf{x}_i / \|S\mathbf{x}_i\|_2$ the renormalized vector, and $\Phi_i = \sqrt{q_i\phi_i}$ the vector that is parallel to $\phi_i$ and has $\ell^2$ norm equal to $\sqrt{q_i}$. Our working hypothesis is that constructing a K-DPP with time scattering as the descriptor of choice will preserve the auditory diversity across eco-acoustic samples. We repeat the same operation for every signal $\mathbf{x}_i \in \mathcal{X}$.

The likelihood kernel of the associated K-DPP is defined as

$$L_{i,j} = \langle \Phi_i | \Phi_j \rangle = \sqrt{q_i q_j} \langle \phi_i | \phi_j \rangle. \quad (6)$$

Given a set of distinct indices $\sigma = \{\sigma_1...\sigma_K\}$ between 1 and $N$, we denote by $L_\sigma$ the restriction of the matrix $L$ to the rows and columns whose indices belong to $\sigma$. Thus, the K-DPP with kernel $L$ is a random variable over the K-uplets of $1...N$, in which the probability of drawing a specific K-uplet $\sigma$ is proportional to the determinant of the matrix $L_\sigma$:

$$\mathbb{P}[\mathcal{X} = (\mathbf{x}_{\sigma_1}...\mathbf{x}_{\sigma_K})] \propto \det L_\sigma. \quad (7)$$

# 3. APPLICATION TO ECO-ACOUSTICS

## 3.1. Protocol

We apply our protocol to a dataset of $N = 432$ audio snippets, which we recorded between February 14th and February 16th in the dry tropical forest of San Jacinto (Bolívar, Colombia) by means
<figure><img src="image_88.png" /><figcaption>Figure 2: Likelihood kernel $\mathbf{L}$ for the audio snippets under study, arranged in a chronological order over three days. Darker colors denote a greater joint probability of sampling the row snippet and the column snippet as part of the same DPP draw.</figcaption></figure>

of a Wildlife Acoustics “Song Meter 2” acoustic sensor. This sensor is equipped with an omnidirectional microphone and records intermittently at a rate of one five-second snippet every ten minutes, twenty-four hours a day. Such data acquisition campaign belongs to a larger endeavor for biodiversity coordination, which is being coordinated by the Alexander von Humboldt Biological Resources Research Institute [6]. For our experiment, we rely on a Python implementation of K-DPP via an open-source library named DPPy [7] by using the aforementioned criteria of relevance and diversity. We extract the TFSD via the scikit-maad package [8] and the time scattering transform via the Kymatio package [9].

Figure 2 presents the likelihood kernel $\mathbf{L}$ as computed over this eco-acoustic dataset. Note that the diagonal of the matrix $\mathbf{L}$ gives the relevance of observations: for every $i$, $L_{i,i} = q_i$. We find that the relevance term, as described by the TFSD eco-acoustic indicator, roughly follows a daily periodicity: it is highest at dawn and dusk, lowest at midnight, and takes irregular values around noon. Future research is necessary to indicate whether this temporal pattern aligns with the chronobiology of vocalizing animals in the site under study.

In Figure 2, we also observe that $\mathbf{L}$ has a block-wise structure, which again, aligns with the daily cycle of animal vocalizations in the forest. Relatedly, we find that the soundscape under study seems to exhibit greater acoustical diversity, as measured by the scattering transform, during the night than during the day.

Mathematically speaking: for snippets $(x_i, x_j)$ which are acoustically similar (e.g., because they are one day apart), their $ℓ^2$-normalized scattering transforms $\phi_i$ and $\phi_j$ are almost colinear. As a result, the inner product $\langle \phi_i | \phi_j \rangle$ is close to one, and the determinant

associated to $\sigma = (i, j)$ is:

$$ \det \mathbf{L}_{\sigma} = q_i q_j - \sqrt{q_i q_j} \langle \phi_i | \phi_j \rangle \sqrt{q_j q_i} \langle \phi_j | \phi_i \rangle \\ = q_i q_j (1 - \langle \phi_i | \phi_j \rangle^2), \qquad (8) $$

which is nonnegative but close to zero. The subtractive term $-\langle \phi_i | \phi_j \rangle^2$ has a repulsive effect over the pair of snippets $x_i$ and $x_j$, of the order of the cosine similarity between features $\phi_i$ and $\phi_j$.

We compare our K-DPP to a naive baseline, which we name “uniform random”. The baseline verifies, for each snippet $i$, the proportionality rule: $\mathbb{P}[x_i \in \mathcal{X}] \propto 1/N$, where $N$ is the total number of audio snippets. In addition, we refine the naive baseline so that the probability of drawing the snippet $x_i$ is proportional to its relevance $q_i$: $\mathbb{P}[x_i \in \mathcal{X}] \propto q_i$. Under both “uniform random” and “relevance weighting” methods, the probabilistic sampling is made K times independent without replacement. Thirdly, we evaluate a well-known method for unsupervised clustering: namely, K-means. This method produces a partition of the full corpus into K disjoint clusters $C_1, \dots, C_K$ so as to minimize the intra-class variance:

$$ \sum_{k=1}^{K} \sum_{x_i \in C_k} \left\| \mathbf{S} x_i - \frac{1}{\text{card} C_k} \sum_{x_j \in C_k} \mathbf{S} x_j \right\|^2 . \qquad (9) $$

We build the subcorpus $\mathcal{X}$ by selecting, for each cluster $C_k$, the snippet $x$ whose scattering transform $\mathbf{S}x$ is closest to the Euclidean centroid of all points in $C_k$. Unlike the first two naive baselines, K-means does not perform independent sampling without replacement: instead, the cluster assignment of every snippet affects the global cost function in Equation 9, and thus indirectly conditions the probability of sampling every other snippet. At the same time, we note that K-means clustering becomes impractical if repeated draws of the subcorpus $\mathcal{X}$ are needed, or if the number of elements K needs to be adjusted dynamically.

## 3.2. Relevance-diversity tradeoff

We run all four methods with $K = 3$ and repeat them 200 times independently. Figure 3 illustrates our findings. As expected, the uniform random method fares poorly on both metrics of relevance and diversity. Relevance weighting improves relevance but still lacks in diversity. Clustering with K-means is outperformed by relevance weighting on both metrics. Last but not least, we observe that the K-DPP reaches a favorable tradeoff between relevance and diversity: it offers a greater diversity than relevance-weighted sampling while guaranteeing a better relevance than K-means. More precisely, K-DPP triples the diversity of relevance-weighted sampling while retaining about 90% of its relevance on average.

## 3.3. Species inventory

In the previous subsection, we have verified that the K-DPP method yields diverse and relevant subcorpora, by our predefined measure of relevance (TFDS) and diversity (scattering transform). It remains to be seen whether these definitions are useful in practice for conservation science. For this purpose, we run every snippet $x_i$ through BirdNET, a pretrained convolutional network for species identification [10]. In this way, our evaluation metric is the total number of distinct species in the K-uplet $\mathcal{X}$, also known as “species richness” in ecology. Moreover, we measure the “precision at K”: that is, the proportion of snippets in $\mathcal{X}$ which contain at least one bird vocalization, whatever be its species.
<figure><img src="image_89.png" /><figcaption>Figure 3: Scatter plot of relevance (x-axis) and diversity (y-axis) of *K*-uplets from our eco-acoustic dataset, as drawn from various sampling techniques (see legend). Dashed lines indicate the Pareto front for each method. The best *K*-uplets are in the top-right corner.</figcaption></figure>

<figure><img src="image_90.png" /><figcaption>Figure 4: Species richness (top) and precision (bottom) of *K*-uplets drawn from all four presented methods, for a set cardinal ranging between *K* = 3 and *K* = 80. Higher is better.</figcaption></figure>

Figure 4 illustrates our findings. We see that the naive baseline has a poor species richness and a poor precision. Meanwhile, sampling with *K*-DPP leads to better results: its *K*-uplets contain a richer species inventory and fewer false positives. Yet, a surprising result, deserving of further inquiry, is that *K*-means clustering matches *K*-DPP in richness, and even outperforms it for very small values of *K*.

## 4. CONCLUSION

We have shown that the use of *K*-determinantal point processes (*K*-DPP) in eco-acoustics allows to explore and summarize large volumes of audio data while satisfying an interesting tradeoff between relevance and diversity. By means of an off-the-shelf classifier of bird species (BirdNET), we have shown that *K*-DPP tend to enrich the species inventory of the subcorpus compared to random uniform sampling; and so, particularly for *K* > 10, when there a substantial risk of accidentally drawing near-duplicate snippets.

We note that the choice of a likelihood kernel plays a large role in the success of *K*-DPP. In our article, this choice was motivated by domain-specific knowledge in eco-acoustics and psycho-acoustics, and was later confirmed by means of a species classifier. Our paper calls attention on the risks underlying the random subsampling of a dataset, especially in the early phase of forming a research hypothesis. Relevance weighting reduces the risk of sampling false positives, yet at the cost of biasing the subcorpus *X* towards a narrow range of extremely salient events. Hence, we have advocated for a balanced approach, which takes both relevance and diversity into account. We have presented a first application of *K*-DPP for exploring biodiversity in a Colombian dry forest, with the hope to encourage more applications of this tool in the future.

## 5. ACKNOWLEDGMENT

We thank Guillaume Gautier and Rémi Bardenet for their help with the DPPy package.

## 6. REFERENCES

[1] J. Sueur and A. Farina, “Ecoacoustics: the ecological investigation and interpretation of environmental sound,” *Biosemiotics*, vol. 8, no. 3, pp. 493–502, 2015.

[2] A. Kulesza, B. Taskar, et al., “Determinantal point processes for machine learning,” *Foundations and Trends in Machine Learning*, vol. 5, no. 2–3, pp. 123–286, 2012.

[3] M. Cartwright, J. Cramer, A. E. M. Mendez, Y. Wang, H.-H. Wu, V. Lostanlen, M. Fuentes, G. Dove, C. Mydlarz, J. Salamon, et al., “SONYC-UST-V2: An urban sound tagging dataset with spatiotemporal context,” in *Proc. DCASE*, 2020.

[4] F. Gontier, C. Lavandier, P. Aumond, M. Lagrange, and J.-F. Petiot, “Estimation of the perceived time of presence of sources in urban acoustic environments using deep learning techniques,” *Acta Acustica united with Acustica*, vol. 105, no. 6, pp. 1053–1066, 2019.

[5] J. Andén and S. Mallat, “Deep scattering spectrum,” *IEEE Transactions on Signal Processing*, vol. 62, no. 16, pp. 4114–4128, 2014.
[6] C. Pizano and H. García, Eds., *El bosque seco tropical en Colombia*. Bogotá, D.C., Colombia.: Instituto de Investigación de Recursos Biológicos Alexander von Humboldt (IAvH), 2014.

[7] G. Gautier, G. Polito, R. Bardenet, and M. Valko, “DPPy: DPP Sampling with Python,” *Journal of Machine Learning Research - Machine Learning Open Source Software (JMLR-MLOSS)*, vol. 20, no. 180, pp. 1–7, 2019.

[8] J. S. Ulloa, S. Haupert, J. F. Latorre, T. Aubin, and J. Sueur, “scikit-maad: An open-source and modular toolbox for quantitative soundscape analysis in Python,” *Methods in Ecology and Evolution*, vol. 12, no. 12, pp. 2334–2340, 2021.

[9] M. Andreux, T. Angles, G. Exarchakis, R. Leonarduzzi, G. Rochette, L. Thiry, J. Zarka, S. Mallat, J. andén, E. Belilovsky, J. Bruna, V. Lostanlen, M. Chaudhary, M. J. Hirn, E. Oyallon, S. Zhang, C. Cella, and M. Eickenberg, “title=Kymatio: Scattering transforms in Python,” *Journal of Machine Learning Research*, vol. 21, no. 60, pp. 1–6, 2020.

[10] S. Kahl, C. M. Wood, M. Eibl, and H. Klinck, “Birdnet: A deep learning solution for avian diversity monitoring,” *Ecological Informatics*, vol. 61, p. 101236, 2021.
# POLYPHONIC SOUND EVENT DETECTION FOR HIGHLY DENSE BIRDSONG SCENES

Alberto García Arroba Parrilla¹

Dan Stowell¹,²

a.g.a.p.albertogarciaarrobaparrilla@
tilburguniversity.edu

1. Tilburg University, the Netherlands

d.stowell@tilburguniversity.edu

2. Naturalis Biodiversity Center, Leiden, the Netherlands

## ABSTRACT

One hour before sunrise, one can experience the dawn chorus where birds from different species sing together. In this scenario, high levels of polyphony, as in the number of overlapping sound sources, are prone to happen resulting in a complex acoustic outcome. Sound Event Detection (SED) tasks analyze acoustic scenarios in order to identify the occurring events and their respective temporal information. However, highly dense scenarios can be hard to process and have not been studied in depth. Here we show, using a Convolutional Recurrent Neural Network (CRNN), how birdsong polyphonic scenarios can be detected when dealing with higher polyphony and how effectively this type of model can face a very dense scene with up to 10 overlapping birds. We found that models trained with denser examples (i.e., higher polyphony) learn at a similar rate as models that used simpler samples in their training set. Additionally, the model trained with the densest samples maintained a consistent score for all polyphonies, while the model trained with the least dense samples degraded as the polyphony increased. Our results demonstrate that highly dense acoustic scenarios can be dealt with using CRNNs. We expect that this study serves as a starting point for working on highly populated bird scenarios such as dawn chorus or other dense acoustic problems.

**Index points** - Sound Event Detection, Polyphony, Birdsong, Dense scene

## 1. INTRODUCTION

Among other domains, acoustic biodiversity studies have flourished as improvements in data science have helped researchers understand, monitor and study many species in detail [1] [2] [3] [4]. Sound event detection tasks have covered many domains where relevant information can be extracted from acoustic scenarios [5] [6] [7]. SED problems can increase in difficulty when many classes are considered and when several events overlap at the same time [8] [9], also known as polyphony. Sound Event Detection (SED) models that can detect overlapping sound events can be addressed as polyphonic SED.

Polyphony is generally defined as the number of sound sources occurring at the same time (i.e., overlapping). In this work we restrict our attention to the number of simultaneous species (i.e., sound types). Polyphony is very common in real-life birdsong scenarios and requires polyphonic SED models to detect all overlapping events. To solve this problem, it requires a model capable of detecting, despite high levels of overlap, all existing species in the

studied area. This addresses scenarios like dawn chorus, defined by [10] as a phenomenon around one hour before sunrise with a high activity of birdsong from many individuals and species. This could reach up to 30+ species [11] presenting a very dense scenario for studying and monitoring wildlife.

Our work aims to build a polyphonic SED model capable of dealing with birdsong real-life scenarios. We focus on two research questions (RQs):

* RQ1: What level of polyphony can be successfully detected on a dense audio scene and dense species list?

* RQ2: How accurately can this model detect very dense scenes with up to 10 overlapping sounds?

To address these research questions, several datasets with different polyphony were created and used to train SED models. This work demonstrates that a single neural network can provide, considering the complexity of the task and conditions of the dataset, encouraging results on low-polyphonic scenes and high-polyphonic ones.

## 2. RELATED WORK

Both convolutional networks and recurrent networks have been developed to deal with SED tasks due to the ability to extract useful features from audio signals (convolution part or CNN) [1] [2] and detect temporal context in those features (recurrence part or RNN) [5] [12], however, a consensus on architecture is clear, as very good results have been obtained applying a commonly used design of CRNN. This consensus has also been reached regarding the input features where spectrograms tend to be the most popular ones [2] [5] [6] [9] [13] [14] [15] [12]. Differences in birdsong are expected to arise from both pitch and timbral variations [16] and thus the spectrogram is generally appropriate for multi-species birdsong recognition.

Reviewing related papers, it can surely be stated that not much research has been carried out in studying the effect of high polyphony in the birdsong audio domain. Some papers study sound event detection addressing polyphony in other kinds of scenes, mostly urban sounds, [6] [9] [14] [17] or using non-specific datasets while others employ avian datasets but without giving much focus on dense polyphonic scenes [1] [3] [4] [18].

The most closely related project is the study of polyphony using a convolutional recurrent network of [15]. The author presents a model that can address birdsong detection with good results for recordings with up to 3 overlapping sounds. However, it has limitations as only a total of 5 species are considered making this approach not very close to a real-life dawn chorus scenario. Also related to birdsong, [19] proposed an HMM-based method
to determine the temporal information of bird sounds without establishing any intrinsic limit on the polyphony. However, this study does not reflect how neural networks work on these problems.

To study dense polyphonic scenes, datasets should include a considerable number of classes, but most of the datasets used in the literature do not get over 16. Many papers have employed data synthesis to force a certain level of polyphony in the samples but if the number of classes is limited, then the polyphony will be too. Meaning that, if a dataset only contains *k* possible classes, the maximum overlapping sounds will be *k*, therefore limiting the polyphonic scenario.

Polyphony approaching a level of 30 or more [11] appears to be beyond the current state of the art, both in datasets and SED systems. Out work builds towards this goal, by training and evaluating an SED system for high-density polyphonic scenes typical in birdsong.

# 3. METHODS

## 3.1. Dataset selection

The chosen dataset [20], contains 77 recordings of dawn chorus from the USA contemplating 48 species and a total of 16,052 event activations. Every audio file has a fixed sample rate of 32kHz, fixed length of 5 minutes and has an associated annotation file with detailed information about every event (exact onset, offset and label). Annotation files also contain information about the frequency range (higher and lower values) the audio file is limited to, however, this information is not used in the present experiment as every file is processed without any upper or lower boundaries, and only temporal extents are predicted (start/end time).

A 48-species multi-class multi-label model presents a very complex SED problem, so after studying the distribution of the data, it could be seen that there were unbalanced class sizes, where many have less than 10 annotations in the entire dataset. In this work, we use the data in their empirical (unbalanced) distribution of sound events.

Only those species with at least 100 activations were considered for the detection model, resulting in 20 species. This process reduced the complexity while still allowing to increase the polyphony up to 10. Species with fewer than 100 activations are acoustically present but considered here as background sound.

<figure><img src="image_91.png" /><figcaption>Figure 1: Annotation files merge process. Tables in the figure show how the data is displayed in the files (removing several decimals for visual clarity) and the colored diagram represents the reduced annotation matrix of presence/absence of the species stated in their respective files. It can be observed in those diagrams that the polyphony increases when merging annotations.</figcaption></figure>

## 3.2. Preprocessing and subsets creation

To feed the recordings to the model, frame blocking was carried out where the audio signal was split into analysis frames of 5-second-fixed length, shifted with a fixed hop size of 2.5 seconds. Additionally, data augmentation methods such as time stretch, pitch shift and time shift were applied to help with generalization. However, the final set would have the same unbalanced proportion of classes mentioned in Section 3.1.

Three subsets were created for polyphony study of 3, 6 and 10 using data synthesis where several 5-second clips were selected and merged. Every created subset contains audio scenes where each clip has the same number of overlapping events (or less) as the determined polyphony. The maximum polyphonies for each subset were chosen by considering previous related work: 3 is the maximum polyphonic scenario successfully studied in an avian SED task; the maximum polyphonic scene reached successfully in other audio domains is 6 and 10 was selected as an approximation of a real birdsong acoustic scenario (which coincides with half the number of classes in the studied dataset).

Annotations were also merged resulting in a new list of all events of the species appearing in the involved files. This process can be observed in the tables found in Figure 1.

As an input feature, a NxM Mel spectrogram was calculated with a window size of 1024, a hop length half that size and 128 Mel bands where N is the number of frames and M the Mel bands. The annotation is transformed into a SxN binary (presence-absence) matrix, as shown in the diagrams of Figure 1, where S denotes the number of classes.

## 3.3. Model

The model used is a Convolutional Recurrent Neural Network (CRNN) inspired by [9]. Firstly, the convolutional part consists of 5 layers following an evolution of increasing features in the order of: (64, 128, 128, 128, 264). After every convolutional step: layer normalization, leaky ReLu activation function and 2D max pooling on the frequency axis. Then, two bidirectional gated recurrent units (GRU) with 128 neurons each, with tanh activation and dropout of 0.5. And, finally, a time-distributed fully connected part of 2 dense layers of 128 neurons each using Leaky ReLu as activation function and a follow-up dropout of 0.5. The architecture of the model is depicted in Figure 2 as well as the previous and following steps of the workflow.

Adam was used as the model's optimizer with a fixed learning rate of 0.001 and binary cross-entropy was calculated as loss function.

Every different polyphonic model was trained with 10,000 training samples and 200 epochs, using a training batch size of 128 and train-test split of 90-10. This split was carried out using random items without taking into consideration the unbalanced classes in the dataset.

Three models were created and fitted using subsets of different polyphony as training. From this point, the models are referred to as O3, O6 and O10, whose numeric IDs correspond to the maximum polyphony found in their training subsets.
<table><caption>Table 1: F score and error rate of the three models O3, O6 and O10 using their testing sets and 3 additional subsets with exactly 3, 6 and 10 overlapping events</caption><thead><tr><th rowspan="2"></th><th colspan="2">Matched testing set, with polyphony 1 to O_</th><th colspan="2">Samples with polyphony of 3</th><th colspan="2">Samples with polyphony of 6</th><th colspan="2">Samples with polyphony of 10</th></tr><tr><th>F Score</th><th>ER</th><th>F Score</th><th>ER</th><th>F Score</th><th>ER</th><th>F Score</th><th>ER</th></tr></thead><tbody><tr><td>Model O3</td><td>0.49±0.17</td><td>0.61±0.19</td><td>0.45±0.18</td><td>0.65±0.21</td><td>0.32±0.14</td><td>0.77±0.11</td><td>0.30±0.13</td><td>0.79±0.12</td></tr><tr><td>Model O6</td><td><strong>0.51±0.15</strong></td><td><strong>0.60±0.15</strong></td><td><strong>0.52±0.18</strong></td><td><strong>0.57±0.20</strong></td><td><strong>0.47±0.13</strong></td><td>0.64±0.12</td><td>0.41±0.09</td><td>0.72±0.08</td></tr><tr><td>Model O10</td><td>0.47±0.11</td><td>0.65±0.18</td><td>0.46±0.14</td><td>0.88±0.40</td><td>0.46±0.12</td><td><strong>0.62±0.11</strong></td><td><strong>0.45±0.09</strong></td><td><strong>0.68±0.07</strong></td></tr></tbody></table>

<figure><img src="image_92.png" /><figcaption>Figure 2: Data workflow of the Convolutional Recurrent model (CRNN). Input transformation and feature selection steps are stated as well as the main architecture design (layers, activations, number of neurons) of the model. Output of the model before thresholding is also shown as part of the workflow process.</figcaption></figure>

Every section of the model and processing of the data was carried out using Python, and libraries such as Keras and sklearn. The code for the preprocessing, model and evaluation is available in github.com/SrArroba/hd_sed.

### 3.4. Evaluation metrics

An evaluation metric system designed for SED tasks called *sed_eval* [21] is used to measure the performance of the models. F-score and error rate segment-based metrics were used to evaluate the models with a segment size (*tcollar*) of 0.1 seconds. For every model, 4 subsets are used for testing: 3 subsets with fixed polyphony of 3, 6 and 10, respectively, and a testing set matching each model's training session conditions (i.e., files with up to the specified value of polyphony).

<figure><img src="image_93.png" /><figcaption>Figure 3: Output after a sample is being run through the model. From left to right: prediction obtained with values between 0 and 1; the same prediction after applying a threshold of 0.5; ground truth.</figcaption></figure>

After performing a prediction, the output is binarized using 0.5 as thresholding value, as seen in Figure 3, where the x-axis represents the time frames (up to 313 frames in 5 seconds) while the y-frame contains the 20 classes/bird species alphabetically sorted. Thresholding can be used to have a better

understanding of the models' behavior and pick out further anomalies or faulty tendencies. After this task, the predicted and ground truth can be compared to evaluate the model.

### 4. RESULTS

While training the models, the loss was measured to study their evolution and how efficiently the models learnt. The first 100 epochs were recorded and shown together in Figure 4.

Analyzing Table 1, it can be observed that O6 performs generally better than the other two models except on higher levels of polyphony than it was trained for, while the O10 model provides consistent results for all types of samples. Despite its complexity, the CRNN model performs adequately matching the conclusions reached by [12] regarding the better fitness of CRNN to deal with sound event detection compared to other networks, such as simple convolutional or recurrent networks.

Additionally, F score and error rate were calculated for each bird species individually to check if there is a correlation between the model's performance on detecting them and their number of activations in the original dataset. As is commonly observed, the items with higher representation in the dataset also gives the strongest performance. Table 2 shows the number of annotations and evaluation metrics for the 3 most common and 3 least common species.

### 5. DISCUSSION

The goal of this study was to find a model that distinguishes bird species from a large list and temporally detects them when dealing with a dense scene of birdsong. Additionally, it aims to study whether a model can detect the species of a scene with up to 10 overlapping sounds (i.e., very dense scenario).

It could be hypothesized that models that trained with denser samples, learn faster than those who work with simpler scenarios. However, as seen in Figure 4, every model learns in a similar way while the range of their loss evolutions depends on how complex the samples are: the more complex, the higher the range of loss evolution due to the difficulty to learn denser scenes. Generally, models trained with denser scenes are very consistent for all densities while the model trained with simpler samples shows a degradation as the complexity increases. Additionally, models trained with denser scenes perform similar or better in simpler scenes. On the other hand, O10 gets worse results than the O6 model when dealing with emptier scenes. This could be because of the distribution of the training data regarding their polyphony.
<table><caption>Table 2: Individual F score and ER for the 3 most common and 3 least common bird species in the dataset.</caption><thead><tr><td>Species</td><td>Number of annotations</td><td>% of annotations</td><td>Mean F Score</td><td>Mean ER</td></tr></thead><tbody><tr><td>Eastern towhee (EATO)</td><td>5238</td><td>36.63%</td><td>0.73±0.23</td><td>0.53±0.25</td></tr><tr><td>Wood thrush (WOTH)</td><td>1826</td><td>11.38%</td><td>0.42±0.38</td><td>0.51±0.36</td></tr><tr><td>Northern cardinal (NOCA)</td><td>884</td><td>5.51%</td><td>0.24±0.37</td><td>0.58±0.67</td></tr><tr><td>Hermit thrush (HETH)</td><td>162</td><td>1.01%</td><td>0</td><td>0.12±0.33</td></tr><tr><td>Red-bellied woodpecker (RBWO)</td><td>145</td><td>0.90%</td><td>0.07±0.08</td><td>0.07±0.25</td></tr><tr><td>Blue-winged warbler (BAWW)</td><td>142</td><td>0.88%</td><td>0</td><td>0.12±0.32</td></tr></tbody></table>

Considering how every synthetic file is generated for each subset, it can denote that the higher the polyphony, the denser the scene and, therefore, more examples per class can be obtained in a single sample. This means that the O3 subset will have fewer representing features of the classes in a single input compared to O6 or O10. However, this does not mean that the models with denser samples learn faster as can be seen in Figure 4, where all models provide a similar evolution in their loss but at different values range. This difference can be explained as the more complex samples are, the more difficult to learn, and therefore, the higher the loss.

On the other hand, as shown in Table 1, models with denser scenes (such as O10) are very consistent for all densities while the model that learns with simpler samples, O3, shows a degradation as the complexity increases, having very poor performance on samples with 10 overlapping sounds. On the other hand, denser scenes mean more complexity, which slows the learning process of the model in that type of scenario. This can be observed in the O10 model where more complex samples are used for training but gets worse results than the O6 model when dealing with emptier scenes. Generally, models trained with denser scenes perform in a similar or better way on simpler samples.

While calculating the performance for each species, it could be hypothesized that it holds a correlation with the total number of activations found in the original dataset. Performing an evaluation using specific species it was demonstrated in Table 2 that species with an elevated number of activations provide better evaluation metrics when studied individually while species with very low appearance in the dataset gives significantly worse performance.

<figure><img src="image_94.png" /><figcaption>Figure 4: Training loss for the first 100 epochs. The evolution is shown together for the three established models: O3, O6, O10.</figcaption></figure>

Further work could be carried out investigating deeper architectures or more innovative feature selection to achieve better results on denser scenarios. When dealing with higher levels of polyphonies, CNN filters and GRU units should also increase, as suggested by [9].

## 6. CONCLUSION

In this project, a birdsong dataset of 20 classes was used to train several SED models capable of labeling and temporally allocating the species of birds present on highly dense scenes. Addressing the research questions initially stated, the three models created: O3, O6 and O10 performed suitably considering the complexity of the task.

It could be observed that every model displayed a similar learning curve, nevertheless, using too dense samples could affect the performance of the model when dealing with less crowded scenes but maintaining consistent results in all scenarios. Models' accuracy decayed when dealing with samples more complex than the ones used in training. Additionally, species that have more activations in the original dataset obtained the best results when studying them separately.

This work serves to prove the feasibility of elements of modern design, in terms of architecture or features, when facing a task of detecting dense acoustic scenarios. Moreover, it also serves as motivation for other researchers to try new creative ways of dealing with this problem and as a challenge to complete it by adding a location task (SELD).

## 7. REFERENCES

[1] V. Lostanlen, J. Salamon, A. Farnsworth, S. Kelling & J. Bello, «Robust sound event detection in bioacoustic sensor networks,» *PLoS ONE*, pp. volume 14, issue 10, Pe0214168 2019 DOI: 10.1371/journal.pone.0214168, 2019.

[2] S. Madhusudhana, Y. Shiu, H. Klinck, E. Fleishman, X. Liu, E. Nosal, T. Helble, D. Cholewiak, D. Gillespie, A. Sirovi & et al., «Improve automatic detection of animal call sequences with temporal context,» *Journal of the Royal Society Interface*, 18(180):20210297, 2021.

[3] D. Stowell, M. D. Wood, H. Pamuła, Y. Stylianou & H. Glotin, «Automatic acoustic detection of birds through deep learning: The first bird audio detection
challenge.,» *Methods in Ecology and Evolution, vol. 10, no. 3, pp. 368–380*, 2019.

[4] S. Kahl, T. Wilhelm-Stein, H. Klinch, D. Kowerko & M. Eibl, «Recognizing birds from sound - the 2018 birdclef baseline system,» *CoRR*, vol. arXiv: 1804.07177, 2018.

[5] E. Cakir, G. Parascandolo, T. Heittola, H. Huttunen & T. Virtanen, «Convolutional recurrent neural networks for polyphonic sound event detection,» *IEEE/ACM Transactions on Audio, Speech, and Language Processing*, 25(6), 1291-1303, 2017.

[6] Y. Cao, Q. Kong, T. Iqbal & F. An, «Polyphonic sound event detection and localization using a two-stage strategy,» *arXiv:1905.00268*, 2019.

[7] T. Heittola, «Computational Audio Content Analysis in Everyday Environments,» *Tampere University Dissertations - Tampereen yliopiston väitöskirjat; Vol. 434*, 2021.

[8] J. Abeßer, «USM-SED-A Dataset for Polyphonic Sound Event Detection in Urban Sound Monitoring Scenarios,» *arXiv:2105.02592*, 2021.

[9] S. Adavanne, A. Politis & T. Virtanen, «Multichannel sound event detection using 3D convolutional neural networks for learning inter-channel features,» *international joint conference on neural networks (IJCNN)* (pp. 1-7), 2018.

[10] P. R. Marler & H. Slabbekoorn, «Nature's music: the science of birdsong,» *Elsevier*, 2004.

[11] A. Farina, M. Ceraulo, C. Bobryk, N. Pieretti, E. Quinci & E. Lattanzi, «Spatial and temporal variation of bird dawn chorus and successive acoustic morning activity in a Mediterranean landscape,» *Bioacoustics*, 24(3), 269-288, 2015.

[12] R. Lu & Z. Duan, «Bidirectional GRU for sound event detection,» *DCASE*, 2017.

[13] S. Adavanne, A. Politis & T. Virtanen, «Localization, detection and tracking of multiple moving sound sources with a convolutional recurrent neural network,» *arXiv:1904.12769*, 2019.

[14] E. Cakir & T. Virtanen, «Convolutional recurrent neural networks for rare sound event detection,» *Deep Neural Networks for Sound Event Detection*, 12., 2019.

[15] M. Crous, «Polyphonic Bird Sound Event Detection With Convolutional Recurrent Neural Networks,» *10.13140/RG.2.2.11943.09126*, 2019.

[16] M. Faiß, T. Riede & F. Goller, «Tonality over a broad frequency range is linked to vocal learning in birds,» *The Royal Society*, 2022.

[17] K. Drossos, S. I. Mimilakis, S. Gharib, Y. Li & T. Virtanen, «Sound event detection with depthwise separable and dilated convolutions,» *International Joint Conference on Neural Networks (IJCNN)*, 2020.

[18] T. Pellegrini, «Densely connected CNNs for bird audio detection,» *25th European Signal Processing Conference (EUSIPCO)*, 2017.

[19] D. Stowell & D. Clayton, «Acoustic event detection for multiple overlapping similar sources,» *IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA)*, 2015.

[20] L. M. Chronister, T. A. Rhinehart, A. Place & J. Kitzes, «An annotated set of audio recordings of Eastern North American birds containing frequency, time, and species information,» 2021.

[21] A. Mesaros, T. Heittola & T. Virtanen, «Metrics for polyphonic sound event detection,» *Applied Sciences*, vol. 6, no. 6, pp. 162-178, 2016.
# LANGUAGE-BASED AUDIO RETRIEVAL WITH TEXTUAL EMBEDDINGS OF TAG NAMES

Thomas Pellegrini

IRIT, Université de Toulouse, CNRS, Toulouse INP, UT3, Toulouse, France
Artificial and Natural Intelligence Toulouse Institute (ANITI)
thomas.pellegrini@irit.fr

## ABSTRACT

Language-based audio retrieval aims to retrieve audio recordings based on a queried caption, formulated as a free-form sentence written in natural language. To perform this task, a system is expected to project both modalities (text and audio) onto the same subspace, where they can be compared in terms of a distance. In this work, we propose a first system based on large scale pretrained models to extract audio and text embeddings. As audio embeddings, we use logits predicted over the set of 527 AudioSet tag categories, instead of the most commonly used 2-d feature maps extracted from earlier layers in a deep neural network. We improved this system by adding information from audio tag text embeddings. Experiments were conducted on Clotho v2. A 0.234 mean average precision at top 10 (mAP@10) was obtained on the development-testing split when using the tags, compared to 0.229 without. We also present experiments to justify our architectural design choices<fnref n="1" />.

**Index Terms—** Language-based audio retrieval, Pre-trained representations, PaSST, MPnet, Transformers, AudioSet tags

## 1. INTRODUCTION

The ability to retrieve audio recordings from a text query can be very useful when searching for recordings in a large audio database. Language-based audio retrieval systems rank audio samples according to their match with a queried caption.

Recent systems are usually comprised of two neural networks: a text encoder, responsible for encoding a concise and dense representation of a textual query, and an audio encoder that encodes the audio content of the candidate recordings [1, 2]. Both encoders are expected to provide representations embedded in the same subspace, where the textual and audio samples can be compared in terms of a distance.

In this paper, we present a baseline system that follows this architecture, based on two open-sourced pretrained models to extract text and audio embeddings. Our main innovation is two-fold: i) we use logits as basic audio embeddings [3], instead of the more usual 2-d feature maps extracted from a hidden layer of a pretrained deep neural network [4, 5, 6], ii) we improve our baseline system by incorporating information from tags predicted on the audio recordings. We propose to combine the basic audio logit embeddings with the textual embeddings of the tag names.

After describing our system architectures and experimental setup, we report results obtained on Clotho v2 [7]. We then discuss our architecture design choices in depth.

## 2. SYSTEM DESCRIPTION

Our system architecture is shown in Fig. 1, with the audio encoder on the right part of the figure, and the caption encoder on the left part. We used two open-sourced pretrained transformers as encoders. Both were “frozen”: the embeddings were extracted only once offline and used as input to our systems. The learnable part of the model corresponds to a single linear layer that projects the audio embeddings onto the caption embedding subspace.

### 2.1. Text encoder: sentence embeddings with MPnet

The textual queries (captions) are encoded as sentence embeddings, obtained by averaging the word embeddings outputted by a sentence transformer. The all-mpnet-base-v2 model [8], further referred to as MPnet, was used to extract these embeddings $\vec{e}_s$, which are 768-dimensional $\ell_2$-normalized vectors. MPnet is a transformer of more than 109 M parameters, trained on over a billion pairs of sentences. It was chosen since it was reported as the best model for sentence embedding against a selected list of other models [9]. We apply Layer normalization [10], followed by $\ell_2$-normalization is applied to the embeddings to obtain the final ones, named $\vec{e}_q$:

$$ \vec{e}_q = \ell_2\text{-norm}(\text{LayerNorm}(\vec{e}_s)) \quad (1) $$

Layer normalization (LayerNorm, LN) is used to normalize the samples of a minibatch independently, with adaptive per-dimension bias and gain (scale). We found that using those led to slight overfitting, thus, we removed this option. Therefore, our text encoder does not have any learnable parameter.

### 2.2. Audio encoder: AudioSet logit embeddings with PaSST

In the proposed system, an audio recording is encoded as a single vector, a so-called scene embedding, using the Patchout transformer named PaSST [11], pretrained on AudioSet [12]. We use the “logits” embedding outputted by PaSST, which is a 527-dimensional dense vector, comprised of the logits predicted for the 527 AudioSet event tag classes. We denote this embedding $\vec{e}_l$.

Layer-norm is then used, in this case with adaptive bias and gain, followed by a linear layer and a final $\ell_2$-normalization. The final embedding is denoted $\vec{e}_a$. We can summarize these steps in Eq. 2:

$$ \vec{e}_a = \ell_2\text{-norm}(\text{Linear}(\text{LayerNorm}(\vec{e}_l))) \quad (2) $$

<footnote>Thanks for funding by the French ANR agency (LUDAU, ANR-18-CE23-0005-01) and "Investing for the Future — PIA3" AI Interdisciplinary Institute ANITI (ANR-19-PI3A-0004).</footnote>

<footnote n="1">¹PyTorch code available: https://github.com/topel/my-audio-retrieval-dcase2022</footnote>
<figure><img src="image_95.png" /><figcaption>Figure 1: System architecture. Left: audio encoder (with and without tag embeddings), right: text encoder. $\vec{e}_t$: logit embedding, $\vec{e}_a$: audio embedding, $\vec{e}_t$: weighted AudioSet tag embedding, $\vec{e}_{at}$: combined audio and tag embedding, $\vec{e}_s$: sentence embedding, $\vec{e}_q$: textual query embedding, $W_t$: matrix of the AudioSet tag name embeddings.</figcaption></figure>

We will, here-after, refer to our baseline system using PaSST and MPnet as PaSST-MPnet.

## 2.3. Augmented audio encoder: tag embeddings with MPnet

Adding information from tags and audio event predictions was shown to help audio captioning systems [13, 14, 15]. It should also help in our retrieval task. In this work, we tried to add such information within the audio encoder, by using MPnet to encode tag names as text embeddings. This tag name encoder is shown in Fig. 1, on the left of the audio encoding branch.

AudioSet includes 527 audio event classes, with names that can be single words, e.g., “Water”, “Rain” or a sequence of words, such as “Child speech, kid speaking”. We precomputed 527 tag name embeddings, extracted with MPnet.

For a given audio file, we use the PaSST logits $\vec{e}_l$ to weight each of the 527 embeddings, sum them to obtain a single tag embedding $\vec{e}_t$, according the following equations. First, the PaSST logit embedding is passed through a sigmoid function to obtain multilabel probabilities $\vec{p}$, then we compute the weighted sum of the precomputed tag embeddings:

$$ \vec{p} = \text{sigmoid}(\vec{e}_l) \qquad (3) $$

$$ \vec{e}_t = \ell_2\text{-norm} \left( \frac{\vec{p}}{\sum_c p_c} W_t \right) \qquad (4) $$

where $W_t \in \mathbb{R}^{527 \times d}$ is the matrix containing the 527 embeddings of the AudioSet tag names. $d$ is the dimension of the embeddings provided by MPnet ($d = 768$). $c$ is an index over the 527 tag categories.

After an $\ell_2$-normalization, $\vec{e}_t$ is combined to $\vec{e}_a$ with a convex linear summation:

$$ \vec{e}_{at} = \lambda \vec{e}_a + (1 - \lambda) \vec{e}_t \qquad (5) $$

where $\lambda \in [0, 1]$ is a weight tuned on a validation subset. The final audio embedding $\vec{e}_{at}$ is $\ell_2$-normalized before being compared to the caption query embedding:

$$ \vec{e}_{at} = \ell_2\text{-norm}(\vec{e}_{at}) \qquad (6) $$

The learnable parameters of the proposed system correspond to the audio encoder ones: in the linear layer and in the element-wise adaptive biases and gains of LN, with a total of 407 k parameters. If we take into account the large-scale transformers, this number increases up to 196M parameters, but as previously mentioned, we did not fine-tune the transformers.

## 2.4. Training objective

We used the same loss and similarity scoring functions than in the challenge baseline system [2]. Similarity between audio and caption embeddings is estimated by taking the dot product between their embeddings, and all our systems were trained with a sampling-based triplet loss [16]. We would like a model to provide a similarity score $S^p$ high for positive audio-query pairs, and low for either an audio embedding not paired with the caption query (audio impostor score $S_a^n$), or for a caption not corresponding to the audio embedding (query impostor score $S_q^n$). The audio and query impostors are randomly selected from the minibatch of samples being processed. The loss function as a function of the network learnable parameters $\theta$ is the sum of the contribution of these three scores:
<table><caption>Table 1: Results on Clotho v2.</caption><thead><tr><th rowspan="2">System</th><th rowspan="2"># params</th><th colspan="4">Development-testing split</th><th colspan="4">Evaluation dataset</th></tr><tr><th>mAP@10</th><th>R@1</th><th>R@5</th><th>R@10</th><th>mAP@10</th><th>R@1</th><th>R@5</th><th>R@10</th></tr></thead><tbody><tr><td>Challenge baseline</td><td>732k</td><td>0.068</td><td>0.032</td><td>0.109</td><td>0.188</td><td>0.061</td><td>0.026</td><td>0.102</td><td>0.176</td></tr><tr><td>CNN-transformer [5]</td><td>195M</td><td>0.260</td><td>0.150</td><td>0.400</td><td>0.530</td><td>0.251</td><td>0.153</td><td>0.387</td><td>0.504</td></tr><tr><td>PaSST-MPnet</td><td>196M</td><td>0.229</td><td>0.134</td><td>0.355</td><td>0.482</td><td>0.212</td><td>0.124</td><td>0.319</td><td>0.448</td></tr><tr><td>PaSST-MPnet-tags</td><td>196M</td><td>0.234</td><td>0.138</td><td>0.364</td><td>0.485</td><td>0.214</td><td><strong>0.128</strong></td><td><strong>0.332</strong></td><td>0.445</td></tr><tr><td>PaSST-MPnet-tags-AC</td><td>196M</td><td>0.240</td><td>0.145</td><td>0.365</td><td><strong>0.500</strong></td><td>0.213</td><td>0.125</td><td>0.322</td><td>0.454</td></tr><tr><td>Ensemble (2 models)</td><td>196M</td><td><strong>0.243</strong></td><td><strong>0.148</strong></td><td><strong>0.369</strong></td><td>0.498</td><td><strong>0.216</strong></td><td>0.127</td><td>0.321</td><td><strong>0.463</strong></td></tr></tbody></table>

$$ \mathcal{L}(\theta) = \frac{1}{N} \sum_{\text{batch}} \max(0, S_a^n - S^p + \eta) + \max(0, S_q^n - S^p + \eta) \quad (7) $$

where N is the number of samples in the minibatch, η a margin.

## 2.5. Evaluation

Once a system is trained, retrieval is performed by scoring the similarity between the caption embedding and the audio embeddings of all the candidate audio files. The same scoring function as during training is used: the dot product. Following the DCASE 2022 challenge requirements, our systems retrieve ten audio files for each queried caption, sorted according to their match with the query.

Standard information retrieval metrics are used to evaluate the audio retrieval models: mean average precision at top-10 (mAP@10) [17] and recall at k, the number of retrieved audio items considered for scoring (R@k with k ∈ {1, 5, 10}) [18]. To estimate these metrics, only a correct audio-caption pair is considered as positive, all the other audio recordings retrieved are considered negative. To rank the systems submitted to the DCASE 2022 challenge, mAP@10 was the main metric [2]. It is a rank-aware metric. For a given queried caption, the precision values are averaged at the positions where the relevant items are in the retrieved rank list. The R@k metric is rank-unaware, and corresponds to the proportion of relevant items among the top-k retrieved results. For both mAP@10 and R@k, the higher the value, the better the system.

# 3. EXPERIMENTAL RESULTS

## 3.1. Datasets

We conducted our experiments on Clotho v2 [7], with the splits defined in the DCASE 2022 challenge [19]. The development set is comprised of 3839 audio clips with 19195 reference captions for training, and 1045 audio recordings with 5225 captions for testing. We refer to this test subset as development-testing, dev-test in short. We did not use the validation set. A Clotho evaluation set was provided in the challenge, used to rank the submitted systems. It contains 1000 audio files with 1000 associated captions [2]. We also used for pretraining the training subset of AudioCaps [20], containing 46231 audio files with one reference caption per file.

## 3.2. Experimental setup

All our models were trained on Clotho v2 for 50 epochs with the same setup: minibatches of 128 samples, a 1e-3 initial learning rate, and the Adam optimizer. We used a reduce-on-plateau scheduler

based on the Clotho validation split loss, with a 0.5 ratio and a 5-epoch patience. Smaller and larger batch sizes were tested, but 128 was found to be the best one. We used η = 0.4, and λ = 0.8 when using the tags, as will be discussed here-after. When pretraining a model on AudioCaps, the number of epochs was 100 and no learning rate scheduler was used.

## 3.3. Results

Table 1 shows the results of our systems, together with the ones of the baseline proposed by the DCASE 2022 Task 6b challenge organizers (based on a convolutional recurrent neural network and pretrained word embeddings), and of a CNN-transformer [5], which reached the second rank out of ten teams in the challenge, with models comprised of a number of parameters similar to ours.

First, our systems based on pretrained transformers for both audio and text encoding largely outperformed the challenge baseline, which was trained from scratch on the Clotho v2 development dataset. Relying on pretrained models is efficient in this case, where labeled training data is scarce. Second, our results are worse by about 0.02 on the evaluation dataset (eval) than on the development-testing subset (dev-test). This is probably due to the fact that model selection was performed on this subset, so overfitting might have happened. It can also be that eval is more difficult than dev-test.

Compared to our baseline model PaSST-MPnet, adding information from the tag embeddings was beneficial: PaSST-MPnet-tags reached 0.234 mAP@10 compared to 0.229, on dev-test, and 0.214 compared to 0.212 on eval.

Pretraining on AudioCaps (PaSST-MPnet-tags-AC) brought a 0.06 absolute mAP@10 improvement on dev-test, compared to PaSST-MPnet-tags. Interestingly, this gain was not observed on eval, so we cannot conclude that pretraining was beneficial or not. PaSST-MPnet-tags models trained on AudioCaps, before finetuning on Clotho, reached about 0.18 mAP@10 on Clotho dev-test.

Finally, an ensemble of two PaSST-MPnet-tags-AC models trained with different seeds, led to our best results. By ensemble, we mean that the distance scores outputted by the two models were averaged. This setting allowed us to obtain the fourth place of the challenge [19].

# 4. ARCHITECTURE DESIGN CHOICES

In the following, all reported experiments were conducted with PaSST-MPnet-tags models.

## 4.1. Influence of LayerNorm in the audio encoder

As described in Section 2, in our models, we use LayerNorm (LN) with adaptive gain and bias applied to the PaSST embeddings in the
<figure><img src="image_96.png" /><figcaption>Figure 2: Similarity score distributions on the training subset when using either a 0.4 (top) or a 1.0 (bottom) value for the margin $η$, when training a PaSST-MPnet-tags model. Pos: positive audio-caption pair, A imp: negative audio-query pair (audio impostor), Q imp: negative query-audio pair (caption impostor).</figcaption></figure>

audio encoder.

First, if we do not use LN in the audio encoder, mAP@10 and the other metrics are much worse: 0.217 mAP@10, compared to 0.234 reported in Table 1.

Second, we tried to use batch normalization (BatchNorm, BN) in the audio encoder, instead of LN. Using BN and keeping all the rest of the architecture unchanged (and for a given random seed), led to a 0.229 mAP@10 value, again worse than 0.234 when using LN. By observing the learning curves, BN seems to overfit a little more than LN. When using adaptive bias and scale, the number of learnable parameters is the same for both types of normalization, i.e. twice the dimension of the embeddings. The only possible reason why BN overfits more is because of the running means and standard deviations estimated on the training subset of data, that may not generalize so well on the test subset. These are necessary for BN but not for LN, since LN performs a per sample normalization.

## 4.2. Impact of the margin $η$

In [16], the authors used a default value of 1.0 for the margin $η$, used in the loss function, given in Eq. 7. In the DCASE challenge baseline, this default value was also used. In our case, the audio and queried caption embeddings are of unit norm. The results of the dot products, i.e., the similarity scores, are theoretically between -1.0 and 1.0. A margin of 1.0 might be to difficult to obtain when training a model. Indeed, $η = 1.0$ led to a 0.225 mAP@10 on dev-test, worse than 0.234 obtained with $η = 0.4$. We tried several

<figure><img src="image_97.png" /><figcaption>Figure 3: mAP@10 values on dev-test according to the audio embedding weight $λ$.</figcaption></figure>

values lower than 1.0, and $η = 0.4$ was the best value.

In Fig. 2, we plotted the score distributions of the positive pairs (Pos, a positive caption-audio pair) and negative pairs (A imp and Q imp for audio and query impostors, resp.), on the training subset. The distributions on the dev-test subset are very similar. The difference between the top and the bottom plots is the value of the margin: $η = 0.4$ (top figure) and $η = 1.0$ (bottom figure). As can be seen, the support interval of the scores is smaller with $η = 0.4$ than with $η = 1.0$: the positive pair scores reach about 0.62 maximum with $η = 0.4$, and about 0.8 with $η = 1.0$. Nevertheless, we can see more overlap with impostor scores in the case of $η = 1.0$, which mean that more mistakes are made by this model.

## 4.3. Influence of the audio/tag weight $λ$

Figure 3 shows the mAP@10 values on dev-test, when varying $λ$ from Eq. 5. When $λ = 0$, the audio encoder provides as output the tag embeddings $\vec{e}_t$, and the model has no learnable parameters at all. In this setting, a 0.105 mAP@10 is obtained. This shows that using the tag name embeddings weighted with their probabilities already brings some information. When $λ = 1$, the audio encoder provides as output the projected PaSST embeddings $\vec{e}_a$, and the model corresponds to PaSST-MPnet, which reached 0.229 mAP@10. In between, the score varies, and the best value was obtained with $λ = 0.8$.

# 5. CONCLUSION

We reported text-based audio retrieval experiments, with systems based on pretrained audio and sentence large scale transformers. In our baseline system, we used logits as audio embeddings, instead of the most commonly used 2-d feature maps extracted from earlier layers in a deep neural network. We improved this system by adding information from AudioSet tags, encoded as sentence embeddings. Finally, we discussed some of our architecture design choices: the use of layer normalization, a value smaller than 1.0 for the margin in the contrastive loss function for learning, and the 0.8 value for weighting the audio embeddings when adding the tag contribution within the audio encoder.

As a short-term perspective, we would like to use logit embeddings together with more standard 2-d feature maps in the audio encoder, to see whether those are complementary [5]. We also would like to investigate the use of external data, either for pretraining models or in a semi-supervised learning setting, where existing algorithms could be adapted to the present task [21].
## 6. REFERENCES

[1] H. Xie, O. Räsänen, K. Drossos, and T. Virtanen, "Unsupervised audio-caption aligning learns correspondences between individual sound events and textual phrases," in *Proc. ICASSP*. IEEE, 2022, pp. 8867-8871.

[2] H. Xie, S. Lipping, and T. Virtanen, "Dcase 2022 challenge task 6b: Language-based audio retrieval," *arXiv preprint arXiv:2206.06108*, 2022.

[3] E. Tzinis, S. Wisdom, J. R. Hershey, A. Jansen, and D. P. Ellis, "Improving universal sound separation using sound classification," in *Proc. ICASSP*. IEEE, 2020, pp. 96-100.

[4] X. Xu, Z. Xie, M. Wu, and K. Yu, "The SJTU System for DCASE2022 Challenge Task 6: Audio Captioning with Audio-Text Retrieval Pre-training," DCASE2022 Challenge, Tech. Rep., July 2022.

[5] X. Mei, X. Liu, H. Liu, J. Sun, M. D. Plumbley, and W. Wang, "Language-based audio retrieval with pre-trained models," DCASE2022 Challenge, Tech. Rep., July 2022.

[6] T. L. de Gail and D. Kicinski, "Take it easy: Relaxing contrastive ranking loss with CIDEr," DCASE2022 Challenge, Tech. Rep., July 2022.

[7] K. Drossos, S. Lipping, and T. Virtanen, "Clotho: An audio captioning dataset," in *Proc. ICASSP*. IEEE, 2020, pp. 736-740.

[8] https://huggingface.co/sentence-transformers/all-mpnet-base-v2.

[9] https://www.sbert.net/docs/pretrained_models.html.

[10] J. L. Ba, J. R. Kiros, and G. E. Hinton, "Layer normalization," *arXiv preprint arXiv:1607.06450*, 2016.

[11] K. Koutini, J. Schlüter, H. Eghbal-zadeh, and G. Widmer, "Efficient training of audio transformers with patchout," *arXiv preprint arXiv:2110.05069*, 2021.

[12] J. F. Gemmeke, D. P. W. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, "Audio set: An ontology and human-labeled dataset for audio events," in *Proc. IEEE ICASSP*, New Orleans, LA, 2017.

[13] Z. Ye, H. Wang, D. Yang, and Y. Zou, "Improving the performance of automated audio captioning via integrating the acoustic and textual information," DCASE2021 Challenge, Tech. Rep., July 2021.

[14] F. Gontier, R. Serizel, and C. Cerisara, "Automated audio captioning by fine-tuning bart with audioset tags," in *Proc. Workshop DCASE*, 2021, pp. 170-174.

[15] Q. Han, W. Yuan, D. Liu, X. Li, and Z. Yang, "Automated audio captioning with weakly supervised pre-training and word selection methods," in *Proc. Workshop DCASE*, Barcelona, Spain, November 2021, pp. 6-10.

[16] D. Harwath, A. Torralba, and J. Glass, "Unsupervised learning of spoken language with visual context," in *Advances in Neural Information Processing Systems*, D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, Eds., vol. 29. Curran Associates, Inc., 2016. [Online]. Available: https://Proc.neurips.cc/paper/2016/file/82b8a3434904411a9fdc43ca87cee70c-Paper.pdf

[17] P. Kaur, H. S. Pannu, and A. K. Malhi, "Comparative analysis on cross-modal information retrieval: A review," *Computer Science Review*, vol. 39, p. 100336, 2021.

[18] A. S. Koepke, A.-M. Oncescu, J. Henriques, Z. Akata, and S. Albanie, "Audio retrieval with natural language queries: A benchmark study," *IEEE Trans. on Multimedia*, pp. 1-11, 2022.

[19] http://dcase.community/challenge2022/.

[20] C. D. Kim, B. Kim, H. Lee, and G. Kim, "Audiocaps: Generating captions for audios in the wild," in *Proc. of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)*, 2019, pp. 119-132.

[21] L. Cances, E. Labbé, and T. Pellegrini, "Comparison of semi-supervised deep learning algorithms for audio classification," *EURASIP Journal on Audio, Speech, and Music Processing*, vol. 2022, no. 23, Sept. 2022. [Online]. Available: https://doi.org/10.1186/s13636-022-00255-6
# LATENT AND ADVERSARIAL DATA AUGMENTATIONS FOR SOUND EVENT DETECTION AND CLASSIFICATION

David Perera, Slim Essid, Gaël Richard

LTCI, Télécom Paris, Institut Polytechnique de Paris, France
{david.perera, slim.essid, gael.richard}@telecom-paris.fr

## ABSTRACT

Invariance-based learning is a promising approach in deep learning. Among other benefits, it can mitigate the lack of diversity of available datasets and increase the interpretability of trained models. To this end, practitioners often use a consistency cost penalizing the sensitivity of a model to a set of carefully selected data augmentations. However, there is no consensus about how these augmentations should be selected. In this paper, we study the behavior of several augmentation strategies. We consider the task of sound event detection and classification for our experiments. In particular, we show that transformations operating on the internal layers of a deep neural network are beneficial for this task.

**Index Terms**— sound event detection, data augmentation, adversarial learning

## 1. INTRODUCTION

Ambient sound analysis is a rapidly growing field, which has several important industrial applications such as security (e.g., audio surveillance), health (e.g., home monitoring, diagnosis based on sound), transportation (e.g., autonomous driving), manufacturing industry (e.g., predictive maintenance) and bioacoustics (e.g., ecosystem evolution tracking). In this context, sound event detection and classification consists in identifying and temporally localizing sound events in a complex acoustic scene. Deep learning has been successfully applied to this task, significantly improving the state of the art. However, this approach has two major drawbacks [1]. On the one hand, it requires a large amount of annotated data, which is costly and time-consuming to gather. On the other hand, the models obtained by this approach lack interpretability, making it hard to assess their reliability in unseen situations.

Data augmentation is an efficient technique which can mitigate the lack of available data and exploit unlabeled data [2]. This method also makes it possible to force trained models to learn specific invariants. By selecting relevant data augmentations, one can increase the model robustness to annotation errors and make them easier to interpret [2]. Moreover, data augmentation can be easily applied to any training algorithm and can improve its performance at a minimal development cost. This explains the popularity of this approach, especially in the field of sound event detection.

Many data augmentation strategies have been designed over the past decades. They now form a large set of methods that is costly to explore exhaustively. As a consequence, default settings are often passed from one system to another with only limited investigation [3], which leads to sub-optimal design choices.

This paper proposes a comparative study of the impact of several data augmentation strategies on task 4 of the DCASE chal-

lenge.<fnref n="1" /> We focus on domain agnostic data augmentations. In this context, we study the impact of adversarial augmentations and latent augmentations (data augmentations applied directly on the latent space of a deep neural network). We propose a common training framework to compare these augmentation strategies. By carefully selecting them, we show that it is possible to outperform the baseline model we compete against and to simplify its training objective.

This article is organized as follows. Section 2 presents a brief state of the art. Section 3 introduces task 4 of the DCASE challenge and describes its baseline. Section 4 explains the experimental framework that we use. Finally, section 5 discusses the obtained results.

## 2. STATE OF THE ART

There are two main strategies used to overcome the lack of training data: exploiting additional data from a related domain, and constraining the algorithm using domain knowledge [2]. Combining both ideas, invariance-based approaches penalize the variations of a model *f* in the neighborhood of the training data. More precisely, these approaches aim to reduce the quantity $||f(x) - f(\tau(x))||$, where *x* denotes a training point, $||\cdot||$ denotes a norm and $\tau$ denotes a data augmentation. This forces the model *f* to learn some invariants, which can be defined explicitly. We thus obtain some guarantees on the behavior of *f*, and consequently improve its interpretability. We briefly review here the main developments in this field.

The Ladder Network [4] adds to the objective function a cost enforcing the invariance of the model to small perturbations of its input and internal representations. A key point is that this method can leverage unlabeled data. In [5], the authors use Dropout augmentation [6] instead of random noise.

Temporal Ensembling (TE) [7] compares the prediction $f(\tau(x))$ computed for an augmented input $\tau(x)$ with an average of the predictions $f^{(n)}(\tau(x))$ computed at different epochs $n$. This method is inspired by the state of the art in stochastic optimization [8]. The idea is to accelerate the convergence of *f* by mitigating the randomness resulting from the selection of the mini-batches and the augmentation of the input.

Mean Teacher (MT) [9] is an improvement of TE: instead of averaging the predictions at each training epoch, it maintains an average of the model parameters. Moreover, it updates this average after each mini-batch instead of updating it after each epoch, which speeds up training.

In [10], the authors use Mixup augmentation [11] within the MT framework. This new training method encourages convex in-

<footnote n="1">¹https://dcase.community/</footnote>
terpolation between two input samples. It is based on the follow-
ing hypothesis: the decision boundaries of a classifier should be
located between samples of different classes. Consequently, instead
of moving an input sample in a random direction, which is inef-
ficient, it is better to move it in the direction of a sample from a
different class.

Following a similar line of research, Virtual Adversarial Training (VAT) promotes invariance to small perturbations of the input. This method uses adversarial perturbations (which maximize the variation of $f$) instead of using random perturbations. According to the authors, adversarial perturbations constitute a better heuristic, and can speed up the convergence of the algorithm. This algorithm is however based on a second order approximation of the adversarial perturbations, which is slow to compute. The Adversarial Noise Layer [12] generalizes the idea of adversarial perturbation to the intermediate layers of a model. It uses a first order approximation, which is faster to compute. In [13], the authors study a variant of this method based on the Dropout augmentation.

Universal Adversarial Perturbation (UAP) [14] is a further generalization of adversarial attacks. While the approaches presented above focused on fooling one model on a single instance of the training set, UAP seek to fool several models on most instances of the training set. The authors have shown that the resulting perturbations exploit the geometry of the classifiers’ decision boundaries. This makes them useful to design an invariance metric. However, their high computational cost discourages any use, aside from evaluation.

Finally, we can mention Mixmatch [15]. In addition to a consistency cost enforcing invariance to data augmentations, this method takes into account an entropy cost. The purpose of this cost is to increase the confidence level of the predictions made by the trained model.

Some of the methods presented above have already been applied to DCASE Task 4. Since 2019, the MT method has been used as the baseline for this task, and many algorithms submitted to the challenge are also based on this method. The authors of [3] propose an in-depth study of the application of MT to this task. The VAT method has also been applied to this task, with three different architectures: A Recurrent Convolutional Network (CRNN) [16], a Gated Recurrent Neural Network [17], and a Gated Recurrent Convolutional Neural Network [18]. More recently, the authors of [19] have studied a variant in which the noise is not adversarial but random. However, both the training method and the architecture change from one of these articles to another. It is therefore difficult to factor out the impact of the data augmentation strategy alone.

Closest to our work, [20] and [21] study the impact of several
types of data augmentations on a single model. The authors focus
on audio-specific data augmentations: pitch shifting, time shifting,
reverberation, frequency masking and time masking. Unlike these
studies, however, we focus our work on adversarial and latent data
augmentation. There are two main reasons for this. First, these aug-
mentations are domain agnostic, and could potentially work across
a vast range of tasks. This makes them very potent and interesting
to study. Second, these augmentations can be restricted to the au-
dio domain, giving birth to adversarial audio augmentations (which
could be seen as realistic worst-case scenarios), and latent audio
augmentations (which would force the internal representations of
a deep neural network to keep the structure of audio data, such as
time-shift or pitch-shift invariance). The aim of this paper is to
provide insights about the impact of adversarial and latent augmen-
tations on sound event detection. We use task 4 of DCASE as a case

study.

3. TASK DEFINITION

Task 4 of DCASE uses the dataset Domestic Environment Sound Event Detection (DESED) [20], which is composed of 10-second audio recordings made in domestic environments. DESED is divided into three distinct datasets: an unlabeled dataset $\mathcal{D}_u$, a weakly labelled dataset $\mathcal{D}_w$, and a synthetic strongly labelled dataset $\mathcal{D}_s$. There are ten possible classes for the annotation of sound recordings. Each recording in $\mathcal{D}_w$ is annotated with the set of sound events it contains. For $\mathcal{D}_s$, each event is temporally localized in addition to being identified by a label.

Two metrics are used to evaluate the models. To measure the temporal accuracy of the predictions, we use an event-based F1 macro score, which is denoted by *macro* in the following. In order to measure the labeling accuracy, we use the Polyphonic Sound Detection Score [22], which is denoted by *psds* in the following. For the computation of these two metrics, we adopt the parameters proposed for the 2020 edition of the DCASE challenge. We use the *macro* score, computed on the public evaluation dataset<fnref n="2" /> proposed in this same edition, to evaluate and compare the models.

In order to increase the usability of our results, we build our study on [20], which already investigates several design choices concerning task 4 of DCASE. In particular, we use the same baseline model<fnref n="3" /> as a basis for our experiments. This baseline exploits a CRNN architecture, which is commonly used in audio. It achieves high performance on the task. Moreover, the impact of its various components has been exhaustively studied. The baseline takes as input Mel spectrograms with 128 Mel bands, built with an analysis window of size 2048 and a hop length of 255. The input signals are sampled at 16kHz. The convolutional block of the CRNN is composed of 7 layers with filter sizes (16, 32, 64, 128, 128, 128, 128) and a kernel of size 3x3. Each convolution is followed by a Batch Normalization layer, a Gated Linear Unit, a Dropout layer with probability 0.5, and a Maxpooling layer. The recurrent block is composed of two Gated Recurrent Units with 128 layers each. It is followed by the attention pooling layer described in [23]. Median filtering is applied during post-processing. This architecture features 11 million trainable parameters. The model is trained on 200 epochs with the Adam optimizer.

4. AUGMENTATION STRATEGIES

4.1. Training framework

We compare several training objectives. In each case, the training
objective can be decomposed into three cost functions: a classifica-
tion metric penalizing prediction errors $\mathcal{L}_{class}$, a consistency metric
promoting invariance to data augmentations $\mathcal{L}_{const}$ and a regular-
ization metric $\mathcal{L}_{reg}$.

We segment the training mini-batches into three parts, each cor-
responding to one of the datasets $\mathcal{D}_u$, $\mathcal{D}_w$ and $\mathcal{D}_s$. We use the re-
spective proportions (1/2, 1/4, 1/4).

We note $f$ the baseline and $f_{ema}$ its exponential moving av-
erage across iterations. If we note $f^{(n)}$ the model obtained after
iteration $n$, $\theta^{(n)}$ its parameters (with similar notations for $f_{ema}$ and

<footnote n="2">²https://zenodo.org/record/3588172</footnote>

<footnote n="3">³https://github.com/turpaultn/dcase20_task4/</footnote>
$\theta_{ema}$) and $\alpha^{(n)}$ a coefficient varying during the training, then $f_{ema}$ is defined by its parameters update

$$ \theta_{ema}^{(n+1)} = \alpha^{(n)}\theta^{(n)} + (1 - \alpha^{(n)})\theta_{ema}^{(n)}. \quad (1) $$

Noting $x$ a recording in the DESED dataset, $y$ the corresponding label when it exists, $d$ a Gaussian noise vector, $\mathcal{L}_{BCE}$ the binary cross-entropy and $\mathcal{L}_{MSE}$ the mean square error, we can define the training objective of the baseline as follows:

$$ \mathcal{L}_{\text{class}} = \begin{cases} \mathcal{L}_{\text{BCE}}[f(x), y] & \text{if } (x, y) \in \mathcal{D}_w \cup \mathcal{D}_s, \\ 0 & \text{else} \end{cases}, \quad (2) $$

$$ \mathcal{L}_{\text{const}} = \mathcal{L}_{\text{MSE}}[f(x), f_{\text{ema}}(x + d)], \quad (3) $$

$$ \mathcal{L}_{\text{reg}} = 0. \quad (4) $$

Through our experiments, we keep the classification cost used by the baseline. However, we will experiment with other consistency and regularization costs.

## 4.2. Consistency costs

The simplest consistency cost we considered is the $\mathcal{L}_2$ distance between a prediction $f(x)$ and a perturbed prediction $f(x + d)$ (with $d$ a Gaussian noise vector):

$$ \mathcal{L}_{\text{const}} = \mathcal{L}_{\text{MSE}}[f(x), f(x + d)]. \quad (5) $$

We have also considered an adversarial consistency cost,

$$ d = \nabla_d \mathcal{L}_{\text{MSE}}[f(x), f(x + d)] |_{d=0}, \quad (6) $$

$$ \mathcal{L}_{\text{const}} = \mathcal{L}_{\text{MSE}}[f(x), f(x + d)], \quad (7) $$

as well as the following variant of VAT:

$$ d = \operatorname*{argmax}_{\|d\| \le \epsilon} \mathcal{L}_{\text{MSE}}[f(x), f(x + d)], \quad (8) $$

$$ \mathcal{L}_{\text{const}} = \mathcal{L}_{\text{MSE}}[f(x), f(x + d)]. \quad (9) $$

The original VAT algorithm uses the KL-divergence as a metric to compare the two outputs of the classifier $f(x)$ and $f(x + d)$. However, the $\mathcal{L}_2$ distance proved to be empirically better for this task. This choice also has the added benefit to homogenize the definition of the consistency costs $\mathcal{L}_{const}$ that we study. Indeed, these costs differ now only by the definition of the perturbation $d$.

Finally, we use a Mixup consistency cost, computed from a second sample $x'$

$$ \mathcal{L}_{\text{const}} = \mathcal{L}_{\text{MSE}}[\text{Mixup}(f(x), f(x'))], \quad (10) \\ f(\text{Mixup}(x, x'))]. $$

All these augmentations can be applied to the internal representations of the model. We apply data augmentation to the output of the CNN block and to the output of the RNN block of the baseline (see Figure 1).

## 4.3. Regularization costs

The baseline does not use an explicit regularization cost. This is because MT combines a consistency criterion and regularization criterion into a single training objective

$$ \mathcal{L}_{\text{const}} = \mathcal{L}_{\text{MSE}}[f(x), f_{\text{ema}}(x + d)]. \quad (11) $$

<figure><img src="image_98.png" /><figcaption>Figure 1: Architecture of the baseline system and location of the data augmentation modules.</figcaption></figure>

However, this method prevents us from fine tuning the trade-off between regularization and consistency in the final training objective. Yet, we have observed that this fine tuning is crucial when the model is dealing with difficult augmentations, such as adversarial attacks (see subsection 5.1).

For this reason, we have also experimented with a more traditional $\mathcal{L}_2$ regularization cost

$$ \mathcal{L}_{\text{const}} = \mathcal{L}_{\text{MSE}}[f(x), f(x + d)], \quad (12) $$

$$ \mathcal{L}_{\text{reg}} = \mathcal{L}_2[f], \quad (13) $$

where $\mathcal{L}_2[f]$ is computed on the model parameters.

# 5. EXPERIMENTAL RESULTS

The different training methods that we have used and the scores that we have obtained are summarized in Table 1. Hyperparameters have been selected using the validation score computed on a split of $\mathcal{D}_s$.

<table><caption>Table 1: Comparison of training methods. Highest values are shown in bold, and scores above the baseline are underlined. The baseline uses random noise as an augmentation strategy.</caption><thead><tr><th rowspan="2"></th><th rowspan="2">Augmentation</th><th colspan="2">Location</th><th colspan="2">Regularization</th><th colspan="2">Scores</th></tr><tr><th>Input</th><th>Latent</th><th>MT</th><th>l<sub>2</sub></th><th>macro</th><th>psds</th></tr></thead><tbody><tr><td>1</td><td>None</td><td></td><td></td><td></td><td></td><td>0.291</td><td>0.500</td></tr><tr><td>2</td><td>Baseline</td><td>x</td><td></td><td>x</td><td></td><td><strong>0.381</strong></td><td><strong>0.552</strong></td></tr><tr><td>3</td><td>Random</td><td>x</td><td></td><td></td><td>x</td><td><strong>0.397</strong></td><td>0.534</td></tr><tr><td>4</td><td>Random</td><td></td><td>x</td><td></td><td>x</td><td><strong>0.388</strong></td><td><strong>0.565</strong></td></tr><tr><td>5</td><td>Adversarial</td><td></td><td>x</td><td></td><td>x</td><td>0.374</td><td>0.559</td></tr><tr><td>6</td><td>VAT</td><td>x</td><td></td><td>x</td><td></td><td>0.311</td><td>0.461</td></tr><tr><td>7</td><td>VAT</td><td>x</td><td></td><td></td><td>x</td><td>0.358</td><td>0.539</td></tr><tr><td>8</td><td>VAT</td><td></td><td>x</td><td></td><td>x</td><td>0.362</td><td>0.542</td></tr><tr><td>9</td><td>Mixup</td><td></td><td>x</td><td></td><td>x</td><td>0.351</td><td>0.507</td></tr></tbody></table>
## 5.1. Separate regularization and consistency costs

The experiments that we conducted on random and adversarial augmentations suggest that it is advantageous to keep a separate regularization and consistency cost.

If we compare lines 2 and 3 of Table 1, which both use input noise as an augmentation strategy and differ only by the regularization method, we notice an improvement in *macro* score (increasing from 0.381 to 0.397) balanced by a drop in *psds* score (decreasing from 0.552 to 0.534). This leads to the following observation. Using *L*₂ regularization instead of MT regularization gives us an additional degree of freedom, which can be used to optimize either the *macro* score or the *psds* score. This property is useful. Indeed, *macro* and *psds* scores are partially conflicting: depending on the use case, it might be advantageous to optimize either of them [24].

If we compare lines 6 and 7 of Table 1, which both use VAT as an augmentation strategy and differ only by the regularization method, we notice this time an improvement both in *macro* score (increasing from 0.311 to 0.358) and *psds* score (increasing from 0.461 to 0.539). We hypothesize that, when the data augmentations become harder to handle for the model, it becomes increasingly advantageous to keep a separate regularization and consistency cost.

## 5.2. Simplified training method

As we have seen by focusing on the *macro* metric and comparing lines 2 and 3 of Table 1, the MT objective can be advantageously replaced by a *L*₂ regularization cost and a consistency cost penalizing the sensitivity of the model *f* to input noise. Further experiments have shown that this advantage is maintained even when the amount of data used for training is decreased. Moreover, the classification score is improved across classes. This algorithm thus seems to escape the curse of class dependency that has been recently discussed in the literature on regularization and data augmentations [25].

This new method is a simplification of the MT approach. As we have already mentioned, it makes it easier to analyze the individual impact of the training objectives (classification, regularization, and consistency), and to fine-tune their relative contribution during training. Moreover, since the *f*<sub>ema</sub> model is no longer necessary, we can divide by two the number of parameters kept in memory during training. Although we did not observe it during our experiments, this method may also lead to faster training. Indeed, the update of *f*<sub>ema</sub> is not necessary anymore, and the added regularization cost is already computed implicitly by some implementation of the Adam optimizer (for instance, this is the case in the widely used framework *Pytorch*). However, these advantages come at the cost of an additional hyperparameter to adjust.

## 5.3. Advantage of depth

The experiments that we conducted on random and adversarial augmentations suggest that it is advantageous to use latent augmentations.

If we compare lines 3 and 4 of Table 1, we notice that using latent noise instead of input noise leads to an increase in *psds* score. This training method offers a compromise between *macro* score (decreasing slightly from 0.397 to 0.388) and *psds* score (increasing from 0.534 to 0.565). This result could be explained by the following. The authors of [2] suggest that adding noise to the input of a classifier moves its decision boundaries away from the training data, and thus improves its generalization power. Following this

hypothesis, adding noise to the internal layers of *f* improves its performance as a classifier. This leads to an improvement of the *psds* score, which is more sensitive to the labeling accuracy. On the other hand, the *macro* score, which is more sensitive to the temporal accuracy, does not vary as much.

## 5.4. Adversarial and Mixup augmentations

Despite encouraging first results, the experiments we performed with adversarial and Mixup augmentations did not yield any improvement over our experiments with random noise. We may explain the failure of adversarial perturbations by observing that the generated samples are not realistic. Consequently, the invariants developed with these methods are not useful for the detection and classification of sound events. This motivates the study of data augmentations that are specific to the audio domain, and are more relevant for this task.

# 6. CONCLUSION

This article analyzes the impact of several domain agnostic data augmentation strategies on the baseline of task 4 of the DCASE challenge. We propose a simple variant of the MT method, which improves its performance. The results that we obtained with adversarial augmentations suggest that it may be advantageous to restrict the search space to realistic augmentations. In a future study, we will test this hypothesis and examine the impact of audio related augmentations on sound event detection. As the dimension of the augmentation space increases, we anticipate that the search strategy will become a central issue.

# 7. ACKNOWLEDGMENT

The material contained in this document is based upon work funded by the Agence National de la Recherche en Intelligence Artificielle (PhD program in AI) and Hi! PARIS through its PhD funding program.

# 8. REFERENCES

[1] Y. Wang, Q. Yao, J. T. Kwok, and L. M. Ni, "Generalizing from a few examples: A survey on few-shot learning," *ACM computing surveys (csur)*, vol. 53, no. 3, pp. 1–34, 2020.

[2] J. E. Van Engelen and H. H. Hoos, "A survey on semi-supervised learning," *Machine Learning*, vol. 109, no. 2, pp. 373–440, 2020.

[3] N. Turpault and R. Serizel, "Training sound event detection on a heterogeneous dataset," *arXiv preprint arXiv:2007.03931*, 2020.

[4] A. Rasmus, M. Berglund, M. Honkala, H. Valpola, and T. Raiko, "Semi-supervised learning with ladder networks," in *Advances in Neural Information Processing Systems*, vol. 28. Curran Associates, Inc., 2015. [Online]. Available: https://proceedings.neurips.cc/paper/2015/hash/378a063b8fdb1db941e34f4bde584c7d-Abstract.html

[5] P. Bachman, O. Alsharif, and D. Precup, "Learning with pseudo-ensembles," in *Advances in Neural Information Processing Systems*, vol. 27. Curran Associates, Inc., 2014. [Online]. Available: https://proceedings.neurips.cc/paper/2014/hash/66be31e4c40d676991f2405aaecc6934-Abstract.html
[6] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: a simple way to prevent neural networks from overfitting,” *The journal of machine learning research*, vol. 15, no. 1, pp. 1929–1958, 2014.

[7] S. Laine and T. Aila, “Temporal ensembling for semi-supervised learning,” *arXiv preprint arXiv:1610.02242*, 2016.

[8] B. T. Polyak and A. B. Juditsky, “Acceleration of stochastic approximation by averaging,” *Journal on control and optimization*, vol. 30, no. 4, pp. 838–855, 1992.

[9] A. Tarvainen and H. Valpola, “Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results,” *Advances in neural information processing systems*, vol. 30, 2017.

[10] V. Verma, K. Kawaguchi, A. Lamb, J. Kannala, Y. Bengio, and D. Lopez-Paz, “Interpolation consistency training for semi-supervised learning,” *arXiv preprint arXiv:1903.03825*, 2019.

[11] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, “mixup: Beyond empirical risk minimization,” *arXiv preprint arXiv:1710.09412*, 2017.

[12] Z. You, J. Ye, K. Li, Z. Xu, and P. Wang, “Adversarial noise layer: Regularize neural network by adding noise,” in *International Conference on Image Processing (ICIP)*. IEEE, 2019.

[13] S. Park, J. Park, S.-J. Shin, and I.-C. Moon, “Adversarial dropout for supervised and semi-supervised learning,” in *Proceedings of the AAAI conference on artificial intelligence*, vol. 32, no. 1, 2018.

[14] S.-M. Moosavi-Dezfooli, A. Fawzi, O. Fawzi, and P. Frossard, “Universal adversarial perturbations,” in *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition*, 2017, pp. 1765–1773.

[15] D. Berthelot, N. Carlini, I. Goodfellow, N. Papernot, A. Oliver, and C. A. Raffel, “Mixmatch: A holistic approach to semi-supervised learning,” *Advances in Neural Information Processing Systems*, vol. 32, 2019.

[16] A. Agnone and U. Altaf, “Virtual adversarial training system for DCASE 2019 task 4,” *Detection and Classification of Acoustics Scenes and Events (DCASE) Challenge*, 2019.

[17] M. Zöhrer and F. Pernkopf, “Virtual adversarial training and data augmentation for acoustic event detection with gated recurrent neural networks.” in *Interspeech*, 2017.

[18] R. Harb and F. Pernkopf, “Sound event detection using weakly-labeled semi-supervised data with GCRNNS, vat and self-adaptive label refinement,” *arXiv preprint arXiv:1810.06897*, 2018.

[19] H. Dinkel, X. Cai, Z. Yan, Y. Wang, J. Zhang, and Y. Wang, “A lightweight approach for semi-supervised sound event detection with unsupervised data augmentation,” in *Proceedings of the 6th Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE)*, Online, 2021.

[20] N. Turpault, R. Serizel, A. Parag Shah, and J. Salamon, “Sound event detection in domestic environments with weakly labeled data and soundscape synthesis,” in *Workshop on Detection and Classification of Acoustic Scenes and Events*, New York City, United States, October 2019. [Online]. Available: https://hal.inria.fr/hal-02160855

[21] M. Bertola, “Data augmentation methods exploration for sound event detection,” DCASE2022 Challenge, Tech. Rep., June 2022.

[22] C. Bilen, G. Ferroni, F. Tuveri, J. Azcarreta, and S. Krstulovic, “A framework for the robust evaluation of sound event detection,” *arXiv preprint arXiv:1910.08440*, 2019. [Online]. Available: https://arxiv.org/abs/1910.08440

[23] N. Turpault, R. Serizel, and E. Vincent, “Analysis of weak labels for sound event tagging,” Apr. 2021, working paper or preprint. [Online]. Available: https://hal.inria.fr/hal-03203692

[24] G. Ferroni, N. Turpault, J. Azcarreta, F. Tuveri, R. Serizel, Ç. Bilen, and S. Krstulović, “Improving sound event detection metrics: insights from DCASE 2020,” in *International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. IEEE, 2021.

[25] R. Balestriero, L. Bottou, and Y. LeCun, “The effects of regularization and data augmentation are class dependent,” *arXiv preprint arXiv:2204.03632*, 2022.
# STARSS22: A DATASET OF SPATIAL RECORDINGS OF REAL SCENES WITH SPATIOTEMPORAL ANNOTATIONS OF SOUND EVENTS

Archontis Politis<fnref n="1" />, Kazuki Shimada<fnref n="2" />, Parthasaarathy Sudarsanam<fnref n="1" />, Sharath Adavanne<fnref n="1" />, Daniel Krause<fnref n="1" />
Yuichiro Koyama<fnref n="2" />, Naoya Takahashi<fnref n="2" />, Shusuke Takahashi<fnref n="2" />, Yuki Mitsufuji<fnref n="2" />, Tuomas Virtanen<fnref n="1" />,

## ABSTRACT

This report presents the Sony-TAu Realistic Spatial Soundscapes 2022 (STARS22) dataset of spatial recordings of real sound scenes collected in various interiors at two different sites. The dataset is captured with a high resolution spherical microphone array and delivered in two 4-channel formats, first-order Ambisonics and tetrahedral microphone array. Sound events belonging to 13 target classes are annotated both temporally and spatially through a combination of human annotation and optical tracking. STARSS22 serves as the development and evaluation dataset for Task 3 (Sound Event Localization and Detection) of the DCASE2022 Challenge and it introduces significant new challenges with regard to the previous iterations, which were based on synthetic data. Additionally, the report introduces the baseline system that accompanies the dataset with emphasis on its differences to the baseline of the previous challenge. Baseline results indicate that with a suitable training strategy a reasonable detection and localization performance can be achieved on real sound scene recordings. The dataset is available in https://zenodo.org/record/6600531.

**Index Terms—** Sound event localization and detection, sound source localization, acoustic scene analysis, microphone arrays

## 1. INTRODUCTION

Sound event localization and detection (SELD) refers to the task of simultaneously detecting the presence and tracking the location of sound types of interest over time. It relates strongly to the more established tasks of sound event detection (SED) and sound source localization (SSL) but it adds spatial information to the first and semantic information to the second. The SELD task has recently seen increased research interest in part due to its introduction to the DCASE Challenge in 2019 [1]. The challenge dataset was generated with a collection of spatial room impulse responses (SRIRs) from 5 spaces and multiple source positions, convolved with dry isolated sound event recordings [2]. The next iteration of DCASE2020 increased the dataset diversity by including SRIRs of 10 additional rooms with stronger reverberation and, more importantly, by emulating dynamic scenes with both moving and static sound sources [3], while the DCASE2021 dataset introduced additionally directional interfering events out of the target classes [4].

The three SELD datasets of DCASE2019-2021 contributed to the continuous development and improvement of SELD methods by aiming to emulate accurately spatial and acoustical properties of sound scenes and to increase gradually scene complexity towards more realistic conditions. However, there are certain limitations inherent to generating synthetic mixtures. One such limitation is

the random presence of target classes and the random sequencing of sound events, discarding natural temporal occurrences or co-occurrences of certain sounds in a real scene. Another limitation is the randomized spatial distribution of sound events ignoring their spatial constraints and connections in a scene. To overcome such limitations, SELD systems should transition to training and evaluation with recordings of real sound scenes. Such datasets require strong event labels provided by human annotators and simultaneous spatial annotations provided by some form of automated tracking. Due to the required annotation effort and complexity, there are no published SELD datasets we know of except for the SECL-UMons one in [5], capturing natural sound events of 11 classes occurring at pre-defined locations in two spaces. However, even though the events have a natural spatial distribution, the dataset is limited to single event recordings in isolation or to combinations of two simultaneous events, ignoring spatio-temporal information linking events in a natural scene. A few more synthetic SELD datasets exist with the same limitations as the DCASE datasets, based on captured SRIRs and targeting certain applications, such as wearable arrays [6] or positional localization in a room with distributed arrays [7].

This report presents the first SELD dataset we are aware of where realistic scenes, loosely acted by multiple actors, are captured and annotated with strong labels temporally and spatially. The challenges of such annotations are dealt with a combination of human listening and optical tracking, employing multiple sensors and modalities. Since the sound scenes are acted naturally, the dataset overcomes the limitations of synthetic datasets discussed earlier. Target sound classes are not combined randomly but are instead constrained by the environment and the participants, while the presence of each class is determined by the natural composition of each scene. Causal and sequential occurrences of sound events, as well as co-occurrences, follow the actions of the actors and their interactions with the environment. The same holds for the location of events and their trajectories in case they are moving; their spatial distributions are naturally constrained by the type of event, while event trajectories can reveal scene information on the agents and their actions. Hence, the dataset opens certain new possibilities for SELD systems apart from evaluation in realistic scenarios.

The STARSS22 dataset serves as the development and evaluation dataset of DCASE2022 Task 3, and it is followed by a suitable baseline and evaluation setup. Changes with respect to the previous DCASE challenges are elaborated. Since the duration of the dataset is limited compared to the synthetic datasets used in previous years, use of external data is allowed in this iteration to improve model training and generalization. An example strategy based on additional synthetic data is presented for the baseline. Finally, results are presented on the development and evaluation set.

<footnote n="1">¹ Audio Research Group, Tampere University, Tampere, Finland</footnote>

<footnote n="2">² Sony Group Corporation, Tokyo, Japan</footnote>
## 2. DATASET

The Sony-TAu Realistic Spatial Soundscapes 2022 (STARSS22) dataset consists of recordings of real scenes captured with a high channel-count spherical microphone array (SMA). The recordings are conducted by two different teams at two different sites, Tampere University facilities in Tampere, Finland, and Sony facilities in Tokyo, Japan. Recordings at both sites share the same capturing and annotation process, organized in sessions corresponding to distinct rooms, human participants, and sound making props with a few exceptions. In each session, various clips are recorded with combinations of that session's participants acting some simple scenes and interacting between them and with the sound props. The scenes are not strongly scripted; they are based on generic instructions on the desired sound events and are otherwise improvised by the participants. The instructions serve as a rough guide to ensure adequate event activity and occurrences of the target sound classes in a clip.

Similarly to the previous challenges, the recordings are converted to two 4-channel spatial formats: first-order Ambisonics (FOA) and tetrahedral microphone array (MIC), both derived from the original 32-channel recordings. Conversion of the Eigenmike recordings to FOA following the SN3D normalization scheme (or ambiX) was performed with measurement-based filters according to [8]. Regarding the MIC format, channels 6, 10, 26, and 22 of the Eigenmike were selected, corresponding to a nearly tetrahedral arrangement. Analytical expressions of the directional responses of each format can be found in the DCASE2020 challenge report [3]. Finally, the converted recordings were downsampled to 24kHz.

The dataset is split into a development set (dev-set) and evaluation set (eval-set). The development set totals about 4 hrs 52 mins, of which 70 recording clips amounting to about 2 hrs are recorded in 4 different rooms in Tokyo and 51 recordings amounting to about 3 hrs are recorded in 7 different rooms in Tampere. To aid the development process, the development set is further split into a training part (dev-set-train, 40+27 clips in 2+4 rooms in Tokyo+Tampere) and a testing part (dev-set-test, 30+24 clips in 2+3 rooms in Tokyo+Tampere). The evaluation set is close to 2 hrs, recorded in 2 different rooms in Tokyo (35 clips) and in 3 different rooms in Tampere (17 clips).

### 2.1. Recording setup and process

Each scene was captured with 4 types of sensors: a) a high resolution 32-channel SMA (Eigenmike em32¹) recording the main multi-channel audio for the challenge, b) a 360° camera (Ricoh Theta V<fnref n="2" />) mounted about 10 cm above the SMA, c) a motion capture (mocap) system of infrared cameras surrounding the scene, tracking reflective markers mounted on the main actors and sound sources of interest (Optitrack Flex 13³), and d) wireless microphones mounted on the same tracked actors and sound sources, providing close-miked recordings of the main sound events (Røde Wireless Go II<fnref n="4" />). For each recording session, a suitable position of the Eigenmike and Ricoh Theta V was determined in order to cover the scene from a central position, while taking into account the intended scenarios and the specific room constraints. The origin of the mocap system was then set at ground level on the same position and the height of the Eigenmike was set at 1.5 m, while the mocap cameras were

<table><caption>Table 1: Relation of target classes to specific Audioset classes. Target classes not included in the table have an one-to-one relationship with the similarly named Audioset ones.</caption><thead><tr><td>Target Class</td><td>Related Audioset subclasses</td></tr></thead><tbody><tr><td>Telephone</td><td>Telephone bell ringing, Ringtone (no musical ringtones)</td></tr><tr><td>Domestic sounds</td><td>Vacuum cleaner, Mechanical fan, Boiling (produced by hoover, air circulator, water boiler)</td></tr><tr><td>Door, open or close</td><td>Combination of Door & Cupboard, open or close</td></tr><tr><td>Music</td><td>Background music & Pop music, (played by a loudspeaker in the room)</td></tr><tr><td>Musical instrument</td><td>Acoustic guitar, Marimba, Xylophone, Cowbell, Piano, Rattle (instrument)</td></tr><tr><td>Bell</td><td>Combination of sounds from hotel bell and glass bell, closer to Bicycle bell & single Chime</td></tr></tbody></table>

positioned at the boundaries of the room. Tracking markers were mounted to independent sound sources (such as next to the water sink, on a mobile phone on a table, on a hoover, or next to a guitar's soundhole). Head markers were additionally provided to the participants before each scene recording, in the form of headbands or hats. Tracking the head served as the reference point for all human made sounds. Mouth position for speech and laughter sounds, feet stepping position for footstep sounds, and hand position for clapping sounds were each approximated with a fixed translation from the head-tracking center close to the top of the head. Regarding clapping, participants were instructed to clap about 20 cm in front of their face to improve the position approximation. Head rotations were also logged during the scene with respect to the global coordinate frame of the mocap system. Finally, the wireless microphones were mounted to the lapel of each actor and to additional independent sound sources. A clapper sound was used to initiate the acting and to serve as a reference signal for synchronization between the different types of recordings.

### 2.2. Annotation process

Spatiotemporal annotations of the sound events were conducted manually by the authors and research assistants. Three types of information were required in order to obtain such annotations: a) the subset of the target classes that were active in each scene, b) the temporal activity of such class instances, and c) the position of each such instance when active. (a) was observed and logged during each scene recording. (b) was manually annotated by listening to the wireless microphone recordings. Since each such microphone would capture prominently sounds produced by the human actor or source it was assigned to, onset, offsets, source, and class information of each event could be conveniently extracted. In scenes or instances where associating an event to a source was ambiguous purely by listening, annotators would consult the video recordings to establish the correct association. The temporal annotation resolution was set to 100 msec.

After onset, offset, and class information of events was established for each source and actor in the scene, the positional annotations (c) were extracted for each such event by masking the tracker data with the temporal activity window of the event. Additionally,

<footnote n="1">¹https://mhacoustics.com/products#eigenmike1</footnote>

<footnote n="2">²https://theta360.com/en/about/theta/v.html</footnote>

<footnote n="3">³https://optitrack.com/cameras/flex-13/</footnote>

<footnote n="4">⁴https://rode.com/en/microphones/wireless/wirelessgoii</footnote>
<table><caption>Table 2: Dataset class activity information. The mean polyphony is computed over active frames only having one or more events present.</caption><thead><tr><th></th><th>Global</th><th>Fem.<br>speech</th><th>Male<br>speech</th><th>Clap</th><th>Phone</th><th>Laugh</th><th>Dom.<br>sounds</th><th>Footsteps</th><th>Door</th><th>Music</th><th>Music.<br>instr.</th><th>Faucet</th><th>Bell</th><th>Knock</th></tr></thead><tbody><tr><td>Frame coverage<br>(% total frames)</td><td>84.7</td><td>20.4</td><td>37.6</td><td>0.7</td><td>1.4</td><td>2.7</td><td>17.9</td><td>1.3</td><td>0.6</td><td>29.4</td><td>4.0</td><td>1.7</td><td>1.5</td><td>0.1</td></tr><tr><td>Max. polyphony</td><td>5</td><td>2</td><td>3</td><td>2</td><td>1</td><td>4</td><td>1</td><td>1</td><td>1</td><td>1</td><td>4</td><td>1</td><td>1</td><td>1</td></tr><tr><td>Mean polyphony</td><td>1.5</td><td>1.04</td><td>1.07</td><td>1.17</td><td>1.00</td><td>1.18</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.86</td><td>1.00</td><td>1.00</td><td>1.00</td></tr><tr><td>Polyphony 1<br>(% active frames)</td><td>61.5</td><td>96.1</td><td>93.3</td><td>83.4</td><td>100</td><td>84.0</td><td>100</td><td>100</td><td>100</td><td>100</td><td>52.2</td><td>100</td><td>100</td><td>100</td></tr><tr><td>Polyphony 2</td><td>29.55</td><td>3.9</td><td>6.5</td><td>16.6</td><td>0</td><td>14.5</td><td>0</td><td>0</td><td>0</td><td>0</td><td>16.6</td><td>0</td><td>0</td><td>0</td></tr><tr><td>Polyphony 3</td><td>7.15</td><td>0</td><td>0.2</td><td>0</td><td>0</td><td>1.1</td><td>0</td><td>0</td><td>0</td><td>0</td><td>24.2</td><td>0</td><td>0</td><td>0</td></tr><tr><td>Polyphony 4</td><td>1.6</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0.4</td><td>0</td><td>0</td><td>0</td><td>0</td><td>7.0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>Polyphony 5</td><td>0.2</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr></tbody></table>

class-specific translations to the tracking data were applied if necessary, as mentioned earlier for most human made sounds. Positional information was logged in Cartesian coordinates with respect to the mocap system's origin and subsequently converted to directions-of-arrival with respect to the center of the Eigenmike. Finally, the class, temporal, and spatial annotations were combined and converted to the text format used in the previous DCASE2019-2021 challenges. Validation of the annotations was performed by observing and listening to the 360° videos, overlaid with labeled markers positioned at the DOAs of the annotated events on the 360° video plane.

## 2.3. Target sound classes

A set of 13 target sound classes are selected to be annotated, based on the sound events captured prominently in the recorded scenes. The class labels are selected to conform to the Audioset ontology [9] and they are: *female speech/woman speaking*, *male speech/man speaking*, *clapping*, *telephone*, *laughter*, *domestic sounds*, *walk/footsteps*, *door open or close*, *music*, *musical instrument*, *water tap/faucet*, *bell*, *knock*. Since some of these labels correspond to superclasses with a large diversity of sounds and number of subclasses in the ontology (e.g. *domestic sounds* or *musical instrument*) we provide some additional information on the subset of sounds encountered in the recordings for some of the target classes, in the form of more specific audioset-related labels. This information is summarized in Table 1 and it can aid training and testing of methods. Certain directional sound events in the recordings are not annotated and are treated as directional interferers; examples include *computer keyboard*, *shuffling cards*, and *dishes, pots, and pans*. Additionally, there is natural background noise in all recordings, mostly HVAC-related, ranging from low to considerable levels. Based on the annotations, information on the percentage of frames that each class is active and the degree of polyphony globally and of each class separately is presented in Table 2.

# 3. BASELINE

## 3.1. Model architecture

The baseline of the DCASE2022 Task 3 challenge is similar to the one used in DCASE2021; a SELDnet-inspired CRNN architecture [10] improved with the ACCDOA output representation and loss [11]. However, due to the inability of the original ACCDOA representation to handle co-occuring events of the same class, the baseline adopts the recent *multi-ACCDOA* (mACCDOA) extension [12]. The mACCDOA model receives a sequence of *T* STFT frames of multichannel features and outputs $T/5 \times N \times C \times 3$

vector coordinates, where *C* is the number of target classes and *N* the maximum assumed number of co-occuring events in the recordings. For the current baseline *N* is set to 3 maximum simultaneous sources, while a value of 0.5 is used as the threshold on the length of the output vectors to indicate track and class activity. Note that a reduction of the STFT temporal resolution by a factor of 5 is performed to match the resolution of the annotations at every 100 msec.

Input features remain similar to the previous challenge [3]; namely, 4 channel 64-band log-mel spectrograms combined with acoustic intensity vectors for the FOA format or combined with generalized cross-correlation (GCC) sequences for the MIC format, following [13]. Additionally, the option of the SALSA-lite spatial features for the MIC format is added in the current baseline, recently shown to offer better performance than GCC in multi-source scenarios [14]. In this case, the original STFT spectrograms and the SALSA-lite features are truncated to include bins up to about 9 kHz, without mel-band aggregation, following [14]. A block diagram of the model architecture is presented in Fig. 1.

## 3.2. Model training

The baseline model is trained and evaluated twice: firstly only on the development set reporting baseline results for the participants to compare against during development. Secondly, it is trained on the development set and tested on the evaluation set, with results reported after the completion of the evaluation phase of the challenge. Since, the amount of training material is insufficient for the complexity of the task, additional material is synthesized for training. Those synthetic mixtures are generated with the same generation method and SRIRs as the DCASE2020-2021 datasets [3, 4]. 1200 one-minute spatial mixtures are synthesized (*synth-set*) using measured SRIRs from 9 rooms in TAU and sound event samples sourced from FSD50K [15]. The samples are chosen to match the target classes on the basis of their annotated labels which follow the Audioset ontology. The synthetic mixtures are made publicly available for reproducibility<fnref n="5" /> along with the list of the selected FSD50K sound samples. Additionally, the SRIRs are also publicly shared<fnref n="6" /> along with the scene generation code<fnref n="7" />, so that participants can generate their own synthetic mixtures for training following the same process if desired. The sets and splits for training and testing of the baseline for each phase are summarized in Table 3.

<footnote n="5">⁵https://doi.org/10.5281/zenodo.6406873</footnote>

<footnote n="6">⁶https://doi.org/10.5281/zenodo.6408611</footnote>

<footnote n="7">⁷https://github.com/danielkrause/DCASE2022-data-generator</footnote>
<figure><img src="image_99.png" /><figcaption>Figure 1: Baseline CRNN model with mACCDOA output.</figcaption></figure>

## 4. EVALUATION

Intermediate development set results are reported by the participants in the development set, while evaluation set results are computed by the organizers based on the submitted system outputs on the unseen evaluation set. Contrary to the previous challenges, participants are allowed to use external data during training, such as sample banks of sound events, room simulators, SRIR databases, spatial background noise recordings, pre-trained networks, and others. Generating the *synth-set* dataset to improve the baseline performance constitutes just one such example of external data usage.

The submissions are evaluated with the joint localization-detection metrics studied in [16, 1] and introduced first-time in DCASE2020. These are the location-dependent error rate ($ER_{20°}$) and F1-score ($F_{20°}$) for a spatial threshold 20° and the class-dependent localization error ($LE_{CD}$) and localization recall ($LR_{CD}$). Contrary to the previous challenges, in which $F_{20°}$ was micro-averaged, in this challenge evaluation is based on macro-averaging of F1-score to account better for the imbalanced presence of the target classes in the dataset.

<table><caption>Table 3: Datasets & splits used for baseline training and evaluation.</caption><thead><tr><td>Phase</td><td>Training</td><td>Testing</td></tr></thead><tbody><tr><td>Development</td><td>synth-set + dev-set-train</td><td>dev-set-test</td></tr><tr><td>Evaluation</td><td>synth-set + dev-set-train + dev-set-test</td><td>eval-set</td></tr></tbody></table>

<table><caption>Table 4: Baseline results on development and evaluation set. FOA-real refers to training only on the development set of STARSS22, FOA/MIC-mixed refers to training using additionally synthetic data.</caption><thead><tr><th></th><th>ER<sub>20°</sub> ↓</th><th>F<sub>20°</sub> ↑<br>(macro)</th><th>F<sub>20°</sub> ↑<br>(micro)</th><th>LE<sub>CD</sub> ↓</th><th>LR<sub>CD</sub> ↑</th></tr></thead><tbody><tr><td colspan="6"><strong>Development set</strong></td></tr><tr><td>FOA-real</td><td>0.78</td><td>0.11</td><td>-</td><td>64.1°</td><td>0.24</td></tr><tr><td>FOA-mixed</td><td>0.71</td><td>0.21</td><td>0.36</td><td>29.3°</td><td>0.46</td></tr><tr><td>MIC-mixed</td><td>0.71</td><td>0.18</td><td>0.36</td><td>32.2°</td><td>0.47</td></tr><tr><td colspan="6"><strong>Evaluation set</strong></td></tr><tr><td>FOA-mixed</td><td>0.61</td><td>0.24</td><td>0.39</td><td>22.9°</td><td>0.51</td></tr><tr><td>MIC-mixed</td><td>0.61</td><td>0.22</td><td>0.41</td><td>25.9°</td><td>0.48</td></tr></tbody></table>

## 4.1. Results

Results of the baseline on the development and evaluation set are presented on Table 4, for both FOA and MIC formats. The baseline was trained as indicated in Sec. 3.2 using the additional synthetic spatial mixtures of *synth-set*. For comparison purposes, an example of the model with FOA input trained only with real recordings is also reported (FOA-real), with the training and testing splits of Table 3 excluding the synthetic data (*synth-set*). It can be seen that the performance is very low in this case, at least without using data augmentation strategies. Two training strategies were tested with regards to incorporating the synthetic data. The first was based on initial training of the model on the synthetic data, followed by fine-tuning with the development dataset. The second simply mixed both the synthetic and the development recordings and trained with the combined dataset. Better results were obtained with the mixed strategy and these are the ones presented here (FOA/MIC-mixed). It is noted that the SRIRs used for the generation of *synth-set* were captured in TAU spaces that were different than the ones were the scene recordings of the STARS22 dataset occurred. Regarding the MIC format, both the GCC features and the SALSA-lite features were tested. Slightly better results were obtained with the GCC features and reported here. That may be attributed to the fact that even though the SALSA-lite features show a clear advantage for densely populated multi-source scenes such as the ones in DCASE2021 dataset [14], for the more sparse scenes of STARSS22 that advantage may be diminished. Finally, both the micro and macro versions of the F1-score are presented here, with a clear drop in performance in the macro version, as expected with a dataset of such unbalanced presence of target classes (evident in Table 2).

## 5. CONCLUSIONS

This report presents the specifications of the STARS22 dataset, intended for evaluation of SELD systems in challenging real conditions with a natural composition of sound events. The dataset serves as the development and evaluation dataset of the SELD challenge of DCASE2022 and it is accompanied by a baseline model which, with use of external data and a suitable training strategy, can achieve a reasonable performance on the evaluation dataset.

## 6. ACKNOWLEDGMENT

The dataset collection and annotation at Tampere University has been funded by Google. This work was carried out with the support of Centre for Immersive Visual Technologies (CIVIT) research infrastructure, Tampere University, Finland.
## 7. REFERENCES

[1] A. Politis, A. Mesaros, S. Adavanne, T. Heittola, and T. Virtanen, "Overview and evaluation of sound event localization and detection in DCASE 2019," *IEEE/ACM Trans. Audio, Speech, and Language Proc.*, 2020.

[2] S. Adavanne, A. Politis, and T. Virtanen, "A multi-room reverberant dataset for sound event localization and detection," in *Work. on Detection and Classification of Acoustic Scenes and Events (DCASE)*, October 2019, pp. 10–14.

[3] A. Politis, S. Adavanne, and T. Virtanen, "A dataset of reverberant spatial sound scenes with moving sources for sound event localization and detection," in *Work. on Detection and Classification of Acoustic Scenes and Events (DCASE)*, 2020, pp. 165–169.

[4] A. Politis, S. Adavanne, D. Krause, A. Deleforge, P. Srivastava, and T. Virtanen, "A dataset of dynamic reverberant sound scenes with directional interferers for sound event localization and detection," in *Work. on Detection and Classification of Acoustic Scenes and Events (DCASE)*, 2021, pp. 125–129.

[5] M. Brousmiche, J. Rouat, and S. Dupont, "Secl-umons database for sound event classification and localization," in *IEEE Int. Conf. on Acoustics, Speech and Sig. Proc. (ICASSP)*, 2020, pp. 756–760.

[6] K. Nagatomo, M. Yasuda, K. Yatabe, S. Saito, and Y. Oikawa, "Wearable seld dataset: Dataset for sound event localization and detection using wearable devices around head," in *IEEE Int. Conf. on Acoustics, Speech and Sig. Proc. (ICASSP)*, 2022, pp. 156–160.

[7] E. Guizzo, R. F. Gramaccioni, S. Jamili, C. Marinoni, E. Mas-saro, C. Medaglia, G. Nachira, L. Nucciarelli, L. Paglialunga, M. Pennese, et al., "L3DAS21 challenge: Machine learning for 3D audio signal processing," in *IEEE Int. Work. on Machine Learning for Sig. Proc. (MLSP)*, 2021, pp. 1–6.

[8] A. Politis and H. Gamper, "Comparing modeled and measurement-based spherical harmonic encoding filters for spherical microphone arrays," in *IEEE Work. on Applications of Sig. Proc. to Audio and Acoustics (WASPAA)*, 2017, pp. 224–228.

[9] J. F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, "Audio set: An ontology and human-labeled dataset for audio events," in *IEEE Int. Conf. on Acoustics, Speech and Sig. Proc. (ICASSP)*, 2017, pp. 776–780.

[10] S. Adavanne, A. Politis, J. Nikunen, and T. Virtanen, "Sound event localization and detection of overlapping sources using convolutional recurrent neural networks," *IEEE J. Selected Topics in Sig. Proc.*, vol. 13, no. 1, pp. 34–48, 2018.

[11] K. Shimada, Y. Koyama, N. Takahashi, S. Takahashi, and Y. Mitsufuji, "ACCDOA: Activity-coupled cartesian direction of arrival representation for sound event localization and detection," in *IEEE Int. Conf. on Acoustics, Speech and Sig. Proc. (ICASSP)*, 2021, pp. 915–919.

[12] K. Shimada, Y. Koyama, S. Takahashi, N. Takahashi, E. Tsunoo, and Y. Mitsufuji, "Multi-accdoa: Localizing and detecting overlapping sounds from the same class with auxiliary duplicating permutation invariant training," in *IEEE Int. Conf. on Acoustics, Speech and Sig. Proc. (ICASSP)*, 2022, pp. 316–320.

[13] Y. Cao, Q. Kong, T. Iqbal, F. An, W. Wang, and M. Plumbley, "Polyphonic sound event detection and localization using a two-stage strategy," in *Work. on Detection and Classification of Acoustic Scenes and Events (DCASE)*, 2019.

[14] T. N. T. Nguyen, D. L. Jones, K. N. Watcharasupat, H. Phan, and W.-S. Gan, "Salsa-lite: A fast and effective feature for polyphonic sound event localization and detection with microphone arrays," in *IEEE Int. Conf. on Acoustics, Speech and Sig. Proc. (ICASSP)*, 2022, pp. 716–720.

[15] E. Fonseca, X. Favory, J. Pons, F. Font, and X. Serra, "Fsd50k: an open dataset of human-labeled sound events," *IEEE/ACM Trans. on Audio, Speech, and Language Proc.*, vol. 30, pp. 829–852, 2021.

[16] A. Mesaros, S. Adavanne, A. Politis, T. Heittola, and T. Virtanen, "Joint measurement of localization and detection of sound events," in *IEEE Work. on Applications of Sig. Proc. to Audio and Acoustics (WASPAA)*, 2019, pp. 333–337.
# IMPROVING NATURAL-LANGUAGE-BASED AUDIO RETRIEVAL WITH TRANSFER LEARNING AND AUDIO & TEXT AUGMENTATIONS

Paul Primus<fnref n="1" />, Gerhard Widmer<fnref n="1" />,²

¹Institute of Computational Perception (CP-JKU)

²LIT Artificial Intelligence Lab

Johannes Kepler University, Austria

## ABSTRACT

The absence of large labeled datasets remains a significant challenge in many application areas of deep learning. Researchers and practitioners typically resort to transfer learning and data augmentation to alleviate this issue. We study these strategies in the context of audio retrieval with natural language queries (Task 6b of the DCASE 2022 Challenge). Our proposed system uses pretrained embedding models to project recordings and textual descriptions into a shared audio-caption space in which related examples from different modalities are close. We employ various data augmentation techniques on audio and text inputs and systematically tune their corresponding hyperparameters with sequential model-based optimization. Our results show that the used augmentations strategies reduce overfitting and improve retrieval performance.

**Index Terms—** Language-based Audio Retrieval, Transfer Learning, Audio Augmentation, Text Augmentation

## 1. INTRODUCTION

Natural-language-based audio retrieval is concerned with ranking audio recordings depending on their content's similarity to textual descriptions. Retrieval tasks like this are typically solved by converting recordings and textual descriptions into high-level representations and then aligning them in a shared audio-caption space; ranking can then be done based on the distance between embeddings. These systems' retrieval performance highly depends on the quality of the audio and text embedding models, which must extract features that accurately and discriminatively represent the high-level content. Current state-of-the-art approaches [1, 2, 3] create such feature extractors by training models with millions of parameters directly from raw input features, i.e., deep learning. These large embedding models require a large number of training examples, such as the 400 million image-text pairs used to train CLIP [4], a cutting-edge image-retrieval model. However, publicly available audio-caption datasets like Clotho and AudioCaps are significantly smaller. This work showcases how to use off-the-shelf pretrained audio and text neural networks to create a state-of-the-art retrieval model under this limiting condition. We evaluate our approach in the context of task 6b<fnref n="1" /> of the 2022's DCASE Challenge [5], which is concerned with audio retrieval from natural language descriptions. We demonstrate how an already well-performing baseline model can be further improved by using a range of audio and text augmentation methods and pretraining on AudioSet.

<figure><img src="image_100.png" /><figcaption>Figure 1: The proposed audio-retrieval system in a nutshell: Audio and descriptions are transformed into the shared audio-caption embedding space via the audio and description embedding models $\phi_a$ and $\phi_c$, respectively. The contrastive loss maximizes the similarities between matching pairs.</figcaption></figure>

## 2. RELATED WORK

The idea of aligning text and audio features for content-based retrieval is not new: Early audio retrieval methods connected bag-of-words text queries and MFCC features via density or discriminative models [6]. However, the handcrafted features and the relatively small vocabulary limited these methods' performance. Current methods build on top of learnable feature extractors that produce high-level audio and text representations from raw input features. Xie et al. [2], for example, used a convolutional recurrent neural network to extract frame-wise acoustic embeddings and aligned those to Word2Vec features via a linear transformation. Recently, language-based audio retrieval has received increased attention due to the newly introduced task 6b in the 2022's DCASE challenge [5]. The task's objective was to create a retrieval system that takes natural-language queries as input and retrieves the ten best-matching recordings from a test set. The top ranking systems among the nine submitted ones leveraged large pretrained audio and text embedding models like CNN14 [7] and BERT [8], respectively. While most systems applied SpecAugment [9], other data augmentation methods, especially text augmentations, have received little to no attention. We address this paucity and study a range of audio and text augmentation methods in the context of audio retrieval.

<footnote n="1">¹https://dcase community/challenge2022/ task-language-based-audio-retrieval</footnote>
<figure><img src="image_101.png" /><figcaption>Figure 2: Overview of the audio augmentation pipeline.</figcaption></figure>

### 3. RETRIEVAL SYSTEM

Our model uses separate audio and caption embedding networks $\phi_a(\cdot)$ and $\phi_c(\cdot)$ to embed tuples of spectrograms and descriptions $\{(a_i, c_i)\}_{i=1}^N$ into a shared $D$-dimensional space in a manner that represents of matching audio-caption pairs are close. This behavior is achieved by contrastive training, which equalizes the embeddings of matching audio-caption pairs $(a_i, c_i)$, while pushing the representations of mismatching pairs $(a_i, c_{j:j \neq i})$ apart. The agreement between audio $a_i$ and description $c_j$ is estimated via the normalized dot product in the shared embedding space:

$$C_{ij} = \frac{\phi_a(a_i)^T \cdot \phi_c(c_j)}{\|\phi_a(a_i)\|^2 \|\phi_t(c_j)\|^2}$$

The similarity matrix $C \in \mathbb{R}^{N \times N}$ holds the agreement of matching pairs on the diagonal and the agreement of mismatching pairs off-diagonal. We train the system using the NT-Xent [10] loss, which is defined as the average Cross Entropy (CE) loss over the audio and text dimension; the ground truth is given by the identity matrix $I \in \mathbb{R}^{N \times N}$:

$$\mathcal{L} = \frac{1}{2 \cdot N} \sum_{i=1}^{N} \mathrm{CE}(\mathbf{C}_{i*}, \mathbf{I}_{i*}) + \mathrm{CE}(\mathbf{C}_{*i}, \mathbf{I}_{*i})$$

### 4. AUDIO AUGMENTATIONS

To reduce overfitting of the audio embedding model and improve generalization, we employ three regularization techniques during training: Gain augmentation, MixStyle [11] along the frequency dimension (Freq-MixStyle), and SpecAugment [9]. Figure 2 gives an overview of the audio augmentation pipeline.

**Gain Augmentation** tries to make the model invariant changes in volume by randomly altering the loudness of the raw audio input signal. Volume manipulations are done by multiplying the waveform with factor $W$:

$$w = 10^{(g/20)}$$

The change in volume (in dB) is controlled with hyperparameter $g$; its value is randomly drawn from a uniform distribution in the range $[-g_{max}, g_{max}]$.

**SpecAugment** [9] randomly masks time and frequency stripes in the input spectrogram, thereby reducing the audio embedding model's reliance on specific input patterns. The number of stripes along the time and frequency dimensions is controlled via hyperparameters $n_f$ and $n_t$, respectively. Parameters $w_f$ and $w_t$ control the maximum width of the time and frequency stripes, respectively.

<table><caption>Table 1: Overview of the text augmentation pipeline</caption><thead><tr><th>Augmentation</th><th>Caption</th></tr></thead><tbody><tr><td>Original</td><td>The rain pours down.</td></tr><tr><td>Back Translation</td><td>It rains cats and dogs.</td></tr><tr><td>Insert</td><td>It tree rains cats and dogs.</td></tr><tr><td>Delete</td><td>It rains cats and dogs.</td></tr><tr><td>Swap</td><td>It and cats rains dogs.</td></tr><tr><td>Synonym</td><td>It drizzles cats and dogs.</td></tr></tbody></table>

The actual width and the offset of the stripes are chosen from a uniform distribution; masked values are replaced with zeros. We omitted the warping transformation proposed in the original work as it is computationally expensive and reportedly only lead to marginal improvements.

**Freq-MixStyle** [11] aims to transfer device-style characteristics between recordings by exchanging statistics along the frequency dimension of spectrograms. To this end, the original spectrogram is first normalized to zero mean unit variance along the frequency dimension and then un-normalized with adjusted mean and standard deviation statistics. The adjusted statistics are a convex combination of the original statistics ($\mu_i, \sigma - i$) and the statistic of a randomly selected spectrogram ($\mu_j, \sigma_j$):

$$\mu_{\text{new}} = \lambda\mu_i + (1 - \lambda)\mu_j$$

$$\sigma_{\text{new}} = \lambda\sigma_i + (1 - \lambda)\sigma_j$$

The coefficient $\lambda$ is drawn from a symmetric beta distribution in a manner that the original statistics always receive a higher weight:

$$\lambda \sim \text{Beta}(\alpha, \alpha)$$

$\alpha$ controls the shape of the Beta distribution. Freq-MixStyle is applied to each input example with a probability of $p_{MS}$.

### 5. TEXT AUGMENTATIONS

We apply Back Translation [12] and Easy Data Augmentation [13] (in that order) to reduce overfitting of the sentence embedding model. Examples of these augmentations are given in Table 1.

**Back Translation (BT)** [12] introduces variation into the input sentence without changing its semantics by translating the input sentence to a foreign language and back to the source language. We translate the training captions from English to German, French, or Spanish, and back to English using Google Translate.

**Easy Data Augmentation (EDA)** [13] chooses one of four word-level manipulations and applies the selected operation to each word with a certain probability (indicated in parenthesis): insertion of a random word ($p_{ins}$), deletion ($p_{del}$), swap with another word in the sentence ($p_{swp}$), or replacement with a synonym according to WordNet [14] ($p_{syn}$). EDA is applied with a probability of $p_{EDA}$.

### 6. EXPERIMENTS

We first established a baseline without augmentation and then conducted a series of experiments to investigate the impact of using pretrained weights for the audio embedding model, augmenting the
audio and text inputs, and pretraining on AudioCaps. We further investigate the impact of all augmentation methods separately in an ablation study. The model architecture and the exact experimental setup are discussed below.

## 6.1. Dataset & Input Features

We trained our proposed system on ClothoV2.1 [15], which contains 10-30 second long audio recordings sampled at 32kHz and five human-generated captions for each recording. We used the training, validation, and test split into 3839, 1045, and 1045 examples, respectively, as suggested by the dataset's creators. To make processing in batches easier, we zero-padded all audio snippets to the maximum audio length in the batch. The resulting waveforms were converted to 64-bin log-MEL spectrograms using a 1024-point FFT (32ms) and hop size of 320 (10ms). The audio features were normalized via batch normalization [16] along the frequency dimension before feeding them into the CNN10 embedding model. The input sentences were pre-processed by converting all characters to lowercase and removing punctuation. The resulting strings were tokenized with the WordPiece tokenizer [17], padded to the maximum sequence length in the batch, and truncated to 32 tokens.

## 6.2. Audio Embedding Model

We used a slightly modified version of the popular CNN10 architecture [7] to embed spectrograms into the 1024-dimensional audio-caption space. The architecture is detailed in Table 2. The network aggregates the output after the last convolutional block over the frequency and time dimensions and transforms the result with a two-layer neural network. The audio embedding model has approximately 9 Million parameters. We chose this simple architecture because it allowed us to train on a single customer-grade GPU with reasonable batch size. For the experiments with pretrained audio embedding model parameters, we transferred the weights of the convolutional blocks from a custom pretrained audio tagger and randomly initialized the fully-connected layers. The data set used for pretraining the embedding model, AudioSet [18], contains approximately 2 Million ten-second audio recordings labeled for 527 hierarchically organized classes. We used the pre-defined split of AudioSet into a large, unbalanced set for training and two smaller, more balanced sets for validation and testing. Pretraining of the embedding model was done as described in [7].

<table><caption>Table 2: The architecture of the audio embedding model (CNN10).</caption><thead><tr><th></th><th>CNN10</th></tr></thead><tbody><tr><td></td><td>2 × (3 × 3)@64, BN, ReLU<br>Pool (2 × 2)</td></tr><tr><td></td><td>2 × (3 × 3)@128, BN, ReLU<br>Pool (2 × 2)</td></tr><tr><td></td><td>2 × (3 × 3)@256, BN, ReLU<br>Pool (2 × 2)</td></tr><tr><td></td><td>2 × (3 × 3)@512, BN, ReLU<br>Pool (2 × 2)</td></tr><tr><td>Frequency Pooling (mean)</td><td>Time Pooling (average of mean and max)</td></tr><tr><td>FC 2048, ReLU</td><td>FC 1024</td></tr></tbody></table>

## 6.3. Text Embedding Model

We used a pretrained BERT model [8] ('bert-base-uncased') to generate embeddings for the audio captions. BERT is a bi-directional self-attention-based sentence encoder that was pretrained on Book-Corpus [19] and WikiText datasets [20] for masked language modeling and next sentence prediction. The learned semantic representations proved effective in multiple downstream tasks. We projected the output vector that corresponds to the class token into the shared audio-caption space by using a neural network with one hidden layer of size 2048 and ReLu activations. The text embedding model has approximately 112 Million parameters.

## 6.4. Training & Evaluation

We train all variants of the proposed system on ClothoV2.1's training set, select hyperparameters according to the performance on the validation set, and report the final results on the test set in section 7. Our main evaluation criterion was the mean Average Precision among the top-10 results (mAP) because this criterion takes the rank of the correct recording into account. We also report the recall among the top-1, top-5, and top-10 retrieved results. All results are averaged over three runs. Both embedding models were jointly optimized using gradient descent with a batch size of 30. We used the Adam update rule [21] for 50 epochs, set the initial learning rate to $10^{-4}$, and dropped it by a factor of 3 every 10 epochs. The hyperparameters of the optimizer were set to PyTorch's [22] defaults.

## 6.5. Sequential Model-Based Optimization

We performed sequential model-based optimization (SMBO) in the hyperparameter space of the audio and text augmentations to optimize the mAP-score on the validation set without manual tuning. Sequential Model-based Optimization (SMBO) utilizes the outcomes of prior experiments to build a surrogate model that estimates the relationship between validation-mAP and a given parameter configuration. Subsequent runs use this surrogate model to sample hyperparameter configurations from a distribution that is proportional to the expected mAP improvement. We initialized SMBO with ten runs using randomly chosen hyperparameters. After that, we performed 100 trials with hyperparameters sampled using the Tree-structured Parzen Estimator algorithm [23]. To reduce the overall computation time, we stopped runs for which the mAP on the validation set did not increase for ten consecutive epochs. Table 4 defines the hyperparameter search space for the SMBO.

## 7. RESULTS & DISCUSSION

The results of our experiments are summarized in Table 3 and discussed in the following section.

<table><caption>Table 3: Audio retrieval performance of the DCASE baseline and the custom system in four variants.</caption><thead><tr><th></th><th>R@1</th><th>R@5</th><th>R@10</th><th>mAP@10</th></tr></thead><tbody><tr><td>DCASE baseline</td><td>3.50</td><td>11.50</td><td>19.50</td><td>7.50 ± 0.00</td></tr><tr><td>baseline</td><td>6.63</td><td>20.06</td><td>31.52</td><td>12.53 ± 0.08</td></tr><tr><td>+ AudioSet pretraining</td><td>13.18</td><td>35.30</td><td>48.61</td><td>22.80 ± 0.29</td></tr><tr><td>+ augmentations</td><td>14.50</td><td>37.24</td><td>51.04</td><td>24.27 ± 0.19</td></tr><tr><td>+ AudioCaps pretraining</td><td>14.34</td><td>38.12</td><td>52.04</td><td>24.57 ± 0.15</td></tr></tbody></table>
## 7.1. Baseline

The performance of our custom baseline system, which was trained with randomly initialized audio embedding model parameters and without augmentation, is given in Table 3. The resulting system’s mAP is 5 pp. higher than the mAP of the DCASE baseline system [5] which we attribute to the more powerful text embedding model (we used BERT [8] instead of Word2Vec [24]).

## 7.2. AudioSet Pretraining

Next, we investigated the impact of using pretrained weights to initialize the audio embedding model. To that end, we retrained our baseline system but transferred the initial audio embedding model parameters from a CNN10 pretrained for tagging on AudioSet. The resulting model is approximately 10 pp. mAP better than the system that used randomly initialized weights for the audio encoder. This confirms that using pretrained audio embedding models is an effective strategy to alleviate the data scarcity problem.

## 7.3. Augmentations

We build upon the system that uses the AudioSet pretrained embedding model and perform SMBO to find a good hyperparameter configuration. The resulting best hyperparameters on the validation set and the performance on the test set are given in Tables 4 and 3, respectively. The best configuration found suggests that the parameter which controls the frequency of EDA is superfluous as the best value is very close to one. Synonym replacement appears beneficial, and future experiments should search for the optimal value for this parameter in a larger range. The probability of swapping and inserting random words is close to zero, suggesting that these two transformations are less beneficial or even detrimental. All in all, we observed an absolute improvement of approximately 1.5 pp. mAP when training with text and audio augmentations.

<table><caption>Table 4: Hyperparameter search space for the sequential model-based optimization, and the best configuration found.</caption><thead><tr><th></th><th>Augmentation</th><th>Parameter</th><th>Range</th><th>best</th></tr></thead><tbody><tr><td rowspan="6">Text</td><td rowspan="5">EDA</td><td>p<sub>EDA</sub></td><td>[0, 1.0]</td><td>.9936</td></tr><tr><td>p<sub>syn</sub></td><td>[0, 0.3]</td><td>.2962</td></tr><tr><td>p<sub>swp</sub></td><td>[0, 0.3]</td><td>.0085</td></tr><tr><td>p<sub>ins</sub></td><td>[0, 0.3]</td><td>.0269</td></tr><tr><td>p<sub>del</sub></td><td>[0, 0.3]</td><td>.1944</td></tr><tr><td>Backtranslation</td><td>p<sub>bt</sub></td><td>[0, 1.0]</td><td>.1812</td></tr><tr><td rowspan="6">Audio</td><td rowspan="4">SpecAugment</td><td>n<sub>f</sub></td><td>{0, 1}</td><td>1</td></tr><tr><td>w<sub>f</sub></td><td>{1, ..., 32}</td><td>4</td></tr><tr><td>n<sub>t</sub></td><td>{0, ..., 8}</td><td>7</td></tr><tr><td>w<sub>t</sub></td><td>{1, ..., 64}</td><td>58</td></tr><tr><td>Audio Gain</td><td>g<sub>max</sub></td><td>{0, ..., 6}</td><td>3</td></tr><tr><td>Freq-MixStyle</td><td>p<sub>MS</sub></td><td>[0, 1.0]</td><td>.1045</td></tr><tr><td></td><td>α</td><td>[0, 1.0]</td><td>.8286</td></tr></tbody></table>

## 7.4. AudioCaps Pretraining

We hypothesized that pretraining the retrieval system on additional audio-caption pairs could further improve audio-retrieval results; we, therefore, pretrained the system (with AudioSet pretraining and augmentations) on the 46K training examples in AudioCaps [25].

To this end, we used the same training procedure as described in Section 6.4 for pretraining and fine-tuning but decreased the initial learning rate for fine-tuning by a factor of 10. Table 3 gives the results. Pretraining on AudioCaps resulted only in a marginal improvement of 0.3pp mAP, which suggests that using AudioCaps for transfer learning in this naive way has no significant impact.

## 7.5. Ablation Study: Augmentations

Based on the previous results, we performed another ablation study to investigate the effect of the audio and text augmentations. To that end, we re-trained the system with AudioSet pretraining and augmentations twice: once without audio augmentations and once without text augmentations. The results are given in Table 5. Using all augmentations gave the best results. We observed a drop of 1.1 and 0.5 pp mAP without text and audio augmentations, respectively. This might indicate that the text augmentations have a larger impact than the audio augmentations, which might be caused by the large difference in trainable parameters between the sentence and audio embedding models.

To isolate the effect of each individual augmentation method, we further re-trained the system (with AudioSet pretraining and augmentations) in five variants, always leaving out one of the augmentation methods. The results are summarized in Table 5. The text augmentations have the largest impact, which is in line with the previous results: Leaving out EDA and BT reduced the mAP by 1.0 and 0.7 pp., respectively. Eliminating SpecAugment and Freq-MixStyle reduced the performance by 0.7 and 0.6 pp., respectively. Gain augmentation seems to have the least impact: eliminating it reduced the mAP by only 0.2pp.

<table><caption>Table 5: Results of the ablation study on data augmentation.</caption><thead><tr><th></th><th>R@1</th><th>R@5</th><th>R@10</th><th>mAP@10</th></tr></thead><tbody><tr><td>SMBO</td><td>14.50</td><td>37.24</td><td>51.04</td><td>24.27 ± 0.19</td></tr><tr><td>no audio aug</td><td>13.88</td><td>36.94</td><td>51.06</td><td>23.74 ± 0.16</td></tr><tr><td>no text aug</td><td>13.12</td><td>35.77</td><td>49.25</td><td>22.91 ± 0.08</td></tr><tr><td>no SpeAugment</td><td>13.50</td><td>36.60</td><td>50.91</td><td>23.53 ± 0.20</td></tr><tr><td>no FreqMixStyle</td><td>13.61</td><td>36.91</td><td>50.69</td><td>23.62 ± 0.14</td></tr><tr><td>no Gain Augment</td><td>14.84</td><td>37.81</td><td>50.95</td><td>24.05 ± 0.26</td></tr><tr><td>no BT</td><td>13.61</td><td>36.38</td><td>50.00</td><td>23.43 ± 0.18</td></tr><tr><td>no EDA</td><td>13.33</td><td>37.02</td><td>49.94</td><td>23.27 ± 0.03</td></tr></tbody></table>

## 8. CONCLUSION

This study set out to investigate transfer learning and data augmentation strategies to alleviate the data scarcity problem in natural-language-based audio retrieval. Our research has shown that using pretrained audio and text embedding models greatly increases the retrieval performance on ClothoV2. We enriched this already well-performing retrieval system with a range of augmentation methods and showed that augmenting both text and audio inputs significantly reduces overfitting. Finally, we further found that pretraining on AudioCaps only leads to non-significant improvements.

## 9. ACKNOWLEDGMENT

The LIT AI Lab is financed by the Federal State of Upper Austria.
## 10. REFERENCES

[1] A. Onceşcu, A. S. Koepke, J. F. Henriques, Z. Akata, and S. Albanie, "Audio retrieval with natural language queries," in 22nd Annual Conf. of the Int. Speech Communication Association, Interspeech, 2021.

[2] H. Xie, O. Räsänen, K. Drossos, and T. Virtanen, "Unsupervised audio-caption aligning learns correspondences between individual sound events and textual phrases," in Proc. IEEE Int. Conf. Acoustic., Speech and Signal Process., ICASSP, 2022.

[3] B. Elizalde, S. Deshmukh, M. A. Ismail, and H. Wang, "CLAP: learning audio concepts from natural language supervision," CoRR, vol. abs/2206.04769, 2022.

[4] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, "Learning transferable visual models from natural language supervision," in Proc. of the 38nd Int. Conf. on Machine Learning, ICML, 2021.

[5] X. Huang, S. Lipping, and T. n. Virtanen, "DCASE 2022 Challenge Task 6b: Language-Based Audio Retrieval," CoRR, vol. abs/2206.06108, 2022.

[6] D. Turnbull, L. Barrington, D. A. Torres, and G. R. Lanckriet, "Semantic annotation and retrieval of music and sound effects," IEEE Trans. Speech Audio Process., 2008.

[7] Q. Kong, Y. Cao, T. Iqbal, Y. Wang, W. Wang, and M. D. Plumbley, "PANNs: Large-scale pretrained audio neural networks for audio pattern recognition," IEEE ACM Trans. Audio Speech Lang. Process., 2020.

[8] J. Devlin, M. Chang, K. Lee, and K. Toutanova, "BERT: pre-training of deep bidirectional transformers for language understanding," in Proc. of the North American Ch. of the Ass. for Computational Linguistics: Human Language Technologies, NAACL-HLT, 2019.

[9] D. S. Park, W. Chan, Y. Zhang, C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le, "SpecAugment: A simple data augmentation method for automatic speech recognition," in 20th Annual Conf. of the Int. Speech Communication Association, Interspeech, 2019.

[10] T. Chen, S. Kornblith, M. Norouzi, and G. E. Hinton, "A simple framework for contrastive learning of visual representations," in Proc. of the 37nd Int. Conf. on Machine Learning, ICML, 2020.

[11] K. Zhou, Y. Yang, Y. Qiao, and T. Xiang, "Domain generalization with mixstyle," in 9th Int. Conf. on Learning Representations, ICLR, 2021.

[12] R. Sennrich, B. Haddow, and A. Birch, "Improving neural machine translation models with monolingual data," in Proc. of the 54th Annual Meeting of the Association for Computational Linguistics, ACL, 2016.

[13] J. W. Wei and K. Zou, "EDA: easy data augmentation techniques for boosting performance on text classification tasks," in Proc. of the 2019 Conf. on Empirical Methods in Natural Language Processing and the 9th Int. Joint Conf. on Natural Language Processing, EMNLP-IJCNLP, K. Inui, J. Jiang, V. Ng, and X. Wan, Eds., 2019.

[14] G. A. Miller, "Wordnet: A lexical database for english," Commun. ACM, 1995.

[15] K. Drossos, S. Lipping, and T. Virtanen, "Clotho: an Audio Captioning Dataset," in Proc. IEEE Int. Conf. Acoustic., Speech and Signal Process., ICASSP, 2020.

[16] S. Ioffe and C. Szegedy, "Batch normalization: Accelerating deep network training by reducing internal covariate shift," in Proc. of the 32nd Int. Conf. on Machine Learning, ICML, 2015.

[17] Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey, J. Klingner, A. Shah, M. Johnson, X. Liu, L. Kaiser, S. Gouws, Y. Kato, T. Kudo, H. Kazawa, K. Stevens, G. Kurian, N. Patil, W. Wang, C. Young, J. Smith, J. Riesa, A. Rudnick, O. Vinyals, G. Corrado, M. Hughes, and J. Dean, "Google's neural machine translation system: Bridging the gap between human and machine translation," CoRR, vol. abs/1609.08144, 2016.

[18] J. F. Gemmeke, D. P. W. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, "Audio Set: An ontology and human-labeled dataset for audio events," in Proc. IEEE Int. Conf. Acoustic., Speech and Signal Process., ICASSP, 2017.

[19] Y. Zhu, R. Kiros, R. S. Zemel, R. Salakhutdinov, R. Urtasun, A. Torralba, and S. Fidler, "Aligning books and movies: Towards story-like visual explanations by watching movies and reading books," in IEEE Int. Conf. on Computer Vision, ICCV, 2015.

[20] S. Merity, C. Xiong, J. Bradbury, and R. Socher, "Pointer sentence mixture models," in 5th Int. Conf. on Learning Representations, ICLR, 2017.

[21] D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization," in 3rd Int. Conf. on Learning Representations, ICLR, 2015.

[22] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf, E. Z. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, "PyTorch: An imperative style, high-performance deep learning library," in Annual Conf. on Neural Information Processing Systems, NEURIPS, 2019.

[23] J. Bergstra, R. Bardenet, Y. Bengio, and B. Kégl, "Algorithms for hyper-parameter optimization," in Annual Conf. on Neural Information Processing Systems, NEURIPS, 2011.

[24] T. Mikolov, K. Chen, G. Corrado, and J. Dean, "Efficient estimation of word representations in vector space," in 1st Int. Conf. on Learning Representations, ICLR, 2013.

[25] C. D. Kim, B. Kim, H. Lee, and G. Kim, "AudioCaps: Generating captions for audios in the wild," in Proc. of the North American Ch. of the Ass. for Computational Linguistics: Human Language Technologies, NAACL-HLT, 2019.
# DESCRIPTION AND ANALYSIS OF NOVELTIES INTRODUCED IN DCASE TASK 4 2022 ON THE BASELINE SYSTEM

Francesca Ronchini<fnref n="1" />*, Samuele Cornell²*, Romain Serizel<fnref n="1" />, Nicolas Turpault<fnref n="1" />, Eduardo Fonseca³, Daniel P. W. Ellis³

¹ Universite de Lorraine, CNRS, Inria, Loria, Nancy, France

²Department of Information Engineering, Universita Politecnica delle Marche, Italy

³ Google Research, United States

## ABSTRACT

The aim of the Detection and Classification of Acoustic Scenes and Events Challenge Task 4 is to evaluate systems for the detection of sound events in domestic environments using an heterogeneous dataset. The systems need to be able to correctly detect the sound events present in a recorded audio clip, as well as localize the events in time. This year's task is a follow-up of DCASE 2021 Task 4, with some important novelties. The goal of this paper is to describe and motivate these new additions, and report an analysis of their impact on the baseline system. We introduced three main novelties: the use of external datasets, including recently released strongly annotated clips from Audioset, the possibility of leveraging pre-trained models, and a new energy consumption metric to raise awareness about the ecological impact of training sound events detectors. The results on the baseline system show that leveraging open-source pre-trained on AudioSet improves the results significantly in terms of event classification but not in terms of event segmentation.

***Index Terms***— Sound event detection, synthetic soundscapes, deep learning, external data, energy consumption

## 1. INTRODUCTION

Machine Listening has a substantial impact on applications such as noise monitoring in smart cities [1, 2], smart homes and home security solutions [3, 4], health monitoring systems [5], bio-acoustics [6], and hearing aids [7], among others. Sound Event Detection (SED) is one particularly important task in the field of Machine Listening. Its goal is to correctly output the class of different sound events present in an audio clip, together with each sound event's time boundaries [8]. Multiple events can be active simultaneously in each audio recording, but target sound events can also overlap with other non-target sound events. This research area is in continual development, attracting an expanding community.

Since 2018, the Detection and Classification Acoustic Scene and Events (DCASE) Challenge Task 4 evaluates systems for detection of sound events in audio clips recorded in domestic environments. The challenge's main goal is to explore to what extent it is possible to exploit an unbalanced and heterogeneous dataset, e.g. containing a limited subset of weakly labeled and strongly labeled data and a larger amount of unlabeled data. This is especially important as strongly labeled data is expensive and time-consuming to obtain while weakly labeled and in particular unlabeled data are much more accessible and scalable. Moreover, unlabeled data and

weakly labeled data raise significantly fewer privacy concerns. In fact, weak labels could be obtained in an automated manner and, as such, human annotation is not required. This allows using on-device federated techniques which are more privacy-friendly.

Unlabeled data is usually leveraged via self-supervised learning. Indeed, past iterations of DCASE Challenge Task 4 have mainly explored this path, with most of the participants employing the mean-teacher technique [9] for this purpose. However, another possible direction is leveraging pre-trained models from related tasks such as Sound Event Classification (SEC) to obtain a SED system, e.g. via pseudo-labeling, fine-tuning or by using the pre-trained models' internal activations as additional high-level features. In the last couple of years several SEC models [10, 11] trained on the large AudioSet [12] have been open-sourced, making this approach particularly appealing. The use of such pre-trained models has been recently boosted by advancements in self-supervised learning, which enables the training in an unsupervised way on massive amounts of unlabeled data. These models can then be fine-tuned for multiple downstream tasks. A shining example in the audio domain is Wav2Vec 2.0 [13]. While powerful, these models are usually expensive to train and run (in particular self-attention based ones such as Wav2Vec 2.0 [13]), and unsuitable for widespread deployment on on-edge devices. On this premise, this year we introduced the following novelties for the DCASE Challenge Task 4:

* we allowed the use of external datasets and embeddings extracted from open-source pre-trained models. Participants were encouraged to propose open-source models to use and external data sources, such as AudioSet [12], which might also include real-word strongly annotations [14].

* we developed a new codebase<fnref n="1" /> and new baseline models to encourage participants to explore pre-trained models and such new datasets. We present the results of such baselines in Section 5.

* finally, we introduced a new energy consumption metric based on CodeCarbon toolkit [15]. This direction aims to foster interest among participants in finding new solutions for effective but also efficient SED systems. This was mainly a pilot experiment as the reliability of CodeCarbon across various computational platforms has not been fully investigated. We explain in detail the new metric in Section 3.2, together with the challenges that needs to be addressed in order to meaningfully compare the energy consumption of different models on different hardware

<footnote>* denotes equal contribution.</footnote>

<footnote n="1">¹Available at github.com/DCASE-REPO/DESED_task/blob/master/recipes/dcase2022_task4_baseline</footnote>
platforms.

The goal of this paper is to describe and motivate these new directions of this year’s challenge, and to analyse their impact on the baseline performance.

## 2. DCASE 2022 CHALLENGE TASK 4: NOVELTIES AND MOTIVATIONS

In every edition, the DCASE Challenge Task 4 proposes to focus on new research questions, targeting specific aspects of the SED task that are considered interesting and timely for the research community, with the ultimate goal of advancing the state-of-the-art.

This year, we investigate three main aspects that are motivated by the availability of large scale datasets related to the SED task, the recent popularity of pre-trained generic audio representation and the growing concerns regarding the environmental impact of our digital life. These developments were proposed in order to target three key scientific questions related to SED systems that we believe are worth addressing.

### 2.1. What is the impact of using external data and pre-trained models on SED systems?

In the DCASE 2021 Challenge Task 4 [16], we found that using a separation model trained in an unsupervised way using MixIT [17] on the massive YFC100m dataset [18] yielded a significant performance boost when used in conjunction with the baseline SED model on the evaluation set even if results on the development set were not that promising. Motivated by such result, this year we allowed participants to also use external data for the purpose of improving SED performance. To encourage participants to explore this path, we also provided a baseline which employs embeddings from two state-of-the-art popular models trained on AudioSet [12]: PANNs [10] and AST [11]. In addition to being allowed to use AudioSet and these two models, participants were granted the use of other pre-trained models such as YAMNet [19] and also datasets not strictly related to SED such as MUSAN [20] and ImageNet (used in AST [11] for example). Participants were also welcomed to propose other pre-trained models and external datasets. The full list of external resources allowed for the task can be found on the task website<fnref n="2" /> and in [21, 22].

Each team was allowed to submit four different systems. However, in order to highlight the impact of external resources on SED systems, we required each team to submit at least one system that was not using external data.

### 2.2. Is strongly annotated real-world data necessary to build an effective SED system?

In Hershey et al. [14] it was found that strong labels (with temporally precise onset and offset together with sound-event class label) can bring substantial benefits in SEC applications even when they are provided for a small fraction of the total data, which can remain weakly labeled (only the sound class label is provided without any temporal precision). This is especially remarkable since manual annotating data is costly and time-consuming, but also bias-prone due to human errors and disagreement on the perception of some sound

event onsets and offsets. On the other hand, synthetic data is cheap to obtain, but also inherently mismatched with respect to real-world data, leading to potentially sub-optimal performance [23]. In an attempt to mitigate this mismatch, and make the synthetic audio more realistic, in the past edition, non-target events have been included in the synthetic split of the training dataset [24]. This year, among the allowable external data source, we also included 3470 strongly labeled recorded audio clips coming from AudioSet [14]. The goal is to assess if the substantial improvement observed for SEC in Hershey et al. [14] also translates to SED and to what extent it may be worth spending extra resources for more manual annotation.

### 2.3. What is the environmental footprint of our SED systems?

Current state-of-the-art SED systems heavily rely on deep learning. Numerous recent works [25, 26] have raised concerns about the massive environmental costs of training deep learning models with large amount of parameters on massive amounts of data. For example, focusing on the audio domain, in Parcollet et al. [26], a study on the carbon footprint for ASR training was performed using the CodeCarbon toolkit<fnref n="3" />, a software package that estimates the amount of energy consumption and carbon dioxide produced by the cloud or personal computing resources used to execute the code. One of the key takeaways from this work is the tremendous inefficiency of many current ASR state-of-the-art models that trade off significant energy consumption (and thus pollution) for a marginal increase in performance that is likely not significant in actual deployment scenarios. These results raise important questions regarding the direction of ASR research, and suggests that the blind pursuit of the best possible performance in spite of the energy efficiency is likely not worth from a practical standpoint.

Motivated by this study, this year we added support in the DCASE Challenge 2022 Task 4 baseline and repository for energy consumption benchmarking based on CodeCarbon. Participants were encouraged to submit their CodeCarbon estimated kilowatt-hour kWh energy consumption figures for each submission, both for model training and, contrary to Parcollet et al. [26], also inference on evaluation data. In fact, we argue that the energy consumption in inference is more important than the training one as the model could end up being deployed on thousands of devices and ran for years, with cumulative energy consumption quickly outpacing training phase. We describe more in detail the new CodeCarbon energy consumption metric and report energy consumption figures for this year baselines in Section 3.2.

## 3. DCASE 2022 CHALLENGE TASK 4 BASELINE SYSTEM

This year's challenge baseline is based on a convolutional recurrent neural network (CRNN) but includes the main novelty of having the possibility of using features extracted from pre-trained models. CRNN was already found to be the architecture, mainly taken from [27], is composed of a CNN module followed by a 2-layers bi-directional gated recurrent unit (biGRU). The CNN has 7 layers, each composed of batch normalization, gated linear unit and dropout. Input features are Log-Mel Filterbank Energies extracted with a 128 ms window and 16 ms stride. The model is trained with the mean-teacher strategy [9, 27] on audio data resampled at 16 kHz and outputs one frame-wise prediction each 64 ms. To leverage more effectively weakly and unlabeled labeled data, attention pooling is employed, as outlined in [27], to derive clip-level predictions

<footnote n="2">²dcase community/challenge 2022/
task-sound-event-detection-in-domestic-environments#
external-data-resources</footnote>

<footnote n="3">³https://codecarbon.io/</footnote>
from frame-level predictions. From 2020, small changes, such as MixUp [28], and hyper-parameters improvements have been implemented on the baseline system, based on top-ranked systems submitted every years from participants.

## 3.1. Integration with Pre-Trained Models

As mentioned in Section 2.1, among the main novelties in this year challenge is the allowance of pre-trained models and external data. Participants were encouraged to explore this direction by an expanded codebase in the official challenge repository. We implemented support for two different pre-trained models: PANNs and AST, which we use to extract embeddings to aid in the SED task. In detail, these embeddings are late-fused into the CRNN classifier described previously, before the biGRU module, by combining them with the features extracted by the CNN module. During training the pre-trained model is kept frozen, and it is used only as an embeddings extractor. The CRNN SED classifier is instead trained with the mean-teacher strategy as outlined before.

We consider two types of embeddings extracted from pre-trained models internal activations: global embeddings, which are extracted at clip-level $\in \mathbb{R}^C$, and frame-wise embeddings, which instead are frame-wise $\in \mathbb{R}^{S_e \times C}$, where $C$ is the dimension of each vector and $S_e$ is the length of the embeddings sequence.

Regarding PANNs, global embeddings are extracted after the mean and max pooling layers, following the original work [10]. The frame-wise embeddings instead are extracted from the third convolutional block, after dropout. Regarding AST, also in this case we follow the original work for global embeddings [11] and derive them from the layer before the classification head. Frame-wise embeddings instead come from the last transformer layer.

As illustrated in Figure 1, left panel, global embeddings are simply fused with CNN features $\in \mathbb{R}^{S_c \times D}$ via concatenation at each time-step $s \in [1, \dots, S_c]$. In this work $D = 128$. We firstly map the embeddings to same dimensionality as the CNN features $D$ and apply layer-normalization. Then these are concatenated with the CNN features at each time-step along the channel dimension, obtaining a tensor $\mathbb{R}^{S_c \times 2D}$, another dense layer plus layer-normalization is used to shrink back the representation to $\mathbb{R}^{S_c \times D}$ before the biGRU. For frame-wise embeddings they cannot be concatenated right away as usually $S_e \neq S_c$, since PANNs, AST and the CNN front-end have different pooling factors. Thus we employ a single-layer BiGRU to encode the frame-wise embeddings into a representation with fixed dimensionality. As illustrated in Figure 1, right panel, we use the biGRU encoder last step output $\in \mathbb{R}^H$, where $H$ is 1024 channels, twice the hidden size of the biGRU encoder, here we use 512 neurons. This encoded representation is then fed to the same pipeline employed for global embeddings.

## 3.2. CodeCarbon metric

As mentioned in Section 2.1, this year we proposed to optionally report the energy consumption in kWh at training and test time. The goal here is to raise awareness regarding the environmental footprint of machine listening systems and SED systems in particular. Since every participant uses a different hardware platform to perform training and inference, absolute energy consumption figures are not directly comparable and cannot be used to assess each system efficiency. To allow for a more fair and meaningful comparison we asked for the participants to also provide, for the dev-test and eval datasets, the energy consumption obtained when inference is

<figure><img src="image_102.png" /><figcaption>Figure 1: Combining pre-trained models embeddings with the CRNN Baseline via late-fusion before the biGRU module. Left: fusion with global embeddings. Right: fusion with frame-wise embeddings. We report the tensor dimensions as sequenceLength × Channels.</figcaption></figure>

performed with the CRNN baseline system. The CRNN baseline kWh can then be used effectively as a common measure, mitigating the factors of variations from hardware difference. As such, in order to analyze the systems performance in terms of SED together with their energy consumption, we used a tentative, trivial energy weighted polyphonic sound detection score (EW-PSDS):

$$ \text{EW-PSDS} = \text{PSDS} * \frac{\text{kWh}_{\text{baseline}}}{\text{kWh}_{\text{submission}}} \quad (1) $$

where PSDS is the polyphonic sound event detection scores [29], kWh$_{baseline}$ is the energy consumption reported for the baseline, and kWh$_{submission}$ is the energy consumption of the submitted system. Since providing energy consumption was not mandatory for participants, this initial experiment aims to provide insights to design more reliable protocols to obtain energy consumption report from challenge participants and more reliable metrics to report SED performance and energy consumption altogether. In particular, the proposed metric is very challenge-centric as it systematically relies on the energy consumption of the baseline as reference. Additionally, early results have shown that it is heavily biased by large energy consumption differences.

## 4. CHALLENGE DATASETS AND EVALUATION METRICS

The dataset considered on this paper is the DESED dataset<fnref n="4" /> [30, 31], which is the same as provided for the DCASE 2021 Challenge Task 4. It is composed of 10 seconds length audio clips either recorded in a domestic environment or synthesized to reproduce such an environment<fnref n="5" />. The synthetic part of the dataset is generated with Scaper [32], a Python library for soundscape synthesis and augmentation.

The foreground events (both target and non-target) are obtained from the Freesound Dataset (FSD50k) [33], while the background sounds are obtained from the SINS dataset (activity class “other”) [34] and TUT scenes 2016 development dataset [35]. The event co-occurences are computed on a set of strong annotations from Audioset [14]. More information regarding the generation of the DESED dataset can be found in Ronchini et al. [24].

<footnote n="4">⁴https://project.inria.fr/desed/</footnote>

<footnote n="5">⁵For a detailed description of the DESED dataset and how it is generated the reader is referred to the original DESED article [31] and DCASE 2021 task 4 webpage: http://dcase community/challenge2021</footnote>
<table><caption>Table 1: Results for the baseline system with additional external data or pre-trained models embeddings. We report PSDS for two application scenarios as described in Section 4.</caption><thead><tr><th rowspan="2"></th><th colspan="2">PSDS-1 ↑</th><th colspan="2">PSDS-2 ↑</th></tr><tr><th>dev-test</th><th>eval</th><th>dev-test</th><th>eval</th></tr></thead><tbody><tr><td>Baseline</td><td>0.336</td><td>0.315</td><td>0.536</td><td>0.543</td></tr><tr><td>w. AudioSet strong</td><td>0.351</td><td><strong>0.345</strong></td><td>0.552</td><td>0.540</td></tr><tr><td>w. AST-frame</td><td>0.313</td><td>0.290</td><td><strong>0.722</strong></td><td><strong>0.678</strong></td></tr><tr><td>w. AST-global</td><td>0.205</td><td>0.192</td><td>0.369</td><td>0.305</td></tr><tr><td>w. PANNs-frame</td><td>0.354</td><td>0.304</td><td>0.635</td><td>0.597</td></tr><tr><td>w. PANNs-global</td><td><strong>0.375</strong></td><td>0.308</td><td>0.668</td><td>0.584</td></tr></tbody></table>

<table><caption>Table 2: Energy consumption (kWh) and Energy Weighted PSDS scores obtained on an Nvidia A100 GPU during inference on dev-test and evaluation.</caption><thead><tr><th rowspan="2"></th><th colspan="2">kWh ↓</th><th colspan="2">EW-PSDS-1 ↑</th><th colspan="2">EW-PSDS-2 ↑</th></tr><tr><th>Dev-test</th><th>Eval</th><th>Dev-test</th><th>Eval</th><th>Dev-test</th><th>Eval</th></tr></thead><tbody><tr><td>Baseline</td><td><strong>0.030</strong></td><td><strong>0.617</strong></td><td><strong>0.336</strong></td><td><strong>0.315</strong></td><td><strong>0.536</strong></td><td><strong>0.543</strong></td></tr><tr><td>w. AST-frame</td><td>0.061</td><td>0.901</td><td>0.149</td><td>0.198</td><td>0.344</td><td>0.464</td></tr><tr><td>w. AST-global</td><td>0.063</td><td>0.873</td><td>0.097</td><td>0.136</td><td>0.181</td><td>0.215</td></tr><tr><td>w. PANNs-frame</td><td>0.045</td><td>0.713</td><td>0.236</td><td>0.263</td><td>0.423</td><td>0.516</td></tr><tr><td>w. PANNs-global</td><td>0.045</td><td>0.724</td><td>0.262</td><td>0.285</td><td>0.445</td><td>0.497</td></tr></tbody></table>

As evaluation metrics we mainly use polyphonic sound event detection scores (PSDS) [29] and consider two different applications scenarios. The first scenario targets the need of the systems to accurately detect the onset and offset of the sound event, while the second scenario penalizes more confusion between classes, but the temporal localization is less crucial [23].

## 5. IMPACT OF 2022 CHALLENGE NOVELTIES ON BASELINE SYSTEM

Tables 1 and 2 report the results of this study. In Table 1 we study the impact of pre-trained models and additional strongly labeled data on the challenge baseline SED system described in Section 3. Firstly, we can observe that the addition of strongly-labeled data from Audioset, as expected, improves the PSDS-1 but has practically no effect on PSDS-2. Instead, the use of embeddings from pre-trained models seems to have the opposite trend, it brings benefits mainly to PSDS-2, with AST-frame and PANNs-global coming on top. This is intuitive as PANNs and AST are trained to perform SEC. AST-global seems to perform very poorly. This is explained by the fact that the embeddings in this model are taken directly before the final classification output projection and thus the representation may be too much biased towards SEC. On the other hand, as far as PANNs embeddings are concerned, global embeddings offer the best performance overall but the two are very close. Note that differently from AST, the global embeddings in PANNs are not taken from the layer just before the final linear classification projection but rather after the mean pooling layer.

The takeaway from these experiments is that downstream performance can heavily depend on the choice of the layer at which the pre-trained model embeddings are extracted. Instead of considering just one layer, future works could consider fusing features from many different layers, for example via self-attention.

In Table 2 we compare the energy consumption of the different architectural variations of the baseline system: PANNs versus AST and frame versus global embeddings. In detail we report kWh using

CodeCarbon as explained in Section 3.2 on dev-test and evaluation. We also report the relative EW-PSDS scores so that we can take into account each model efficiency. We can notice that AST models consistently require more energy in inference than PANNs. This is in accordance with the number of parameters 80M for PANNs and 88.1M for AST, and with the fact that AST are based on self-attention which is more computationally expensive than convolution. Secondly we can notice that there is little to no difference between frame and global embeddings. This is unexpected because we know that the baseline with frame-wise embeddings should be slightly more computational demanding due to the biGRU encoder as illustrated in Figure 1. This is likely due to the fact that the pre-trained model embedding extraction likely lead the energy consumption figure and CodeCarbon is simply not sensitive enough to pick up this subtle architectural difference. A more fair measure would be floating points operations (FLOP) for each frame prediction in output, however this has also the drawback that it is not easy to derive in a scalable way for multiple different models as current open source toolkits lack support for many operations.

Regarding the energy-weighted PSDS measures we can see that overall the most efficient system seems to be the “plain” baseline. This is because the pre-trained models used are much more computational intensive than the baseline, with one order of magnitude more parameters. More sophisticated techniques such as distillation could however mitigate this and drive down such energy consumption in inference. On the other hand, the EW-PSDS definition in Section 3.2 may in fact be too aggressive and too penalizing regarding energy consumption. Nevertheless indeed the simple CRNN baseline still performs fairly good, it is remarkable that, for example in Table 1, the best results for PSDS-1 are obtained without any additional pretrained model.

## 6. CONCLUSION

In this paper we presented the developments introduced in this year’s DCASE Challenge Task 4 and analyzed their impact on the baseline performance. We focused on the possibility to use external datasets, in particular AudioSet with strong annotations, embeddings extracted with pre-trained models and on monitoring the energy consumption of the different systems at inference time. We have shown that using recorded (in-domain) clips from AudioSet with strong annotations together with synthetic soundscapes improves the PSDS-1 performance which focuses on accurately localizing the sound events in time. Using embeddings extracted from pre-trained SEC models (like PANNs or AST) improves the performance in terms of PSDS-2 which focuses on the accurate estimation on the sound event class (with loose constraints on the time localization). Yet, there are still some open questions regarding the possibility to efficiently exploit these external resources. Indeed, the baseline remains competitive with the systems using external data or pre-trained models while being substantially simpler. This aspect also reflect in terms of energy consumption of the different systems. In particular, the EW-PSDS of the models with pre-trained models is consistently lower than that of the baseline. Finally, the way monitoring energy consumption was introduced here remains naive and should be consolidated but the preliminary results open the way to considering SED systems under another angle.
7. REFERENCES

[1] G. Ciaburro and G. Iannace, “Improving smart cities safety using sound events detection based on deep neural network algorithms,” in *Informatics*, 2020.

[2] P. Zinemanas, P. Cancela, and M. Rocamora, “Mavd: A dataset for sound event detection in urban environments.” *DCASE Workshop*, 2019.

[3] R. Serizel, N. Turpault, H. Eghbal-Zadeh, and A. P. Shah, “Large-scale weakly labeled semi-supervised sound event detection in domestic environments,” *arXiv preprint arXiv:1807.10501*, 2018.

[4] J. Ebbers and R. Haeb-Umbach, “Self-trained audio tagging and sound event detection in domestic environments,” in *DCASE Workshop*, 2021.

[5] R. M. Alsina-Pagès, J. Navarro, F. Alías, and M. Hervás, “homesound: Real-time audio event detection based on high performance computing for behaviour and surveillance remote monitoring,” *Sensors*, 2017.

[6] V. Lostanlen, J. Salamon, and A. e. a. Farnsworth, “Robust sound event detection in bioacoustic sensor networks,” *PloS one*, 2019.

[7] A. Hüwel, K. Adiloğlu, and J.-H. Bach, “Hearing aid research data set for acoustic environment recognition,” in *Proc. of ICASSP*, 2020.

[8] A. Mesaros, T. Heittola, T. Virtanen, and M. D. Plumbley, “Sound event detection: A tutorial,” *IEEE Signal Processing Magazine*, 2021.

[9] A. Tarvainen and H. Valpola, “Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results,” in *Advances in neural information processing systems*, 2017.

[10] Q. Kong, Y. Cao, T. Iqbal, and Y. e. a. Wang, “Panns: Large-scale pretrained audio neural networks for audio pattern recognition,” *IEEE TASLP*, 2020.

[11] Y. Gong, Y.-A. Chung, and J. Glass, “Ast: Audio spectrogram transformer,” *arXiv preprint arXiv:2104.01778*, 2021.

[12] J. F. Gemmeke, D. P. W. Ellis, and D. e. a. Freedman, “Audio set: An ontology and human-labeled dataset for audio events,” in *Proc. of ICASSP*, 2017.

[13] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” *Advances in Neural Information Processing Systems*, 2020.

[14] S. Hershey, D. P. Ellis, and E. e. a. Fonseca, “The benefit of temporally-strong labels in audio event classification,” in *Proc. of ICASSP*, 2021.

[15] V. Schmidt, K. Goyal, and A. J. et al., “CodeCarbon: Estimate and Track Carbon Emissions from Machine Learning Computing,” 2021.

[16] F. Ronchini, S. Cornell, and N. e. a. Turpault, “DCASE 2021 Task 4 Challenge,” https://dcase.community/challenge2021, 2021.

[17] S. Wisdom, E. Tzinis, and H. e. a. Erdogan, “Unsupervised sound separation using mixture invariant training,” *Advances in Neural Information Processing Systems*, 2020.

[18] B. Thomee, D. A. Shamma, and G. e. a. Friedland, “Yfcc100m: The new data in multimedia research,” *Communications of the ACM*, 2016.

[19] S. Hershey, S. Chaudhuri, and D. P. e. a. Ellis, “Cnn architectures for large-scale audio classification,” in *Proc. of ICASSP*, 2017.

[20] D. Snyder, G. Chen, and D. Povey, “Musan: A music, speech, and noise corpus,” *arXiv preprint arXiv:1510.08484*, 2015.

[21] N. Turpault, R. Serizel, A. Parag Shah, and J. Salamon, “Sound event detection in domestic environments with weakly labeled data and soundscape synthesis,” in *DCASE Workshop*, 2019.

[22] R. Serizel, N. Turpault, A. Shah, and J. Salamon, “Sound event detection in synthetic domestic environments,” in *Proc. of ICASSP*, 2020.

[23] F. Ronchini and R. Serizel, “A benchmark of state-of-the-art sound event detection systems evaluated on synthetic soundscapes,” in *Proc. of ICASSP*, 2022.

[24] F. Ronchini, R. Serizel, N. Turpault, and S. Cornell, “The impact of non-target events in synthetic soundscapes for sound event detection,” *arXiv preprint arXiv:2109.14061*, 2021.

[25] P. Henderson, J. Hu, and J. a. a. Romoff, “Towards the systematic reporting of the energy and carbon footprints of machine learning,” *Journal of Machine Learning Research*, 2020.

[26] T. Parcollet and M. Ravanelli, “The energy and carbon footprint of training end-to-end speech recognizers,” 2021.

[27] L. JiaKai, “Mean teacher convolution system for dcase 2018 task 4,” DCASE2018 Challenge, Tech. Rep., 2018.

[28] H. Zhang, M. Cisse, and Y. N. e. a. Dauphin, “mixup: Beyond empirical risk minimization,” *arXiv preprint arXiv:1710.09412*, 2017.

[29] Ç. Bilen, G. Ferroni, and F. e. a. Tuveri, “A framework for the robust evaluation of sound event detection,” in *Proc. of ICASSP*, 2020.

[30] R. Serizel, N. Turpault, A. Shah, and J. Salamon, “Sound event detection in synthetic domestic environments,” in *Proc. of ICASSP*, 2020.

[31] N. Turpault, R. Serizel, A. Parag Shah, and J. Salamon, “Sound event detection in domestic environments with weakly labeled data and soundscape synthesis,” in *Detection and Classification of Acoustic Scenes and Events, Workshop, DCASE*, 2019.

[32] J. Salamon, D. MacConnell, M. Cartwright, P. Li, and J. P. Bello, “Scaper: A library for soundscape synthesis and augmentation,” in *Proc. of WASPAA*, 2017.

[33] E. Fonseca, X. Favory, J. Pons, and e. a. Font, “Fsd50k: an open dataset of human-labeled sound events,” *arXiv preprint arXiv:2010.00475*, 2020.

[34] G. Dekkers, S. Lauwereins, and T. et al., “The sins database for detection of daily activities in a home environment using an acoustic sensor network,” in *DCASE Workshop*, 2017.

[35] A. Mesaros, T. Heittola, and T. Virtanen, “Tut database for acoustic scene classification and sound event detection,” in *EUSIPCO*, 2016.