# Proceedings of the 7th Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE 2022)

Mathieu Lagrange, Annamaria Mesaros, Thomas Pellegrini, Gael Richard, Romain Serizel, Dan Stowell

► To cite this version:

Mathieu Lagrange, Annamaria Mesaros, Thomas Pellegrini, Gael Richard, Romain Serizel, et al. Proceedings of the 7th Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE 2022). Tampere University, pp.1-225, 2022, 978-952-03-2677-7. hal-03850901

HAL Id: hal-03850901

https://inria.hal.science/hal-03850901

Submitted on 14 Nov 2022

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Proceedings of the 7th Workshop on Detection and
Classification of Acoustic Scenes and Events
(DCASE 2022)

Mathieu Lagrange, Annamaria Mesaros, Thomas Pellegrini, Gaël Richard, Romain Serizel and Dan Stowell (eds.)

November 3-4 2022

HITACHI
Inspire the Next

BOSE

<figure><img src="image_2.png" /></figure>

<figure><img src="image_3.png" /></figure>

MITSUBISHI
ELECTRIC

Changes for the Better

<figure><img src="image_4.png" /></figure>

sound
intelligence

amazon

<figure><img src="image_5.png" /></figure>

science

<figure><img src="image_6.png" /></figure>

Loria

Laboratoire lanet de recherche
et d'information et des applications

<figure><img src="image_7.png" /></figure>

<figure><img src="image_8.png" /></figure>

cochl.

<figure><img src="image_9.png" /></figure>

Audio
Analytic

<figure><img src="image_10.png" /></figure>
This work is licensed under a Creative Commons Attribution 4.0
International License. To view a copy of this license, visit:
http://creativecommons.org/licenses/by/4.0/

Citation:

Mathieu Lagrange, Annamaria Mesaros, Thomas Pellegrini, Gaël Richard, Romain Serizel and Dan Stowell (eds.), Proceedings of the 7th Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE 2022), Nov. 2022.

ISBN: 978-952-03-2677-7
# Table of Content

<table><tr><td>Integrating Isolated Examples With Weakly-Supervised Sound Event Detection: A Direct Approach<br>Abdollahi, Mohammad; Serizel, Romain; Rakotomamonjy, Alain; Gasso, Gilles</td><td>1</td></tr><tr><td>Impact Of Temporal Resolution On Convolutional Recurrent Networks For Audio Tagging And Sound Event Detection<br>Boes, Wim; Van Hamme, Hugo</td><td>6</td></tr><tr><td>Confidence Regularized Entropy For Polyphonic Sound Event Detection<br>Choi, Won-Gook; Chang, Joon-Hyuk</td><td>11</td></tr><tr><td>EDANSA-2019: The Ecoacoustic Dataset From Arctic North Slope Alaska<br>Çoban, Enis Berk; Perra, Megan; Pir, Dara; Mandel, Michael I</td><td>16</td></tr><tr><td>Ensemble Of Multiple Anomalous Sound Detectors<br>Deng, Yufeng; Jiang, Anbai; Duan, Yuchen; Ma, Jitao; Chen, Xuchu; Liu, Jia; Fan, Pingyi; Lu, Cheng; Zhang, Wei-Qiang</td><td>21</td></tr><tr><td>Description And Discussion On DCASE 2022 Challenge Task 2: Unsupervised Anomalous Sound Detection For Machine Condition Monitoring Applying Domain Generalization Techniques<br>Dohi, Kota; Imoto, Keisuke; Harada, Noboru; Niizumi, Daisuke; Koizumi, Yuma; Nishida, Tomoya; Purohit, Harsh; Endo, Takashi; Yamamoto, Masaaki; Kawaguchi, Yohei</td><td>26</td></tr><tr><td>MIMII DG: Sound Dataset For Malfunctioning Industrial Machine Investigation And Inspection For Domain Generalization Task<br>Dohi, Kota; Nishida, Tomoya; Purohit, Harsh; Tanabe, Ryo; Endo, Takashi; Yamamoto, Masaaki; Nikaido, Yuki; Kawaguchi, Yohei</td><td>31</td></tr><tr><td>Convolutional Neural Network For Audibility Assessment Of Acoustic Alarms<br>Effa, François; Serizel, Romain; Arz, Jean-Pierre; Grimault, Nicolas</td><td>36</td></tr><tr><td>Detection And Identification Of Beehive Piping Audio Signals<br>Fourer, Dominique; Orlowska, Agnieszka</td><td>41</td></tr><tr><td>Sound Event Localization And Detection For Real Spatial Sound Scenes: Event-Independent Network And Data Augmentation Chains<br>Hu, Jinbo; Cao, Yin; Wu, Ming; Kong, Qiuqiang; Yang, Feiran; Plumbley, Mark D.; Yang, Jun</td><td>46</td></tr><tr><td>CoLoC: Conditioned Localizer And Classifier For Sound Event Localization And Detection<br>Kapka, Sławomir; Tkaczuk, Jakub</td><td>51</td></tr><tr><td>Model Training That Prioritizes Rare Overlapped Labels For Polyphonic Sound Event Detection<br>Koga, Rie; Takamuku, Sawa; Imoto, Keisuke; Natori, Naotake</td><td>56</td></tr><tr><td>Analyzing The Effect Of Equal-Angle Spatial Discretization On Sound Event Localization And Detection<br>Kushwaha, Saksham Singh; Roman, Iran R; Bello, Juan P</td><td>61</td></tr></table>
<table>
  <tbody>
    <tr>
      <td><b>Few-Shot Bioacoustic Event Detection At The DCASE 2022 Challenge</b><br>Nolasco, Ines; Stowell, Dan; Lostanlen, Vincent; Singh, Shubhr; Morfi, Veronica; Strandburg-Peshkin, Ari; Gill, Lisa; Grout, Emily; Vidana-Villa, Ester; Morford, Joe; Emmerson, Michael; Jensen, Frants; Whitehead, Helen; Pamula, Hanna; Kiskin, Ivan</td>
      <td>136</td>
    </tr>
    <tr>
      <td><b>Exploring Eco-Acoustic Data With K-Determinantal Point Processes</b><br>Outidrarine, Mohamed; Baudet, Pierre; Lostanlen, Vincent; Lagrange, Mathieu; Ulloa, Juan Sebastián</td>
      <td>141</td>
    </tr>
    <tr>
      <td><b>Polyphonic Sound Event Detection For Highly Dense Birdsong Scenes</b><br>Garcia Arroba Parrilla, Alberto; Stowell, Dan</td>
      <td>146</td>
    </tr>
    <tr>
      <td><b>Language-Based Audio Retrieval With Textual Embeddings Of Tag Names</b><br>Pellegrini, Thomas</td>
      <td>151</td>
    </tr>
    <tr>
      <td><b>Latent And Adversarial Data Augmentations For Sound Event Detection And Classification</b><br>Perera, David; Essid, Slim; Richard, Gaël</td>
      <td>156</td>
    </tr>
    <tr>
      <td><b>STARSS22: A Dataset Of Spatial Recordings Of Real Scenes With Spatiotemporal Annotations Of Sound Events</b><br>Politis, Archontis; Shimada, Kazuki; Sudarsanam, Parthasarathy Ariyakulam; Adavanne, Sharath; Krause, Daniel A.; Koyama, Yuichiro; Takahashi, Naoya; Takahashi, Shusuke; Mitsufuji, Yuki; Virtanen, Tuomas</td>
      <td>161</td>
    </tr>
    <tr>
      <td><b>Improving Natural-Language-Based Audio Retrieval With Transfer Learning And Audio & Text Augmentations</b><br>Primus, Paul; Widmer, Gerhard</td>
      <td>166</td>
    </tr>
    <tr>
      <td><b>Description And Analysis Of Novelties Introduced In DCASE Task 4 2022 On The Baseline Systems</b><br>Ronchini, Francesca; Cornell, Samuele; Serizel, Romain; Turpault, Nicolas; Fonseca, Eduardo; Ellis, Dan</td>
      <td>171</td>
    </tr>
    <tr>
      <td><b>Sound Event Localization And Detection With Pre-Trained Audio Spectrogram Transformer And Multichannel Separation Network</b><br>Scheibler, Robin; Komatsu, Tatsuya; Fujita, Yusuke; Hentschel, Michael</td>
      <td>176</td>
    </tr>
    <tr>
      <td><b>Knowledge Distillation From Transformers For Low-Complexity Acoustic Scene Classification</b><br>Schmid, Florian; Masoudian, Shahed; Koutini, Khaled; Widmer, Gerhard</td>
      <td>181</td>
    </tr>
    <tr>
      <td><b>Feature Selection Using Alternating Direction Method Of Multiplier For Low-Complexity Acoustic Scene Classification</b><br>Schmidt, Lorenz; Peters, Nils G</td>
      <td>186</td>
    </tr>
    <tr>
      <td><b>Low-Complexity Cnnns For Acoustic Scene Classification</b><br>Singh, Arshdeep; Plumbley, Mark D</td>
      <td>191</td>
    </tr>
    <tr>
      <td><b>Improved Domain Generalization Via Disentangled Multi-Task Learning In Unsupervised Anomalous Sound Detection</b><br>Venkatesh, Satvik; Wichern, Gordon; Subramanian, Aswin Shanmugam; Leroux, Jonathan</td>
      <td>196</td>
    </tr>
    <tr>
      <td><b>Detect What You Want: Target Sound Detection</b><br>Wang, Helin; Yang, Dongchao; Zou, Yuexian; Cui, Fan; Wang, Yujun</td>
      <td>201</td>
    </tr>
  </tbody>
</table>
Matching Text And Audio Embeddings: Exploring Transfer-Learning Strategies For Language-Based
Audio Retrieval

Weck, Benno; Perez Fernandez, Miguel; Kirchhoff, Holger; Serra, Xavier....................................................206

Continual Learning For On-Device Environmental Sound Classification

Xiao, Yang; Liu, Xubo; King, James; Singh, Arshdeep; Chng, Eng Siong; Plumbley, Mark D.; Wang, Wenwu 211

Language-Based Audio Retrieval Task In DCASE 2022 Challenge

Xie, Huang; Lipping, Samuel; Virtanen, Tuomas................................................................................................216

A Mixed Supervised Learning Framework For Target Sound Detection

Yang, Dongchao; Wang, Helin; Wang, Wenwu ; Zou, Yuexian........................................................................221
# HITACHI
Inspire the Next

BOSE

amazon | science

Google

<figure><img src="image_11.png" /></figure>

MITSUBISHI
ELECTRIC

Changes for the Better

<figure><img src="image_12.png" /></figure>

sound
intelligence

<figure><img src="image_13.png" /></figure>

Audio
Analytic

<figure><img src="image_14.png" /></figure>

cochl.

<figure><img src="image_15.png" /></figure>

<figure><img src="image_16.png" /></figure>

Loria

Laboratoire lorrain de recherche
en informatique et ses applications
INTEGRATING ISOLATED EXAMPLES WITH WEAKLY-SUPERVISED SOUND
EVENT DETECTION: A DIRECT APPROACH

Mohammad Abdollahi, Romain Serizel, Alain Rakotomamonjy, Gilles Gasso

Université de Rouen

ABSTRACT

In an attempt to mitigate the need for high quality
strong annotations for Sound Event Detection (SED),
an approach has been to resort to a mix of weakly-
labelled, unlabelled and a small set of representative
(isolated) examples. The common approach to integrate
the set of representative examples into the training pro-
cess is to use them for creating synthetic soundscapes.
The process of synthesizing soundscapes however could
come with its own artefacts and mismatch to real record-
ings and harm the overall performance. Alternatively, a
rather direct way would be to use the isolated examples
in a form of template matching. To this end in this paper
we propose to train an isolated event classifier using the
representative examples. By sliding the classifier across a
recording, we use its output as an auxiliary feature vector
concatenated with intermediate spectro-temporal repre-
sentations extracted by the SED system. Experimental
results on DESED dataset demonstrate improvements in
segmentation performance when using auxiliary features
and comparable results to the baseline when using them
without synthetic soundscapes. Furthermore we show
that this auxiliary feature vector block could act as a
gateway to integrate external annotated datasets in or-
der to further boost SED system's performance.

***Index Terms***— sound event detection, deep learning, posteriorgrams, weakly-supervised learning

# 1. INTRODUCTION

Sound Event Detection (SED) is a machine listening task
that addresses the questions of *What* and *When* of oc-
curring audio events, the responses to which would be
the perceptual class of an event and its location in time
respectively. The necessity of such SED systems mani-
fests itself in applications such as audio information re-
trieval [1], surveillance [2], bioacoustic monitoring [3] and
self-driving cars [4] to a name a few. One main challenge
with SED is the inherent difficulty and cost of acquiring
annotated data of high quality/resolution. Hence there

has been a growing body of research in development of
models using lower quality annotations (lower tempo-
ral resolution, noisy labels, etc.) and un-annotated data
examples potentially augmented with a small set of (iso-
lated) representative examples of each target event class.

Isolated examples are generally used indirectly through
synthesizing soundscapes [5], to generate *strongly-labelled*
examples. The relative contribution of each level of an-
notations in this heterogeneous dataset has been studied
by Turpault et al [6]. Surprisingly, their findings in-
dicate the relative equal contribution of both sets of
weak and strongly (synthetic) labelled examples to sys-
tem's segmentation performance. In pre-deep learning
speech recognition, *Phoneme posteriorgrams* output by
pre-trained phoneme classifiers were used as features
fed to the downstream blocks[7]. In spoken keyword
detection, Chen et al. [8] trained a deep neural network
(DNN) to classify among a number of keywords and
by sliding the DNN across an audio recording used the
output scores (followed by post-processing) to mark the
keyword boundaries. Similarly, isolated examples could
be used for *template matching* by sliding them across
recordings to generate similarity scores and using these
scores for detection of event boundaries.

In this paper we propose to integrate a similar ap-
proach into training the SED system. Rather than us-
ing the similarity scores directly for decision making, we
will use them as auxiliary features to enhance the perfor-
mance of the Weakly-Supervised SED (WSSED) system.
Beyond the isolated examples, we also investigate the
utility of using classifiers trained on other (non-target)
external datasets (e.g. ESC-50) to generate the auxiliary
features and report performance gains.

## 2. BASELINE SED SYSTEM

In this work, we have chosen a CNN-Transformer ar-
chitecture proposed by Myazaki et al. [9] as our baseline
model. Similar to a Convolutional Recurrent Neural Net-
work (CRNN), CNN-Transformer uses a Convolutional
Neural Network (CNN) for extraction of intermediate
representations, but instead of a RNN, it is followed by

Thanks to ANR agency for funding the project LEAUDS.
few layers of transformer encoder for temporal dependency modelling.

Given a time-frequency representation of an audio signal as input (e.g. Mel-spectrogram) $X_{[F^i \times T^i]}$ where $F^i$ and $T^i$ represent the dimensions of the input along frequency and time respectively, a CNN extracts a representation $H_{[C \times F^e \times T^e]}$ with $F^e$ and $T^e$ being the resulting (downsampled) frequency and time dimensions and $C$ the number of extracted feature maps. Merging the $F^e$ and $C$ dimensions, would result a $T^e$-long sequence of $d = C \times F^e$ dimensional feature vectors $h_i$:

$$H = \{h_1, \dots, h_{T^e}\} \qquad (1)$$

Every element of this sequence is first linearly projected to a lower dimensional space using an *Embedding layer*. The resulting sequence of embedding vectors, after being added to the respective positional encoding vectors are passed through N layers of transformer encoders.

A final single-layer Multi-layer Perceptron (MLP) is applied on all elements of the output sequence of the transformer to predict the segment-based labels. The clip-level label is obtained by performing attention-pooling [10] over all elements in the output sequence.

## 3. PROPOSED APPROACH

In this work we take a more direct approach in integrating isolated examples into training a WSSED system. Rather than synthesizing soundscapes with them, we propose to use them to build an isolated event classifier. The idea is that, given a large enough set of isolated examples of the target events, we could build a classifier operating on a certain time scale and slide it across test audio examples in order to generate similarity/likelihood scores. These scores could then be used (in addition to some post-processing) to detect target event boundaries. Such an approach would be very similar to the recent line of work on Keyword Spotting (KWS) [8, 11]. Unlike in the case of KWS however, our set of examples is much smaller and usually suffers from more intra-class acoustic variability. In this paper we propose to integrate a similar approach into training the SED system by using the output of an isolated event classifier as an additional feature. We leverage transfer learning to train the classifier in order to mitigate the effects of low sample-size training dataset. In fact, we use the same pre-trained backbone CNN of the CNN-transformer baseline, and train a single-layer MLP on top of it.

In our proposed system, each segment of the temporal sequence $H$ at the CNN's output is passed to the MLP and the classifier's output scores are appended to the spectro-temporal features of that segment. The overall schematic of the proposed system is illustrated in fig-

ure 1 in which we have introduced an *Auxiliary Feature generator* block.

### 3.1. Event Classifier

Since the goal is to use the classifier for detection and to produce an output probability vector at a specified temporal resolution, we train a point-wise MLP operating on each $h_t$ independently. This means we broadcast the event label to all of it sub-segments across time. Nevertheless, in order to further expand the effective receptive field input to the MLP when classifying $h_t$, we concatenate a set of $2k + 1$ dilated neighbouring feature slices centred around each $h_t$ to be passed to the MLP:

$$o_t = MLP(\begin{bmatrix} h_{t-(k.d)} \\ \vdots \\ h_t \\ \vdots \\ h_{t+(k.d)} \end{bmatrix}) \qquad (2)$$

with $o_t$ being the classifier's output at time instance $t$ and $d$ the dilation factor. Similar to the SED system's output, in order for the classifier to account for event polyphony and non-target sound events, we train the classifier with a *Binary Cross-Entropy (BCE)* loss on sigmoid-activated outputs over the 10 classes of target events. We refer to the output vector in this case as a *posteriorgram*.

### 3.2. Auxiliary Feature Generator Block

The above trained MLP classifier is integrated into the SED system in the form of an *Auxiliary Feature Generator* (AFG) block where the trained MLP is moved across the extracted representations $h_t$ of a recording and generates a posteriorgram $o_t$ for each corresponding $h_t$. We further take the logarithm of these probability scores before using them as features in the rest of the system.

Additionally, in order to emphasize the points in time where the classifiers output experience sharp changes (potential event boundaries) we propose to augment each log-probability vector with its first order dynamics $d_t$:

$$d_t = \sum_{n=1}^{N} n \times (\log(o_{t+n}) - \log(o_{t-n})) \qquad (3)$$

where $N$ is the width of the window used for the calculation of the derivative. The resulting output feature of the AFG block is the concatenation of each $\log(o_t)$ and $d_t$ yielding the auxiliary feature sequence $P$:

$$P = \{p_1, \dots, p_{T^e}\} = \{\begin{bmatrix} \log(o_1) \\ d_1 \end{bmatrix}, \dots, \begin{bmatrix} \log(o_{T^e}) \\ d_{T^e} \end{bmatrix}\} \qquad (4)$$
<figure><img src="image_17.png" /><figcaption>Fig. 1: General schematic of the proposed approach with the AFG block</figcaption></figure>

As illustrated in figure 1, the AFG output features $p_t$ are concatenated with the $h_t$ and the sequence of tandem feature slices are batch-normalized before being passed to the embedding layer and the succeeding transformer encoder layers which will be trained using the weakly-labelled (and synthetic) examples.

## 3.3. Training Procedure

In this work, while training the SED system, we keep the parameters of the classifier (MLP and the backbone CNN) frozen and use the rest of weakly- and strongly-annotated examples only to train the embedding layer and the transformer layers. Given that the CNN is already pre-trained using a large dataset, this should not be a limiting factor on the system's learning capacity. Nevertheless training the entire system end-to-end, which given two tasks and two datasets would be framed as a multitask learning problem, seems more appealing and we intend to address that in our future works.

# 4. EXPERIMENTS

## 4.1. Model Parameters

In this work rather than using a shallow CNN as in the original baseline CRNN model of DCASE task 4, we use a deeper Efficientnet-B2 model [12] used in a work by by Gong et al. [13] pre-trained on Audioset [14].

Each 10-second audio clip is resampled at the rate of 16kHz and is converted to log Mel-spectrogram with 25ms window length, 10ms window hop and 128 Mel bands. The resulting 2D array is passed through the backbone CNN of the system.

The Efficientnet-B2 final convolutional layer yields a feature map of 1408 × 4 × 33 dimension as its output. After average pooling along the frequency axis we end up with a temporal sequence of 33 1408-dimensional feature vectors. This sequence is passed through an embedding layer to produce a sequence of 256-dimensional

embedding vectors, which in turn is passed through the transformer encoder layers. We use 3 layers of transformer encoder blocks, with 512 as the hidden dimension of the positional feed-forward network and 4 attention heads. In the proposed approach the features in this sequence are concatenated with the AFG's output features, and passed to the embedding layer to be projected to a sequence of the 256-dimensional vectors similar to the baseline model. The same pre-trained CNN is used for training the MLP in the AFG block. We chose width $k = 1$ and dilation $d = 2$ to define the temporal extent of information resulting in a 4224-dimensional input to the MLP. The MLP consists of a 512 unit hidden layer with batch-normalization followed by ReLU activation function. With the 10 output classes of the isolated classifier, the auxiliary features will be 20-dimensional vectors. Since the number and duration of examples per event class are highly variable in the set of isolated examples we adopt a weighted sampling strategy in each batch to train the classifier. For training both the SED system and the classifier we use data augmentation pipeline consisting of mixup [15], frequency and time masking [16] and random additive gaussian noise.

## 4.2. Datasets

### 4.2.1. SED Datasets

The datasets that were provided as part of DCASE 2021 task4 challenge (DESED) are listed below.

**Weakly-annotated data:** 1578 10-second long audio clips of domestic recordings taken from Audioset [14] with verified clip-level labels.

**Unlabelled Data:** 14412 clips of 10-second clips selected from Audioset. We did not use the unlabelled data in our experiments.

**Soundbank of isolated events:** 1009 isolated examples of target events taken from FSD50k [17] with clip durations ranging from 55ms to 83.1s. The median
<table><caption>**Table 1:** A comparative performance evaluation of proposed AFG block on WSSED system</caption><thead><tr><th></th><th>w. Synth</th><th>F1</th><th>PSDS1</th><th>PSDS2</th></tr></thead><tbody><tr><td rowspan="2">Baseline</td><td><b>X</b></td><td>23.21</td><td>0.097</td><td>0.336</td></tr><tr><td><b>✓</b></td><td>38.92</td><td>0.251</td><td>0.512</td></tr><tr><td rowspan="2">Ours</td><td><b>X</b></td><td>35.25</td><td>0.183</td><td>0.491</td></tr><tr><td><b>✓</b></td><td>40.20</td><td>0.263</td><td>0.523</td></tr></tbody></table>

number of examples per class is 62 and the median total duration of examples per class is 310s.

**Synthetic Soundscapes:** a collection of 10000 synthetically generated soundscapes using the Scaper [18] library using the soundbank of isolated events and a selected set of background sounds from SINS dataset [19].

### 4.2.2. External Datasets

We also used an external dataset for auxiliary feature generation in our experiments.

**ESC-50:** a collection of 2000 event clips all of 5s durations across 50 sound event classes each containing 40 examples [20].

## 4.3. Results

Table 1 summarizes the performance of our proposed approach against the baseline CNN-Transformer system. The results are reported on the provided validation set. The performance is reported using intersection-based F1 macro score and PSDS measures. PSDS1 and PSDS2 are two specific hyperparameter settings of the PSDS measure where the former emphasizes more accurate localization while the focus of latter is on distinguishing classes from one another. In order to evaluate the potential of AFG block to complement or even replace the use of synthetic soundscapes, we report the performance of the baseline system when trained only with weakly-labelled examples alone (w/o synth) and when including the synthetic soundscapes (w. synth).

## 4.4. AFG, a Gateway for External Datasets

Given the generality of the AFG block, we further investigate the utility of it as a gateway to integrate external annotated datasets. The idea is that nothing restricts the MLP to be trained on the provided soundbank of isolated examples. Even if the target classes of an external dataset does not overlap with the SED task at hand, projecting slices of a recording into a classifier trained on a externally-annotated dataset could provide relevant discriminative features and hence reduce the uncertainty inherent in the weakly-labelled examples.

<table><caption>**Table 2:** Effect of using external datasets for feature generation on WSSED system's performance</caption><thead><tr><th></th><th>w. Synth</th><th>F1</th><th>PSDS1</th><th>PSDS2</th></tr></thead><tbody><tr><td rowspan="2">Baseline</td><td><b>X</b></td><td>23.21</td><td>0.097</td><td>0.336</td></tr><tr><td><b>✓</b></td><td>38.92</td><td>0.251</td><td>0.512</td></tr><tr><td rowspan="2">AFG: ESC-50</td><td><b>X</b></td><td>28.33</td><td>0.131</td><td>0.451</td></tr><tr><td><b>✓</b></td><td>39.56</td><td>0.227</td><td>0.536</td></tr><tr><td rowspan="2">AFG: ESC-50 + isolated examples</td><td><b>X</b></td><td>38.22</td><td>0.202</td><td>0.553</td></tr><tr><td><b>✓</b></td><td>41.31</td><td>0.251</td><td>0.574</td></tr></tbody></table>

Table 2 lists the results of using the ESC-50 dataset to generate auxiliary features and results are reported for both when used alone and when combined with features from isolated examples. When combined, we concatenate the auxiliary features from both trained classifiers before projecting them into the transformer's embedding dimension. The results are shown for both with and without the use of synthetic dataset in training the SED system. Our results suggest this approach as a rather simple way to distil discriminative information inherent in external datasets into WSSED with minimal effort and without requiring any further annotations with respect to the target task.

# 5. CONCLUSIONS

In this work we have presented a different way of incorporating a small set of isolated examples into training a WSSED system. Rather than using them to create synthetic soundscapes, we proposed to use the output of a classifier built using them as an auxiliary feature for each sub-segment of an input recording. We have evaluated our approach using DCASE2021 task-4 setup and dataset. Our results suggest that by means of introducing this feature augmentation we could improve the performance of the baseline system with or without synthetic soundscapes. In addition we showed that the auxiliary features could be generated using external datasets to encode extra knowledge into the system and improve the performance.

In our future work we intend to explore other ways of auxiliary feature generation particularly memory-based methods using the advances in deep metric learning and few-shot learning. Moreover, as the auxiliary feature block and the main SED system share the backbone network, the system could be end-to-end optimized through framing it as a multi-task learning problem.

# 6. REFERENCES

[1] E. Wold, T. Blum, D. Keislar, and J. Wheaten, "Content-based classification, search, and retrieval
of audio,” *IEEE MultiMedia*, vol. 3, no. 3, pp. 27–36, 1996.

[2] R. Radhakrishnan, A. Divakaran, and A. Smaragdis, “Audio analysis for surveillance applications,” in *IEEE Workshop on Applications of Signal Processing to Audio and Acoustics, 2005.*, 2005, pp. 158–161.

[3] Justin Salamon, Juan Pablo Bello, Andrew Farnsworth, Matt Robbins, Sara Keen, Holger Klinck, and Steve Kelling, “Towards the automatic classification of avian flight calls for bioacoustic monitoring,” *PLoS One*, vol. 11, no. 11, Nov. 2016.

[4] Marco Cristani, Manuele Bicego, and Vittorio Murino, “Audio-visual event recognition in surveillance video sequences,” *IEEE Transactions on Multimedia*, vol. 9, no. 2, pp. 257–267, 2007.

[5] Romain Serizel, Nicolas Turpault, Ankit Shah, and Justin Salamon, “Sound event detection in synthetic domestic environments,” in *Proc. ICASSP*, 2020, pp. 86–90.

[6] Nicolas Turpault and Romain Serizel, “Training sound event detection on a heterogeneous dataset,” in *DCASE Workshop*, 2020.

[7] Gethin Williams and Daniel P. W. Ellis, “Speech/music discrimination based on posterior probability features,” in *EUROSPEECH*, 1999.

[8] Guoguo Chen, Carolina Parada, and Georg Heigold, “Small-footprint keyword spotting using deep neural networks,” in *Proc. ICASSP*, 2014, pp. 4087–4091.

[9] Koichi Miyazaki, Tatsuya Komatsu, Tomoki Hayashi, Shinji Watanabe, Tomoki Toda, and Kazuya Takeda, “Weakly-supervised sound event detection with self-attention,” in *Proc. ICASSP*, 2020, pp. 66–70.

[10] Yun Wang, Juncheng Li, and Florian Metze, “A comparison of five multiple instance learning pooling functions for sound event detection with weak labeling,” in *Proc. ICASSP*, 2019, pp. 31–35.

[11] Tara N. Sainath and Carolina Parada, “Convolutional neural networks for small-footprint keyword spotting,” in *Proc. Interspeech 2015*, 2015, pp. 1478–1482.

[12] Mingxing Tan and Quoc Le, “Efficientnet: Rethinking model scaling for convolutional neural networks,” in *International conference on machine learning*. PMLR, 2019, pp. 6105–6114.

[13] Yuan Gong, Yu-An Chung, and James Glass, “PSLA: Improving audio tagging with pretraining, sampling, labeling, and aggregation,” *IEEE/ACM Transactions on Audio, Speech, and Language Processing*, vol. 29, pp. 3292–3306, 2021.

[14] Jort F. Gemmeke, Daniel P. W. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R. Channing Moore, Manoj Plakal, and Marvin Ritter, “Audio set: An ontology and human-labeled dataset for audio events,” in *Proc. IEEE ICASSP 2017*, New Orleans, LA, 2017.

[15] Yuji Tokozume, Yoshitaka Ushiku, and Tatsuya Harada, “Learning from between-class examples for deep sound recognition,” in *International Conference on Learning Representations*, 2018.

[16] Daniel S. Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D. Cubuk, and Quoc V. Le, “SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition,” in *Proc. Interspeech 2019*, 2019, pp. 2613–2617.

[17] Eduardo Fonseca, Xavier Favory, Jordi Pons, Frederic Font, and Xavier Serra, “Fsd50k: An open dataset of human-labeled sound events,” *IEEE/ACM Trans. Audio, Speech and Lang. Proc.*, vol. 30, pp. 829–852, jan 2022.

[18] Justin Salamon, Duncan MacConnell, Mark Cartwright, Peter Li, and Juan Pablo Bello, “Scaper: A library for soundscape synthesis and augmentation,” in *IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WAS-PAA)*, 2017, pp. 344–348.

[19] Gert Dekkers, Steven Lauwereins, Bart Thoen, Mulu Weldegebreal Adhana, Henk Brouckxon, Bertold Van den Bergh, Toon van Waterschoot, Bart Vanrumste, Marian Verhelst, and Peter Karsmakers, “The SINS database for detection of daily activities in a home environment using an acoustic sensor network,” 2017, pp. 1–5, DCASE Workshop.

[20] Karol J. Piczak, “ESC: Dataset for Environmental Sound Classification,” in *Proceedings of the 23rd Annual ACM Conference on Multimedia*. pp. 1015–1018, ACM Press.
# IMPACT OF TEMPORAL RESOLUTION ON CONVOLUTIONAL RECURRENT NETWORKS FOR AUDIO TAGGING AND SOUND EVENT DETECTION

Wim Boes, Hugo Van hamme

ESAT, KU Leuven, Belgium

## ABSTRACT

Many state-of-the-art systems for audio tagging and sound event detection employ convolutional recurrent neural architectures. Typically, they are trained in a mean teacher setting to deal with the heterogeneous annotation of the available data.

In this work, we present a thorough analysis of how changing the temporal resolution of these convolutional recurrent neural networks — which can be done by simply adapting their pooling operations — impacts their performance. By using a variety of evaluation metrics, we investigate the effects of adapting this design parameter under several sound recognition scenarios involving different needs in terms of temporal localization.

**Index Terms**— sound recognition, audio tagging, sound event detection, temporal resolution

## 1. INTRODUCTION

Recently, the popularity of research into audio-related tasks has surged because of, among other reasons, multiple versions of the Detection and Classification of Acoustic Scenes and Events (DCASE) challenge [1, 2, 3, 4, 5, 6]. The latest editions encompassed six categories, each dealing with a distinct sound recognition problem. Subtask number 4 [7] consistently covered sound event detection — joint classification and temporal localization of auditory events — in domestic environments. Submitted systems were ranked utilizing measures which represent scenarios involving variable requirements with regard to the estimation of time boundaries.

The baseline [7] supplied for the fourth problem of the DCASE 2021 and 2022 challenges, a convolutional recurrent neural network trained using the mean teacher principle [8], was based on the second-ranking system [9] of task 4 of DCASE 2019. Its output temporal resolution was predefined and fixed.

Our submission [10] for the fourth subtask of the DCASE 2021 challenge [7] demonstrated that simply adapting the amount of pooling in this network, which is directly linked to its temporal resolution, can significantly impact its performance.

In this work, we provide a more thorough and complete analysis of this interesting finding. We substantially supplement the discussion in multiple ways, as outlined in the two paragraphs below.

Firstly, we examine more than the specific evaluation scenarios prescribed in task 4 of the DCASE 2021 and 2022 challenges. Particularly, we also investigate what happens when the temporal resolutions of convolutional recurrent networks are adapted in the context of audio tagging. In this situation, the goal of the models is to perform clip-level auditory event classification. Unlike for sound event detection, temporal localization is not required in this case.

Secondly, for sound event detection, we inspect what happens using multiple types of measures. More specifically, in addition to the intersection-based scores employed in subtask 4 of the DCASE 2021 and 2022 challenges, we also utilize segment-based and event-based metrics, which are commonly used in this subdomain of machine learning as well. This is further elaborated upon in Section 4.

To this end, the following approach is taken: We start with a suitable baseline, which forms the basis for many state-of-the-art architectures for both audio tagging and sound event detection, such as [11], [12] and [13]. We change the temporal resolution of this model by adapting its pooling operations. We then analyze the results obtained by this modified system in a variety of experimental configurations, representing different evaluation scenarios.

In Section 2, we expand upon the baseline system. Afterwards, in Section 3, we describe how the pooling operations of the considered architecture can be adapted to modify the temporal resolution of the model. Then, in Section 4, we elaborate upon the experimental setup. Next, in Section 5, we analyze the results of the performed experiments, and finally, we draw a conclusion in Section 6.

## 2. BASELINE

In this section, the baseline system, which is nearly the same as in [14], is elaborated upon. It is a slightly adapted version of the baseline supplied for task 4 of the DCASE 2021 and 2022 challenges [7].

### 2.1. Architecture

A schematic visualization of the baseline model is given in Figure 1.

The input to the baseline is a spectral map of an audio recording. The amount of frequency bins is predefined and set to 128.

The first component of the architecture is a convolutional neural network (CNN) made up of seven blocks. Each of those consists of the following five layers: a convolutional layer, a batch normalization layer [15], a ReLU activation layer, a dropout layer [16] with a dropout rate of 33%, and lastly, an average pooling layer.

All convolutional layers use a square kernel of size 3 and utilize a stride of 1. For the first three blocks, the number of output channels of the convolutional operations is equal to 16, 32 and 64 respectively. For the last four blocks, this number is equal to 128.

The hyperparameters of the pooling operations are given per model block in Table 1. The first and second numbers of each tuple are connected to the time and frequency axes respectively.

After the last block, the frequency-related dimension of the spectral (auditory) input map has been brought down to one and the corresponding axis can therefore be squeezed, i.e., removed.

Afterwards, a bidirectional gated recurrent (BiGRU) unit, consisting of two layers with hidden sizes equal to 128, is used to model potential temporal relationships in the auditory data.

<footnote>This work was supported by a PhD Fellowship of Research Foundation Flanders (FWO-Vlaanderen) and the Flemish Government under "Onderzoeksprogramma AI Vlaanderen".</footnote>
<figure><img src="image_18.png" /><figcaption>Figure 1: Schematic representation of the baseline</figcaption></figure>

The output of this recurrent layer is followed by a linear pro-
jection and application of the sigmoid function to obtain multi-label
frame-level probabilities. These values indicate per temporal frame
which sound categories are active, and can be postprocessed to ob-
tain event-level predictions of sounds, which are relevant for sound
event detection, as explained in the next subsection.

Finally, the frame-level sound probabilities are aggregated to
get clip-level auditory event probabilities, which are relevant for the
task of audio tagging, by performing linear (softmax) pooling [17].

## 2.2. Postprocessing

The clip- and frame-level probabilities are converted into binary
values by enforcing a fixed threshold. The frame-level decisions
are also passed through a smoothing median filter with a window
of about 500 ms. Preliminary experiments showed that more com-
plicated postprocessing (e.g., class-wise optimization on validation
data) is unnecessary as it provides relatively insignificant benefits.

The binary clip-level decisions can readily be used to perform
audio tagging. On the contrary, to produce outputs suitable for
sound event detection, an extra step has to be taken: The frame-level
decisions are converted into event-level predictions by performing
a merging operation with a maximum gap tolerance of 200 ms.

## 2.3. Mean teacher training

As expanded upon in Section 4, the training data employed in this
research project is heterogeneously annotated: Some of the samples
include clip-level or weak labels, some come with event-level or
strong labels, and the rest is unlabeled. To deal with this difficulty,
the mean teacher training principle [8] is applied.

<table><caption>Table 1: Kernel sizes and strides of pooling layers in baseline CNN</caption><thead><tr><td>Block</td><td>Kernel size = stride</td></tr></thead><tbody><tr><td>0-1</td><td>(2, 2)</td></tr><tr><td>2-3-4-5-6</td><td>(1, 2)</td></tr></tbody></table>

In the mean teacher training framework, two models called the
student and the teacher are utilized. They share the same architec-
ture, but their parameters are updated completely differently.

The student system is trained in a regular fashion: A differen-
tiable loss function is optimized by an optimization algorithm such
as Adam [18]. This is not the case for the teacher counterpart: The
weights of this model are computed as the exponential moving av-
erage of the student parameters with a multiplicative decay factor of
0.999 per training iteration, also explaining the name of the method.

The loss used to train the student consists of four terms: The
first two are clip-level and frame-level binary cross entropy func-
tions, which are only calculated for the weakly and strongly la-
beled data samples respectively. The remaining two components
are mean-squared error consistency costs between the clip-level and
frame-level output probabilities of the student and teacher models,
which can be computed for all examples, including the unlabeled
ones. The classification and consistency terms are summed with
weights equal to 1 and 2 respectively to obtain the final objective.

## 3. ADAPTATION OF TEMPORAL RESOLUTION

The goal of this project is to investigate how modifying the output
temporal resolution of the considered convolutional recurrent neu-
ral network affects its sound recognition performance under differ-
ent circumstances. In this section, we describe which parts of this
model were changed to achieve these specific adaptations.

In the baseline model outlined in Section 2, the pooling lay-
ers in the first and second blocks of the CNN halve input feature
maps along the temporal dimension, while the others do not change
anything in this regard. This results in a total pooling factor of 4.

As explained in more detail in Section 4, most of the audio
recordings used in this project have a duration of 10 seconds. They
are fed to the baseline convolutional recurrent neural network as
spectral feature maps consisting of 608 time frames. This model
applies a temporal reduction factor of 4, and hence, the frame-level
probabilities contain 152 values per sample. This comes down to a
temporal resolution of about one class prediction vector per 65 ms.

We create adaptations of the baseline convolutional recurrent
network in the following way: Instead of only allowing the first
two pooling layers to apply a reduction in the number of temporal
frames, we create model versions of which the first *x* pooling layers
perform this halving operation, with *x* ranging from 1 up to 6. This
results in systems with time reduction factors of 2, 4, 8, 16 and 32
respectively, which are equivalent to temporal resolutions of about
one frame-level prediction bin per 32.5, 65, 130, 260 and 520 ms.

Important to note is that changing the amount of pooling in the
considered convolutional recurrent neural network does not change
the number of trainable parameters, and thus, its modeling capacity.

## 4. EXPERIMENTAL SETUP

In this section, we provide full details on the setup used during the
experiments, of which the results are analyzed in Section 5.
## 4.1. Data

The data set for problem 4 of DCASE 2021 [7] is a multi-label collection consisting of audio recordings with a maximum length of 10 seconds. There are 10 possible sound event categories, related to domestic environments, which are not mutually exclusive.

The partition used to train models consists of three subsets:

* 1578 real samples with weak (clip-level) annotation

* 10000 synthetic samples with strong (event-level) annotation

* 14412 real samples without annotation

For evaluation purposes, two partitions were available: the so-called validation and public evaluation sets, holding 1168 and 692 real recordings respectively. Both subsets include strong or event-level labels of the active environmental auditory events.

## 4.2. Preprocessing

To get spectral maps of the available audio recordings, which are used as input features for the considered convolutional recurrent nets as explained in Section 2, we resampled all clips to 22050 Hz and performed peak amplitude normalization. Then, log mel spectrograms with 128 frequency bins were extracted using a Hamming window with a size of 2048 samples and a hop length of 363 samples. Lastly, per-frequency bin standardization was carried out.

As mentioned before, for a 10-second audio clip, these steps resulted in a spectral feature map consisting of 608 temporal frames.

## 4.3. Data augmentation

In order to avoid the risk of overfitting, we employed data augmentation during the training of the considered models. In particular, we used mixup [19], which comes down to creating extra learning examples (and associated labels) by linearly interpolating the original samples. We employed this method with a probability of 50% of applying it. The mixing ratios used in this algorithm were randomly sampled from a beta distribution with shape parameters set to 0.2.

Unlike for our related submission [10] for task 4 of the DCASE 2021 challenge [7], we did not employ time and frequency masking [20] in this project. This choice was based on two observations made during initial experiments: Firstly, the best hyperparameters for these techniques seemed to depend on the situation, e.g., used metric. Secondly, these methods only led to minimal improvements. As the focus of this work is on analysis rather than trying to get optimized performance, they were excluded for the sake of clarity.

## 4.4. Training and evaluation

All models were trained and evaluated using PyTorch [21].

### 4.4.1. Training

All models were trained for 200 epochs. Per epoch, 250 batches of 48 samples were given to the networks. Each batch contained 12 weakly labeled, 12 strongly labeled and 24 unlabeled examples.

Adam [18] was employed to train the weights of the student models. Learning rates were ramped up exponentially from 0 to 0.001 for the first 12500 optimization step. Thereafter, they decayed multiplicatively at a rate of 0.99995 per training iteration.

### 4.4.2. Evaluation

For audio tagging, we used the (micro-averaged) clip-based F1 measure [22], which was also utilized in task 4 of the DCASE 2017 challenge [23]. This score was computed for a single operating point, namely, the situation in which probabilities were converted into binary decisions by enforcing a 50% threshold.

For sound event detection, we employed multiple types of metrics. To start with, we utilized the intersection-based measures used for ranking in task 4 of the DCASE 2021 and 2022 challenges [7]. More specifically, we used two polyphonic sound event detection scores [24] representing distinct evaluation scenarios, denoted as PSDS 1 and 2. The former imposes strict requirements on the temporal localization accuracy, the latter is more lenient in this regard.

The hyperparameters utilized for calculating these PSDS measures are summarized in Table 2. These scores were computed using 50 operating points, in which thresholds linearly distributed from 0.01 to 0.99 were used to convert probabilities into binary decisions.

Details on the procedure for calculating PSDS scores and a discussion on the hyperparameters can be found in [24].

<table><caption>Table 2: PSDS hyperparameters</caption><thead><tr><td>Hyperparameter</td><td>PSDS 1</td><td>PSDS 2</td></tr></thead><tbody><tr><td>Detection tolerance criterion</td><td>0.7</td><td>0.1</td></tr><tr><td>Ground truth intersection criterion</td><td>0.7</td><td>0.1</td></tr><tr><td>Cross-trigger tolerance criterion</td><td>N/A</td><td>0.3</td></tr><tr><td>Cost of class instability</td><td>1</td><td>1</td></tr><tr><td>Cost of cross-triggers</td><td>0</td><td>0.5</td></tr><tr><td>Maximum false positive rate</td><td>100</td><td>100</td></tr></tbody></table>

Furthermore, we used the segment-based (micro-averaged) F1 score based on chunks of 1 s [22] to gauge the performance of the considered models. This metric was also employed in task 4 of the DCASE 2017 challenge [23]. Because of the long segment length, this measure quantifies systems in terms of their ability to perform more coarse-grained sound event detection, not unlike PSDS 2.

Lastly, we also employed the (macro-averaged) event-based F1 score with tolerances of 200 ms for onsets and 20% of the audio event lengths (up to a max of 200 ms) for offsets [22]. This measure was used in task 4 of the DCASE 2018, 2019 and 2020 challenges [7, 25]. This metric quantifies models in terms of their ability to perform fine-grained sound event detection, similar to PSDS 1.

These segment-based and event-based scores were computed for a single operating point, in which a threshold of 0.5 was enforced to convert frame-level probabilities into binary decisions.

All measures were computed using the clip-level or frame-level probabilities of the student models after the last training epoch.

## 5. EXPERIMENTAL RESULTS

In this section, we analyze the results obtained by the convolutional recurrent neural systems with varying output resolutions, as explained previously. We report the metrics for audio tagging and sound event detection elaborated upon in Section 4 after applying the following method to significantly diminish the variability of the results: For each experimental configuration, we train 20 models with independent initializations and average the scores they achieve on the public evaluation partition of the employed data set.
<figure><img src="image_19.png" /><figcaption>Figure 2: Results on public evaluation subset</figcaption></figure>

The scores achieved on the public evaluation set are summarized in Figure 2. Precise numerical values are not essential to the interpretation below and are not disclosed due to a lack of space.

Only one of the metrics appears to display a (very slight) negative correlation with the temporal output resolution of the considered models, namely, PSDS 2. Intuitively, this behavior is not unexpected as this intersection-based score was designed to gauge relatively coarse-grained sound event detection performance.

All other measures correlate positively with respect to the time resolution. For the event-based F1 score and PSDS 1, the scores demanding precise estimations of the boundaries of sounds, this seems logical: Naturally, if the length associated with an output prediction bin is too long, this becomes more challenging or even impossible.

However, this positive relationship also appears to hold for the clip-based and segment-based F1 metrics, which inherently require much less accurate temporal differentiation — in the former case, this is not even needed at all. This counterintuitive result is all the more unanticipated given the trend of PSDS 2: Apparently, there is a certain discrepancy in the reaction of the examined types of measures for audio tagging and coarse-grained sound event detection to the temporal output resolution of the considered models.

The sensitivity of the performance of convolutional recurrent neural networks to their output resolutions depends on the criterion. In particular, scores designed for more fine-grained sound event detection show a high responsiveness in this regard: PSDS 1 and the event-based F1 measure drop sharply as the temporal resolution of the models at hand decreases. On the contrary, the clip-and segment-based F1 metrics as well as PSDS 2, used to gauge performance with regard to audio tagging and coarse-grained sound event detection, remain more stable as this hyperparameter varies.

It is also possible to study how the performance and sensitivity of the models differs across sound categories by inspecting class-wise scores. For F1-based measures, these can easily be obtained. For the PSDS metrics, they can also be computed, but unlike for their aggregated counterparts, the cost of class instability (see Section 4) must be disregarded. Exact numerical results are not crucial to the following analysis and are not included due to a lack of space.

In nearly all instances, the class-wise scores correlate to the temporal resolution in the same direction (positively/negatively) as

<figure><img src="image_20.png" /><figcaption>Figure 3: Subset of class-wise event-based F1 scores</figcaption></figure>

their global versions. Nevertheless, the absolute performance and responsiveness strikingly vary across metrics and sound categories.

For all of the measures, there is a substantial link between the scores and the examined kind of sound. As an example, the best-performing category (*speech*) consistently outperforms the worst (*running water*) by a large margin. This behavior can be explained by multiple factors, e.g., qualitative differences between types of audio events and imbalance within the data used for training.

As is the case for the aggregated scores, the class-wise sensitivities are only strongly pronounced for fine-grained sound event detection measures. In these cases, they also greatly depend on the considered audio category. This is exemplified for the event-based F1 metric in Figure 3. For events which are short and localized (e.g., *dog*, *dishes*), reducing the temporal resolution leads to severe deterioration in terms of performance. For more long-lived sounds (e.g., *vacuum cleaner*), this dependency is much less outspoken. In the most extreme case of *frying*, there is even no correlation at all.

## 6. CONCLUSION

Numerous state-of-the-art sound recognition models are based on convolutional recurrent neural architectures. They are usually optimized in a mean teacher framework to deal with the heterogeneous labeling of the supplied data samples. In this work, we provided deeper insight into how changing the temporal resolution of such nets impacts their performance. We analyzed the effect of modifying pooling operations on a multitude of metrics, designed for distinct audio tagging and sound event detection scenarios.

The experiments showed that the performance of the considered models is highly susceptible to changes in terms of their pooling operations. More specifically, metrics designed for fine-grained sound event detection showed a strong, positive relation with respect to the temporal output resolutions of the systems at hand. For sound recognition measures involving less focus on temporal differentiation (among others, F1 score for audio tagging), the direction of the correlation was mixed and contingent on the scoring method.

The responsiveness appeared to depend on the considered evaluation scenario. Generally, measures devised for fine-grained sound event detection displayed a much higher reactivity than metrics built for audio tagging and coarse-grained sound event detection.

A class-wise study showed that for the latter, the sensitivity to temporal resolution was low across all sound types. Conversely, for fine-grained sound event detection scores, the reactivity was decidedly stronger for shorter than for longer-lasting events.
7. REFERENCES

[1] A. Mesaros, T. Heittola, E. Benetos, P. Foster, M. Lagrange, T. Virtanen, and M. D. Plumbley, “Detection and Classification of Acoustic Scenes and Events: Outcome of the DCASE 2016 Challenge,” *IEEE/ACM Transactions on Audio, Speech and Language Processing (TASLP)*, vol. 26, no. 2, pp. 379–393, 2018.

[2] T. Virtanen, A. Mesaros, T. Heittola, A. Diment, E. Vincent, E. Benetos, and B. M. Elizalde, *Proceedings of the Detection and Classification of Acoustic Scenes and Events 2017 Workshop (DCASE2017)*, 2017.

[3] M. D. Plumbley, C. Kroos, J. P. Bello, G. Richard, D. P. Ellis, and A. Mesaros, *Proceedings of the Detection and Classification of Acoustic Scenes and Events 2018 Workshop (DCASE2018)*, 2018.

[4] M. Mandel, J. Salamon, and D. P. Ellis, *Proceedings of the Detection and Classification of Acoustic Scenes and Events 2019 Workshop (DCASE2019)*, 2019.

[5] N. Ono, N. Harada, Y. Kawaguchi, A. Mesaros, K. Imoto, Y. Koizumi, and T. Komatsu, *Proceedings of the 5th Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE 2020)*, 2020.

[6] F. Font, A. Mesaros, D. P. Ellis, E. Fonseca, M. Fuentes, and B. Elizalde, *Proceedings of the 6th Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE 2021)*, 2021.

[7] N. Turpault, R. Serizel, A. Parag Shah, and J. Salamon, “Sound event detection in domestic environments with weakly labeled data and soundscape synthesis,” in *Workshop on Detection and Classification of Acoustic Scenes and Events*, 2019.

[8] A. Tarvainen and H. Valpola, “Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results,” in *Advances in Neural Information Processing Systems*, 2017, pp. 1195–1204.

[9] L. Delphin-Poulat and C. Plapous, “Mean teacher with data augmentation for dcase 2019 task 4,” Orange Labs Lannion, Tech. Rep., 2019.

[10] W. Boes and H. Van hamme, “Optimizing temporal resolution of convolutional recurrent neural networks for sound event detection,” *ESAT, Tech. Rep.*, 2021.

[11] N. K. Kim and H. K. Kim, “Self-training with noisy student model and semi-supervised loss function for DCASE 2021 challenge task 4,” *Gwangju Institute of Science and Technology, Tech. Rep.*, 2021.

[12] X. Zheng, H. Chen, and Y. Song, “Zheng USTC team’s submission for DCASE2021 TASK4 - semi-supervised sound event detection,” University of Science and Technology of China, Tech. Rep., 2021.

[13] J. Ebbers and R. Haeb-Umbach, “Pre-Training And Self-Training For Sound Event Detection In Domestic Environments,” *Paderborn University, Tech. Rep.*, 2022.

[14] W. Boes and H. Van hamme, “Audiovisual transfer learning for audio tagging and sound event detection,” *arXiv preprint arXiv:2106.05408*, 2021.

[15] S. Ioffe and C. Szegedy, “Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift,” in *Proceedings of ICML*, 2015, pp. 448–456.

[16] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: A Simple Way to Prevent Neural Networks from Overfitting,” *The Journal of Machine Learning Research*, vol. 15, no. 1, pp. 1929–1958, 2014.

[17] Y. Wang, J. Li, and F. Metze, “A comparison of five multiple instance learning pooling functions for sound event detection with weak labeling,” in *2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, 2019, pp. 31–35.

[18] D. P. Kingma and J. Ba, “Adam: A Method for Stochastic Optimization,” *arXiv preprint arXiv:1412.6980*, 2014.

[19] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, “mixup: Beyond empirical risk minimization,” *arXiv preprint arXiv:1710.09412*, 2017.

[20] D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le, “SpecAugment: A simple data augmentation method for automatic speech recognition,” *arXiv preprint arXiv:1904.08779*, 2019.

[21] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, “PyTorch: An Imperative Style, High-Performance Deep Learning Library,” in *Advances in Neural Information Processing Systems*, 2019, pp. 8024–8035.

[22] A. Mesaros, T. Heittola, and T. Virtanen, “Metrics for Polyphonic Sound Event Detection,” *Applied Sciences*, vol. 6, no. 6, p. 162, 2016.

[23] A. Mesaros, T. Heittola, A. Diment, B. Elizalde, A. Shah, E. Vincent, B. Raj, and T. Virtanen, “DCASE 2017 challenge setup: Tasks, datasets and baseline system,” in *DCASE 2017-Workshop on Detection and Classification of Acoustic Scenes and Events*, 2017.

[24] Ç. Bilen, G. Ferroni, F. Tuveri, J. Azcarreta, and S. Krstulović, “A framework for the robust evaluation of sound event detection,” in *2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, 2020, pp. 61–65.

[25] Serizel, Romain and Turpault, Nicolas and Eghbal-Zadeh, Hamid and Shah, Ankit Parag, “Large-scale weakly labeled semi-supervised sound event detection in domestic environments,” in *Proceedings of the Detection and Classification of Acoustic Scenes and Events 2018 Workshop (DCASE2018)*, 2018.
# CONFIDENCE REGULARIZED ENTROPY FOR POLYPHONIC SOUND EVENT DETECTION

Won-Gook Choi

Department of Electronic Engineering
Hanyang University, Seoul, Republic of Korea
onlyworld94@hanyang.ac.kr

Joon-Hyuk Chang*

Department of Electronic Engineering
Hanyang University, Seoul, Republic of Korea
jchang@hanyang.ac.kr

## ABSTRACT

One of the main issues of polyphonic sound event detection (PSED) is the class imbalance problem caused by the proportions of active and inactive frames. Since the target sounds occasionally appear, binary cross-entropy makes the model mainly fit on inactive frames. This paper introduces an effective objective function, confidence regularized entropy, which regularizes the confidence level to prevent overfitting of the dominant classes. The proposed method exhibits less overfitted samples and better detection performance than the binary cross-entropy. Also, we compare our method with the other objective function, the asymmetric focal loss also designed to solve the class imbalance problem in PSED. The two objective functions show different system characteristics. From an end-user perspective, we suggest choosing a proper objective function for the purposes.

**Index Terms**— Polyphonic sound event detection, class imbalance problem

## 1. INTRODUCTION

Polyphonic sound event detection (PSED) is one of the acoustic classification and detection tasks that detects the target sound and timestamps in an audio signal. For many years, PSED has had following several challenges:

* Difficult to gather strongly labeled recordings.

* The subjectivity problem of manual labeling.

* Hard to find an analytic model that can cover the various sound patterns.

* The class imbalance problem due to the proportion of active and inactive frames.

The first problem has been solved with two approaches: semi-supervised learning approaches using both labeled and unlabeled data and training with synthetic audio mixed background noise and target sounds. The second problem could be relieved by choosing the metric when comparing the system with others [1]. And the third problem has been solved by deep neural networks using improved convolutional neural networks (CNNs), Transformer, attention mechanisms, etc [2].

This study focuses on the last problem of class imbalance. Most inactive frames (background sound) dominate an audio clip, so the problem arises when detecting a target sound frame by frame (Fig. 1). This phenomenon is not a problem presented only in

<figure><img src="image_21.png" /><figcaption>Figure 1: Simple illustration of an audio clip with target sounds.</figcaption></figure>

PSED task; the image object detection also has suffered from the background-foreground class imbalance [3]. For the image object detection, a solution is the focal loss that controls the weight parameter of cross-entropy so that train the model well for the target object, but vice versa for the background images. Motivated by the focal loss, the previous study in PSED proposed asymmetric focal loss (AFL) [4] that could control the focal weights of entropies for the inactive and active terms, respectively. AFL successfully controlled the imbalance problem, but an adverse effect arose: the system detected repetitive impulsive sounds as a long-duration sound.

In this study, we propose confidence regularized entropy (CRE), which set the confidence threshold to the binary cross entropy (BCE). When calculating the BCE, samples are eliminated for the backpropagation during training steps if the detected results are over the threshold. The proposed method keeps the samples less overfitted, especially for the inactive frames. Compared to the AFL, the proposed entropy resulted in a system that can detect the onsets and offsets of target sounds well. Both CRE and AFL relieved the class imbalance problem for PSED. However, they showed a different system characteristic: the CRE-based system was advantageous in detecting a target event's precise localization on frames, whereas the AFL-based system showed strength in detecting whether a target sound appeared. The details will be discussed in Section 5.2.

## 2. CLASS IMBALANCE PROBLEM WITH PSED

A class imbalance problem is one of the considerations for building and training a neural network. If the class imbalance problem remains unsolved, the model could remain ungeneralized [3]. When collecting data from real world, the target sound would appear intermittently rather than often; thus, one of the factors that cause class imbalance is the imbalance between the number of active and inactive frames [4] in dealing with the PSED tasks. Additionally, the imbalance among the target sounds could appear since each event's duration is entirely different, and the amount of recorded sound is

<footnote>* corresponding author.</footnote>
<figure><img src="image_22.png" /><figcaption>Figure 2: Bar graphs representing the duration of active sounds (blue) and the number of active segments (yellow). Considering the total audio length, the proportion of inactive frames is large.</figcaption></figure>

also diverse according to the datasets.

The total duration and number of segments for each target sound composing the domestic environment sound event detection (DESED) dataset are shown in Fig. 2. In both synthetic and real data, inactive frames have large proportions and, all the events except speech have very low proportions.

Imoto et al. proposed asymmetric focal loss (AFL) [4] to control the entropies of active and inactive frames. The entropy between a ground truth $y_{n,c}$ and a model output $\hat{y}_{n,c}$ is described:

$$ AFL = - \sum_{n,c=0}^{N,C} \left\{ \begin{array}{l} \underbrace{(1 - \hat{y}_{nc})^{\gamma} y_{nc} \log(\hat{y}_{nc})}_{\text{Active term}} \\ + \underbrace{(\hat{y}_{nc})^{\zeta} (1 - y_{nc}) \log(1 - \hat{y}_{nc})}_{\text{Inactive term}} \end{array} \right\} \quad (1) $$

where $\gamma$ and $\zeta$ denote the parameters to control the entropies of active and inactive frames in each, and $N$ and $C$ denote the number of frames and target sounds, respectively. If $\gamma$ and $\zeta$ are set to 0, the entropy is same as the binary cross entropy, and the higher the values, the less focal. Imoto et al. set $\gamma$ and $\zeta$ to 0.0625 and 1, respectively, which means that the objective function focuses more on the active frames.

## 3. CONFIDENCE REGULARIZED ENTROPY

Suppose that there are lot proportion of speech-activated and inactive frames among the datum. If then, the inactive points are converged earlier than the other target sounds (e.g., cat, vacuum cleaner, etc. in Fig. 2). Even if the training epoch is processed enough, the inactive points are still converging more closely to one or zero, whereas the network is less optimized for the other target sounds. To concentrate on training the network for the false positive and false negative data, we propose the confidence regularized entropy (CRE) that can regularize confidences so that they could not converge beyond the threshold.

<figure><img src="image_23.png" /><figcaption>Figure 3: Training scenario when CRE is used for objective function.</figcaption></figure>

$$ CRE = - \frac{1}{NC} \sum_{n,c=0}^{N,C} \left[ \begin{aligned} & \mathbb{I}_{|\hat{y}_{nc} - y_{nc}| > \gamma(\hat{y}_{nc})} \cdot \{y_{nc} \cdot \log \hat{y}_{nc} \\ & + (1 - y_{nc}) \cdot \log(1 - \hat{y}_{nc})\} \end{aligned} \right] \quad (2) $$

where $\mathbb{I}(\cdot)$ denotes an indicator function. We set $\gamma$ to 0.01; the frames are excluded on each optimizing step, if those of confidence are either over the 0.99 or under 0.01. Generally, the mixup augmentation [5] is widely used for training the PSED network, and Eq. (2) also can be used whether the mixup is applied. If the mixup is used, confidences that are too close to the mixed labels are excluded during the training. In Section 5.1, the experimental results will demonstrate that a system that applied both CRE and mixup surpasses the system without either of them.

# 4. EXPERIMENTS

## 4.1. Dataset

To validate our proposed method, we used DESED database<fnref n="1" />[6]. There were ten sound events that could occur in domestic environments. For the training set, there were 10,000 synthetic clips with strong annotations, 3,470 recorded clips with strong labels coming from the Audioset [7], 1,578 recorded clips with weak labels, and 14,412 unlabeled-recorded clips. For the evaluation set, there were 1,168 recorded clips. Each clip had a 10 s duration and was provided either 16 kHz or 44.1 kHz and single or dual channel. All clips were down-mixed to 16 kHz and extracted to log-mel spectrograms. For the details, window size and shift size were used 2048 and 255 samples, respectively, and 128 mel-filter banks.

## 4.2. CNN networks

The CNN architecture for the experiments is shown in Fig. 5. The group size of convolutional layer was 4, and output channel sizes were 32, 64, 128, 256, 256, 256 and 128, respectively. To reduce a temporal size of feature map without temporal pooling, we stacked frames in 4 layers. Also, we designed the axis-wise attention module (AWAM) inspired by parallel temporal-spectral attention [8] to improve the baseline model [9]. AWAM is a module that calculates the sigmoid-based score for each axis and adds to the input feature map, and it was adopted after the 2nd, 4th, and 6th convolutional blocks. The detail of AWAM architecture is shown in Figs. 4. The RNN network was same to the baseline CRNN introduced in [9].

<footnote n="1">¹Strong labeled real recordings were newly released in DCASE 2022 challenge task 4. https://github.com/DCASE-REPO/DESED_task/tree/master/recipes/dcase2022_task4_baseline</footnote>
<figure><img src="image_24.png" /><figcaption>Figure 4: Axis wise attention module. $X_k$ denotes the output of the k-th convolutional block in Fig. 5. The architecture of $f_F(\cdot)$ and $f_T(\cdot)$ are same to $f_C(\cdot)$, but are performed on frequency and time axis, respectively.</figcaption></figure>

<figure><img src="image_25.png" /><figcaption>Figure 5: Block diagram of CNN architecture.</figcaption></figure>

## 4.3. Experimental setup

We adopted the mean teacher [9, 10], one of the semi-supervised learning strategies to train the detection model using the unlabeled data. A minibatch consists of synthetic, strong, weak, and unlabeled-recorded clips with batch sizes of 8, 8, 4, and 40 each. All networks were optimized with the AdamW [11] optimizer and the cosine-annealing learning rate scheduler for 50 epochs after warming up the first 50 epochs from 0 to 0.001. Also, we set the weight decay and dropout to 0.001 and 0.5, respectively. We used event-based f1 score [12] and polyphonic sound detection score (PSDS) [13] for the evaluation metrics<fnref n="2" />.

# 5. RESULTS AND DISCUSSION

## 5.1. Effect of confidence regularization

The experimental results according to the objective functions are compared in Table 1. If the other conditions are same except objective function, the systems built with CRE showed great performances under the event-f1 and PSDS1 metrics (CRE > BCE > AFL). Whereas, the systems built with AFL showed better performances under the PSDS2 metric (AFL > BCE > CRE), and the system with BCE showed medium performances for all metrics. Also, the proposed confidence regularization method was applicable with the mixup augmentation. The results demonstrate that if the detection performances of a mixup-applied system with BCE improved more than the system without the mixup, the mixup-applied system with CRE also improved. Although sounds and labels are mixed up, Eq. (2) keeps an output not too much fitting to the mixed label.

Confidence of detected sound event versus number of frames graphs are shown in Fig. 7. In the aspect of detection as the inactive frame, many frames with detection results close to 0 when BCE was used for the objective function. Most of the detection results of the CRE-based system were also close to 0 but more spread from 0 to 0.02 than the system with BCE. In other words, CRE made the model less overfitted to inactive frames, which shows the class imbalance problem was relieved. Whereas, the detection confidences of the AFL-based system were evenly distributed rather than biased towards zero.

In the aspect of detection as the active frame, all systems show similar results to each other but have a little difference. The peak of the CRE-based system's curve was left-biased due to the threshold; however, the peak of the AFL-based system's curve was right-biased since the focal weight was set to train well for the active frames. It demonstrates that the CRE-based network is trained well up to the regularization threshold and is prevented from overfitting when the confidences come over the threshold. On the other hand, according to the focal weights, the AFL-based network is trained well focused for the active frames but less focused for the inactive frames.

## 5.2. Discussion: system characteristics and PSDS

As shown in the experimental result, the system's scores are different according to the evaluation metrics. For instance, the system trained with CRE outperformed the system with AFL on PSDS1 but vice versa on PSDS2. Then which system should we choose or

<footnote n="2">²The specific parameters settings for all metrics were same to the recent DCASE challenge.
https://dcase.community/challenge2022/
task-sound-event-detection-in-domestic-environments</footnote>
<figure><img src="image_26.png" /><figcaption>Figure 6: Two detection examples of CRE- and AFL-based systems. (From up to bottom: log-mel spectrogram, ground truth, CRE-based system, AFL-based system)</figcaption></figure>

<table><caption>Table 1: Comparison of the detection performances among the different objective functions.</caption><thead><tr><th>Network</th><th>Loss</th><th>Event-F1</th><th>PSDS1</th><th>PSDS2</th></tr></thead><tbody><tr><td rowspan="3">CRNN<br>w/o mixup</td><td>BCE</td><td>43.27</td><td>33.78</td><td>59.91</td></tr><tr><td>AFL</td><td>40.63</td><td>31.99</td><td><strong>65.05</strong></td></tr><tr><td>CRE</td><td><strong>45.50</strong></td><td><strong>33.98</strong></td><td>57.30</td></tr><tr><td rowspan="3">CRNN<br>w/ mixup</td><td>BCE</td><td>44.93</td><td>34.79</td><td>58.60</td></tr><tr><td>AFL</td><td>41.90</td><td>32.79</td><td><strong>60.92</strong></td></tr><tr><td>CRE</td><td><strong>46.40</strong></td><td><strong>35.08</strong></td><td>57.82</td></tr><tr><td rowspan="3">CRNN+AWAM<br>w/ mixup</td><td>BCE</td><td>49.17</td><td>37.51</td><td>66.34</td></tr><tr><td>AFL</td><td>45.16</td><td>34.03</td><td><strong>66.88</strong></td></tr><tr><td>CRE</td><td><strong>51.11</strong></td><td><strong>38.12</strong></td><td>64.33</td></tr></tbody></table>

<figure><img src="image_27.png" /><figcaption>Figure 7: A graph of the number of frames over confidences of detected target sounds. Since there are a large number of inactive frames in the dataset, the graph of confidence near zero was log-scaled. (Top: confidence > 0.9, bottom: confidence < 0.1)</figcaption></figure>

which is better? As discussed in [1], PSDS1 is an effective metric for whether the system could detect the sound’s timestamp correctly; whereas it has a severe problem related to the labeler’s subjectivity. PSDS2 has strength in relieving the labeler’s subjectivity; however it is hard to detect event localization precisely, and highly depends on the long-duration sounds.

For further description, we analyze the different detection patterns of the systems as shown in Fig. 6. Just as people label subjectively according to their background, systems have different characteristics under the objective functions. The CRE-based system tends to detect onsets and offsets precisely, whereas the AFL-based system tends to detect sound longer than the ground truth. In other words, the AFL-based system is proper to detect whether a sound appears in a clip rather than timestamps. The more general analysis is shown in the top graph of Fig. 7. The blue line shows that the confidences are more fitted to zero for inactive frames than green line. Instead, the green line is spread evenly without being biased to one side.

From the standpoint of user experience, the CRE-based system (system having high PSDS1 but low PSDS2) is required for users or environments that need to detect the target sound’s onset and offset precisely. Whereas, the AFL-based system (system having high PSDS2 but low PSDS1) is more suitable in the environments for whether the appearance of target sounds is more important than

detection resolution.

## 6. CONCLUSION

In this paper, we introduced the confidence regularized BCE that could avoid overfitting inactive frames. Compared to AFL, CRE performed better under the event-based f1 score and PSDS1. Furthermore, we suggested choosing the proper objective function according to the user’s requirements. Of course, the system having good performance in both PSDS1 and PSDS2 is the best, but in a situation where you have to choose between the two, we can design a more suitable system by controlling the objective function.

## 7. ACKNOWLEDGEMENT

This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) (No.2021-0-00456, Development of Ultra-high Speech Quality Technology for Remote Multi-speaker Conference System)
8. REFERENCES

[1] G. Ferroni, N. Turpault, J. Azcarreta, F. Tuveri, R. Serizel, Ç. Bilen, and S. Krstulović, “Improving sound event detection metrics: insights from dcase 2020,” in *Proc. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, 2021, pp. 631–635.

[2] H. Sundar, M. Sun, and C. Wang, “Event specific attention for polyphonic sound event detection,” in *Proc. Interspeech*, 2021, pp. 566–570.

[3] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, “Focal loss for dense object detection,” in *Proc. IEEE international conference on computer vision*, 2017, pp. 2980–2988.

[4] K. Imoto, S. Mishima, Y. Arai, and R. Kondo, “Impact of sound duration and inactive frames on sound event detection performance,” in *Proc. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, 2021, pp. 860–864.

[5] H. Zhang, M. Cissé, Y. N. Dauphin, and D. Lopez-Paz, “Mixup: Beyond empirical risk minimization,” in *Proc. International Conference on Learning Representations (ICLR)*, 2018.

[6] S. Hershey, D. P. Ellis, E. Fonseca, A. Jansen, C. Liu, R. C. Moore, and M. Plakal, “The benefit of temporally-strong labels in audio event classification,” in *Proc. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, 2021, pp. 366–370.

[7] J. F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, “Audio set: An ontology and human-labeled dataset for audio events,” in *Proc. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, 2017, pp. 776–780.

[8] H. Wang, Y. Zou, D. Chong, and W. Wang, “Environmental sound classification with parallel temporal-spectral attention,” in *Proc. Interspeech*, 2020, pp. 821–825.

[9] N. Turpault, R. Serizel, J. Salamon, and A. P. Shah, “Sound event detection in domestic environments with weakly labeled data and soundscape synthesis,” in *Proc. Detection and Classification of Acoustic Scenes and Events Workshop (DCASE)*, New York University, NY, USA, October 2019, pp. 253–257.

[10] A. Tarvainen and H. Valpola, “Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results,” *Advances in neural information processing systems*, vol. 30, 2017.

[11] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” *arXiv preprint arXiv:1711.05101*, 2017.

[12] A. Mesaros, T. Heittola, and T. Virtanen, “Metrics for polyphonic sound event detection,” *Applied Sciences*, vol. 6, no. 6, p. 162, 2016.

[13] Ç. Bilen, G. Ferroni, F. Tuveri, J. Azcarreta, and S. Krstulović, “A framework for the robust evaluation of sound event detection,” in *Proc. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, 2020, pp. 61–65.
# EDANSA-2019: THE ECOACOUSTIC DATASET FROM ARCTIC NORTH SLOPE ALASKA

*Enis Berk Çoban*¹, *Megan Perra*², *Dara Pir*³, *Michael I Mandel*¹,⁴

¹ The Graduate Center, CUNY, New York, NY, USA, ecoban@gradcenter.cuny.edu

² Institute of Arctic Biology, UAF, Fairbanks, AK, USA, meperra@alaska.edu

³ Guttman Community College, CUNY, New York, NY, USA, dpir@gradcenter.cuny.edu

⁴ Brooklyn College, CUNY, Brooklyn, NY, USA, mim@sci.brooklyn.cuny.edu

## ABSTRACT

The arctic is warming at three times the rate of the global average, affecting the habitat and lifecycles of migratory species that reproduce there, like birds and caribou. Ecoacoustic monitoring can help efficiently track changes in animal phenology and behavior over large areas so that the impacts of climate change on these species can be better understood and potentially mitigated. We introduce here the Ecoacoustic Dataset from Arctic North Slope Alaska (EDANSA-2019), a dataset collected by a network of 100 autonomous recording units covering an area of 9000 square miles over the course of the 2019 summer season on the North Slope of Alaska and neighboring regions. We labeled over 27 hours of this dataset according to 28 tags with enough instances of 9 important environmental classes to train baseline convolutional recognizers. We are releasing this dataset and the corresponding baseline to the community to accelerate the recognition of these sounds and facilitate automated analyses of large-scale ecoacoustic databases.

**Index Terms**— Ecoacoustics, audio dataset, labeled data, baseline, biophony, anthrophony, geophony, convolutional network

<figure><img src="image_28.png" /><figcaption>Figure 1: Audio recording device locations.</figcaption></figure>

## 1. INTRODUCTION

The Arctic Coastal Plain is an ecosystem in northern Alaska and Canada that hosts over 180 migratory bird species from nearly every continent on the planet. The health of this ecosystem is inextricably linked to other habitats across the globe [1] and is undergoing rapid change due to global warming [2, 3] and land-use change [4]. This region has rich oil and gas resources; extraction and transportation of these fossil fuels increase the usage of machinery and vehicles. As a result, anthrophony from industrial activity or aircraft overflights may change the acoustic environment in the area. Aircraft overflights associated with this activity have been a community concern in the region. One village on the Coastal Plain, Nuiqsut, experiences air traffic equivalent to a city 95 times its size [5]. Past acoustic monitoring studies in Alaska have been smaller in geographic scope and utilized coarse acoustic indices or manual labeling [6, 7]. While some of this research has addressed anthrophony [7], our dataset is the first to account for both developed and undeveloped regions across the Arctic Coastal Plain. Such recordings are valuable for understanding the natural state of the Arctic acoustic environment, how development changes that state, and how that change affects wildlife.

Passive acoustic monitoring is an effective tool to monitor this system—and many others—because acoustic data can tell us about changes in wildlife populations, including phenology [8], biodiversity [9], community structure [10], and distribution [11]. Because

the volume of data produced by acoustic studies makes manual data processing prohibitively expensive, researchers have recently employed convolutional neural networks (CNN) to label the contents of large ecoacoustic datasets [12, 13, 14]. To train a CNN in a supervised fashion, researchers must label a small subset of data to train the model, and that labeled data is what we have provided in this paper. We used an earlier version of this dataset (batch-1, described in Subsection 2.3) in our research to understand the advantages of self-supervised learning and data valuation for audio classification [15]. We are providing the best performing model from that work as the baseline in this paper, which utilizes data augmentation [16, 17, 18] and global temporal pooling [19].

## 2. CORPUS

### 2.1. Monitoring sites

Samples were taken at latitudes between 64° and 70° N, and longitudes between 139° to 150° W, covering predominantly the Arctic Coastal Plain but also spanning tundra, shrub, and boreal forest ecosystems on the north and south of the Brooks Mountain Range in northern Alaska. A map of the recording sites is shown in Figure 1. We used 40 recording devices to cover the Prudhoe Bay oilfields and the 1002 portions of the Arctic National Wildlife Refuge (ANWR) in a grid separating locations by 20 km with a random
<table><caption>Figure 2: Audio recording times for monitors in each region. The top row displays the number of daylight hours for each corresponding month shown below it on the second row. The colored cells show the months where recordings were made for each region.</caption><thead><tr><th>daylight (h)</th><th>4</th><th>6</th><th>10</th><th>13</th><th>17</th><th>22</th><th>17</th><th>13</th><th>10</th><th>6</th><th>4</th><th>5</th></tr></thead><tbody><tr><th>month</th><td>1</td><td>2</td><td>3</td><td>4</td><td>5</td><td>6</td><td>7</td><td>8</td><td>9</td><td>10</td><td>11</td><td>12</td></tr><tr><th>ivvavik</th><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><th>dempster</th><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><th>anwr</th><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><th>dalton</th><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><th>prudhoe</th><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr></tbody></table>

offset. We also had 20 recording devices at sites along the length of the Dempster and Dalton Highways (10 devices each) in the Yukon Territories and Alaska, respectively, and an additional 10 recording devices placed at existing wildlife monitoring sites throughout Ivvavik National Park in the Yukon. Devices are deployed over 3-5 days in each region, starting from the 16th and 23rd of March, 2nd, 3rd, and 4th of May, respectively, in Dalton, Dempster, Ivvavik, Prudhoe, and Anwr. Recordings are saved locally on the device and collected manually. The acoustic recording units (ARUs) were SM4 wildlife recorders from Wildlife Acoustics, sampling at a rate of 48 kHz with gains set to 16 dB. ARUs recorded 150 minutes of audio at a time, with rotating breaks of 120 to 150 minutes in order to cycle through every hour of the day within a 4-day period. In total, devices recorded 2,161 days of audio data throughout 2019.

Recording periods for each region are shown in Figure 2. Recordings from the Prudhoe Bay Oilfields, ANWR, and Ivvavik were able to capture wildlife activity on the Arctic Coastal Plain and in the foothills of the Brooks Range, which serve as the primary breeding grounds for a majority of migratory species in the area. Sites along the Dempster and Dalton Highway captured the arrival and departure of those migratory birds that use the major north-south flyways that converge on the Coastal Plain.

## 2.2. Taxonomy

Labels of our dataset are members of three taxonomic ranks: coarse, medium, and fine. The coarse rank is the highest rank of the taxonomic tree and contains the four most general labels: "anthrophony", "biophony", "geophony", and "silence". A higher rank contains more general labels compared to the lower ranks. The medium rank contains more specific labels for each of the coarse rank labels and includes "bird", "insect", and "aircraft". The fine rank consists of the most specific labels, each belonging to one of the medium rank categories and includes "songbird", "waterfowl" (which in our dataset includes ducks, geese, and swans), and "upland bird" (including grouse and ptarmigan). Labels used in our baseline system are shown on the leftmost column of Table 1.

For example, a sample that contains a birdsong event is annotated with the "biophony", "bird", and "songbird" tags representing labels from the coarse, medium, and fine ranks, respectively. Annotating a sample with a child label will automatically annotate it with the parent label, however, it is possible for a sample to be annotated only with a parent label. Our annotators could assign nearly all samples a designation from the coarse labels. Fine labels under

<table><caption>Table 1: Number of samples per label in each of the four batches, batch-1 (b-1) through batch-4 (b-4). The last row shows the total number of samples in each batch.</caption><thead><tr><th>Label</th><th>b-1</th><th>b-2</th><th>b-3</th><th>b-4</th><th>Total</th></tr></thead><tbody><tr><td>Biophony</td><td>4107</td><td>500</td><td>776</td><td>886</td><td>6269</td></tr><tr><td>Bird</td><td>3821</td><td>493</td><td>78</td><td>52</td><td>4444</td></tr><tr><td>Songbird</td><td>2210</td><td>238</td><td>5</td><td>6</td><td>2459</td></tr><tr><td>Waterfowl</td><td>573</td><td>126</td><td>2</td><td>3</td><td>704</td></tr><tr><td>Upland Bird</td><td>386</td><td>44</td><td>2</td><td>2</td><td>434</td></tr><tr><td>Insect</td><td>372</td><td>36</td><td>734</td><td>846</td><td>1988</td></tr><tr><td>Anthrophony</td><td>328</td><td>217</td><td>1367</td><td>1165</td><td>3077</td></tr><tr><td>Aircraft</td><td>100</td><td>93</td><td>731</td><td>769</td><td>1693</td></tr><tr><td>Silence</td><td>1146</td><td>325</td><td>32</td><td>19</td><td>1522</td></tr><tr><td>Total</td><td>5566</td><td>1045</td><td>2133</td><td>2038</td><td>10782</td></tr></tbody></table>

the bird category were not uncommon, but fine labels under other categories were relatively rare or absent.

## 2.3. Sampling and labeling

Our dataset is composed of four batches differentiated by the sampling and labeling methods used, which are described below. We initially labeled data with broad coverage of time and space in our dataset so as to capture as many sound classes as possible without bias. We pulled our initial batch, batch-1, of random samples from each site within the Arctic National Wildlife Refuge and the adjacent Oilfields (sites 11-50). We selected a random 150-minute contiguous recording within each site, visually examined each recording's spectrogram in Audacity [20], and labeled all visually identified sounds present in that recording via listening. We excluded sound recordings that were inaudible due to wind-related clipping. We describe the details of four examples in Subsection 2.4.

The same expert labeler labeled sound clips in all batches based on the taxonomy described in Subsection 2.2. In almost all cases, sounds could be identified at the coarsest scale (e.g., "anthrophony" or "biophony"), and more specific labels were added as they could be identified. This generated 3083 separately labeled sounds that ranged in length from a few seconds to a few minutes. To generate equal-length samples, they were then split into 5566 non-overlapping, 10-second clips. However, not all clips' lengths were divisible by 10, generating clips less than 10 seconds. Clips less than 2 seconds were discarded, and clips between 2 and 10 seconds were zero-padded to the full 10 seconds.

We divided the whole set of samples into training, validation, and test sets so that all samples from a given site were confined to one of these three sets rather than split between them. To determine which site went to which set, we used a multiple knapsack problem detailed in [15]. This split ensured that we were measuring generalization across sites, and thus to future recordings.

In order to increase the number of examples in the rare class "anthrophony", we labeled more samples that we expected to be relevant from sites where this class was more common using predictions of a model trained on the training set of batch-1. At those sites, we pulled 500 10-second clips not tied to model confidence and 500 clips where model confidence for anthrophony was 0.75 or higher. We split the season into 12 weekly periods starting on May 7 and pulled ~80 samples from each weekly period, 40 of which were not tied to model confidence and were just the first 40 samples from that week, and the other 40 of which were tied to model confidence and tended to be distributed more throughout the week.
<figure><img src="image_29.png" /><figcaption>Figure 3: The depictions in this figure highlight the visual characteristics of spectrograms from four examples produced in Audacity. From left to right, these represent a truck driving on a road, a bird singing, and a loon singing near an acoustic recording unit (in sites 26, 50, and 11, respectively).</figcaption></figure>

For each week, we randomly chose one of the training sites (30, 29, 25, 24, 22, 21, 19, 16, 13, 49, 48, 44, 41, 40, 38, 37, 33) to pull samples from. The proportion of all 1000 samples taken from each training site depended on the proportion of anthrophony that was identified at that site in batch-1. So if 30% of all anthrophony in batch-1 had come from one site, 30% of our samples for batch-2 were also pulled from that site. This meant that, in some cases, we had to use multiple sites to fill our quota of samples for a given week. This process created a second batch of clips, batch-2. All of the clips from this 1000-sample set were labeled by trained undergraduate students, and then their labels were reviewed and corrected by an expert labeler before being finalized.

To ensure the accuracy of our original samples, plus the additional anthrophony samples, we built a user interface (UI) in Python that allowed users to quickly listen to a sample, view its spectrogram, and label it. The expert labeler checked or unchecked boxes next to each possible label to validate the original labels associated with a sample. All 6616 10-second clips that we had previously labeled were reviewed using this process. Note that all numbers provided in Table 1 are after this relabeling.

The UI made it considerably more efficient to review clips, allowing us to label additional samples from the "aircraft" sound class by selecting high-confidence predictions from the baseline model trained on batches 1 and 2. This created our third batch, batch-3. Note that batch-3 is only used for training, so we are less concerned that this selection process might bias the labels. Using this process, we were able to label an additional 2133 clips.

Performance was still below what we had hoped for the sound classes "insect", "anthrophony" and "aircraft", so we decided to collect a final batch of data, batch-4. In this batch, we labeled samples for the validation and test sets. Since selecting samples to label using a single model's confidence scores could lead to choosing only the type of samples that are successfully recognized by this model, we used an ensemble of 7 different iterations of our sound labeling model, including architectural and training variants, developed with earlier versions of the dataset. We normalized the confidence scores of each model across time to be between 0 and 1. Then assigned the maximum confidence across models to each label on each clip. Again, we pulled clips where this combined confidence was high, though this differed for each sound class as follows: 0.7 for anthrophony, 0.25 for aircraft, and 0.99 for the insect. The expert labeler used the UI to review these clips and was able to identify an additional 1,874 instances of labels so that the final set of label-clip associations was over 10,000. Table 1 displays the number of samples per label in each of the four batches of our dataset.

<figure><img src="image_30.png" /><figcaption>Figure 4: Number of clips out of 10,000 randomly sampled that have at most a particular clipping percentage (percentage of samples at maximum or minimum value).</figcaption></figure>

## 2.4. Labeling examples

When labeling batch-1, we viewed the 150-minute contiguous recordings as spectrograms in Audacity so we could locate sounds visually. The spectrogram was shown up to 24 kHz so that all possible sounds were visible. While the harmonics of birdsong could extend to 10-12 kHz, a majority of the visible signal on the spectrogram was at 9 kHz or below. The presence of anthrophony was generally indicated by lower frequency, uniform partials that tend to have a consistent pitch and long duration. The presence of biophony presents as harmonic signals with clear partials that have a wide frequency range and short duration, though they are often part of a larger temporal pattern, or song. Rain was the most common form of geophony and was clearly indicated by extremely brief, nonharmonic signals on the spectrogram that look similar to 'clicks', i.e., they have a vertical, broadband form that does not follow an intentional temporal pattern.

Figure 3 displays depictions that highlight the visual characteristics of spectrograms from three examples produced in Audacity. From left to right, the first depiction, labeled "anthrophony", represents a truck driving on a road, near an acoustic recording unit (site 26) in the oilfields. The lines highlight the visible, flat partials at the lower frequencies that are characteristic of anthrophony sounds. The second depiction represents a bird singing near an acoustic recording unit (site 50) in the Arctic National Wildlife Refuge and is labeled "songbird". The patterns highlight the signals created by the birdsong, which show a complex pattern of partials that extends from low to high frequencies; this complex song is repeated multiple times to produce a 'singing bout'. The third depiction represents a loon singing near an acoustic recording unit (site 11) in the oilfields and is labeled "waterfowl". The patterns highlight partials of the loon call, which show a definitive temporal pattern.

## 2.5. Clipping

Clipping is distortion caused by loud sounds. Recorders have a high and low limit for the amplitude of the sounds they can process; if these thresholds are passed, the data gets corrupted. We count the proportion of sample values in a 10-second interval that are at the maximum or minimum observed levels in a given recording.

An acceptable clipping percentage depends on the specific application. To pick a threshold of acceptable clipping level, we list-
tened to random 10-second clips and observed that a clipping pro-
portion less than 0.1% is almost unnoticeable to a listener. We there-
fore removed samples with more than 0.1% clipping before labeling
in batch-2, batch-3, and batch-4 of our dataset. Figure 4 shows for
10,000 randomly selected clips, how many had clipping less than
a given percentage. In a similar fashion, for batch-1, the expert la-
beler checked the recordings' spectrogram and only labeled those
without any visible clipping artifacts. 80% of these samples from
batch-1 have a clipping percentage lower than the 0.1% threshold.

3. BASELINE

Our initial labeling included examples of 28 unique categories in all levels of the hierarchy, out of 41 we thought we might encounter. Of these, only 9 had more than 100 examples in batch-1 and were used to train our baseline model: “biophony”, “bird”, “songbird”, “waterfowl”, “upland bird”, “insect”, “athrophony”, and “silence”.

As a baseline system<fnref n="1" />, we provide the best system described in [15]. Our baseline system employs the Ecoacoustic Dataset from Arctic North Slope Alaska (EDANSA-2019)<fnref n="2" /> and uses convolutional neural networks (CNNs) together with global temporal pooling and data augmentation. We share our code with MIT and our dataset under Creative Commons 4.0 licenses, which are highly permissive. We decided to use CNNs with hyperparameters inherited from AlexNet [21] due to their common success in sound event detection experiments [22]. Each sample is a 10-second clip, pre-processed and turned into a mel-spectrogram with a hop size of 23 ms, a window size of 42 ms, and 128 mel-frequency bins. We use a stack of 4 convolutional layers where all kernels are 5 × 5, followed by two fully connected layers. We train our model for 1600 epochs and keep the one with the highest mean AUC score over all labels on the validation set. Table 2 shows the AUC per label of the baseline model on the validation and test sets.

4. PREVIOUS WORK

There are a number of open-source datasets, similar to ours, shared along with their research findings. The CityNet dataset, which is collected from London, has diverse anthropogenic classes but the biophony classes are limited to only general labels like “bird”, “insect”, “vegetation”, and “wing beats” [23]. Another soundscape dataset consists of 5 hours of recordings collected from Sonoma County, California, USA and samples are labeled with “anthrophony”, “biophony”, “geophony”, “quiet”, and “interference” [14]. The main difference between these datasets and ours is that ours is recorded in remote locations and over a much larger area. Our dataset consists of 29 hours of labeled data, compared to 19 hours in CityNet and 5 hours from Sonoma County.

There are large datasets focusing on bird calls, which are chal-
lenging to model and of high scientific interest. BIRDCLEF is a
family of such datasets focusing on short targeted recordings as
opposed to long-term continuous recordings. It consists of sound
recordings collected by the Xeno-canto community and new ver-
sions with different purposes have been released every year since
2014. The latest, 2022 version, consists of 15k recordings, total-
ing over 190 hours covering 152 species from Hawaii, specially de-
signed for modeling calls of rare and endangered bird species with

<table><caption>Table 2: AUC per label of the baseline on validation and test sets.</caption>
  <thead>
    <tr>
      <th>Label</th>
      <th>Validation</th>
      <th>Test</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Biophony</td>
      <td>0.95</td>
      <td>0.96</td>
    </tr>
    <tr>
      <td>Bird</td>
      <td>0.96</td>
      <td>0.98</td>
    </tr>
    <tr>
      <td>Songbird</td>
      <td>0.90</td>
      <td>0.96</td>
    </tr>
    <tr>
      <td>Waterfowl</td>
      <td>0.87</td>
      <td>0.90</td>
    </tr>
    <tr>
      <td>Upland bird</td>
      <td>0.87</td>
      <td>0.93</td>
    </tr>
    <tr>
      <td>Insect</td>
      <td>0.90</td>
      <td>0.83</td>
    </tr>
    <tr>
      <td>Anthrophony</td>
      <td>0.88</td>
      <td>0.88</td>
    </tr>
    <tr>
      <td>Aircraft</td>
      <td>0.96</td>
      <td>0.88</td>
    </tr>
    <tr>
      <td>Silence</td>
      <td>0.96</td>
      <td>0.93</td>
    </tr>
    <tr>
      <td>Average</td>
      <td>0.92</td>
      <td>0.92</td>
    </tr>
  </tbody>
</table>

small amounts of training data [24]. Another dataset with 385 min-
utes of dawn chorus recordings was collected from Eastern North
America, including 48 species and 16,052 annotations [25]. Some
of the other datasets with bird calls are BirdVox [26], Nips4Bplus
[27], Freefield1010 [28], Warblrb10k and PolandNFC [29].

Larger general-purpose datasets have been extracted from YouTube such as Audio Set [30] and VGG SOUND [31] and include bioacoustic classes as a small part of their corpus. There are also continuously recorded open-source sound datasets without bioacoustic labels, such as SONYC-UST-V2, which is the output of an urban noise monitoring project and it is a multi-labeled [32]. This dataset is ~51 hours long in total and labeled with 8 main tags that are common in city environments, such as engine, music, and the human voice.

5. CONCLUDING REMARKS

This paper presented the Ecoacoustic Dataset from Arctic North Slope Alaska (EDANSA-2019), collected by autonomous recording units during the summer of 2019, and its corresponding baseline. We provided detail on the recordings and the sampling and labeling methods used to generate the four batches of our dataset. This work should help facilitate the analysis of large-scale ecoacoustic recordings made in arctic conditions, and it would be interesting to examine the extent to which models trained on this data can generalize to data collected in other environments and ecosystems.

6. ACKNOWLEDGMENT

We are grateful to Scott Leorna, Dr. Todd Brinkman, and Dr. Na-
talie T. Boelman for help with data collection and Eleanor Davol for
help with labeling. This work is supported by the National Science
Foundation (NSF) grant OPP-1839185. Any opinions, findings, and
conclusions or recommendations are those of the author(s) and do
not necessarily reflect the views of the NSF.

7. REFERENCES

[1] B. K. Sullender. (2019, Apr) Migratory birds in the heart of the
arctic: A journey through the arctic national wildlife refuge.

[2] J. E. Walsh and B. Brettschneider, “Attribution of recent warming in alaska,” *Polar Sci.*, vol. 21, pp. 101–109, 2019.

[3] P. Fauchald, T. Park, H. Tømmervik, R. Myneni, and V. H. Hausner, “Arctic greening from warming promotes declines

<footnote n="1">¹https://github.com/speechLabBcCuny/EDANSA-2019</footnote>

<footnote n="2">²https://zenodo.org/record/6824272</footnote>
in caribou populations,” *Science Advances*, vol. 3, no. 4, p. e1601365, 2017.

[4] M. K. Raynolds, D. A. Walker, K. J. Ambrosius, J. Brown, K. R. Everett, M. Kanevskiy, G. P. Kofinas, V. E. Romanovsky, Y. Shur, and P. J. Webber, “Cumulative geoecological effects of 62 years of infrastructure and climate change in ice-rich permafrost landscapes, prudhoe bay oilfield, alaska,” *Glob. Ch. Bio.*, vol. 20, no. 4, pp. 1211–1224, 2014.

[5] T. R. Stinchcomb, T. J. Brinkman, and D. Betchkal, “Extensive aircraft activity impacts subsistence areas: acoustic evidence from arctic alaska,” *Environmental Research Letters*, vol. 15, no. 11, p. 115005, oct 2020.

[6] R. Buxton, E. Brown, L. Sharmin, C. M. Gabriele, and M. F. McKenna, “Using bioacoustics to examine shifts in songbird phenology,” *Eco. & Evol.*, vol. 6, pp. 4697–4710, 05 2016.

[7] H. Vincelette, R. Buxton, N. Kleist, M. F. McKenna, D. Betchkal, and G. Wittemyer, “Insights on the effect of aircraft traffic on avian vocal activity,” *Ibis*, vol. 163, no. 2, pp. 353–365, 2021.

[8] R. Y. Oliver, D. P. W. Ellis, H. E. Chmura, J. S. Krause, J. H. Pérez, S. K. Sweet, L. Gough, J. C. Wingfield, and N. T. Boelman, “Eavesdropping on the Arctic: Automated bioacoustics reveal dynamics in songbird breeding phenology,” *Sci Adv*, vol. 4, no. 6, p. eaaq1084, 06 2018.

[9] D. Proppe, C. Sturdy, and C. St. Clair, “Anthropogenic noise decreases urban songbird diversity and may contribute to homogenization,” *Glob. Ch. Bio.*, vol. 19, pp. 1075–84, 04 2013.

[10] C. D. Francis, C. P. Ortega, and A. Cruz, “Noise pollution filters bird communities based on vocal frequency,” *PLOS ONE*, vol. 6, no. 11, pp. 1–8, 11 2011.

[11] N. J. Kleist, R. P. Guralnick, A. Cruz, and C. D. Francis, “Sound settlement: noise surpasses land cover in explaining breeding habitat selection of secondary cavity-nesting birds,” *Ecol Appl.*, vol. 27, no. 1, pp. 260–273, 01 2017.

[12] S. S. Sethi, N. S. Jones, B. D. Fulcher, L. Picinali, D. J. Clink, H. Klinck, C. D. L. Orme, P. H. Wrege, and R. M. Ewers, “Characterizing soundscapes across diverse ecosystems using a universal acoustic feature set,” *PNAS*, vol. 117, no. 29, pp. 17049–17055, 2020.

[13] J. LeBien, M. Zhong, M. Campos-Cerqueira, J. P. Velev, R. Dodhia, J. L. Ferres, and T. M. Aide, “A pipeline for identification of bird and frog species in tropical soundscape recordings using a convolutional neural network,” *Ecological Informatics*, vol. 59, p. 101113, 2020.

[14] C. A. Quinn, P. Burns, G. Gill, S. Baligar, R. L. Snyder, L. Salas, S. J. Goetz, and M. L. Clark, “Soundscape classification with convolutional neural networks reveals temporal and geographic patterns in ecoacoustic data,” *Ecological Indicators*, vol. 138, p. 108831, 2022.

[15] E. B. Çoban, A. R. Syed, D. Pir, and M. I. Mandel, “Towards large scale ecoacoustic monitoring with small amounts of labeled data,” in *Proc. WASPAA*, 2021, pp. 181–185.

[16] T. Ko, V. Peddinti, D. Povey, and S. Khudanpur, “Audio augmentation for speech recognition,” in *Proc. INTERSPEECH*, 2015, pp. 3586–3589.

[17] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, “mixup: Beyond empirical risk minimization,” *arXiv preprint arXiv:1710.09412*, 2017.

[18] D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le, “Specaugment: A simple data augmentation method for automatic speech recognition,” *arXiv preprint arXiv:1904.08779*, 2019.

[19] Q. Kong, Y. Cao, T. Iqbal, Y. Wang, W. Wang, and M. D. Plumbley, “Panns: Large-scale pretrained audio neural networks for audio pattern recognition,” *Tr. ASLP*, vol. 28, pp. 2880–2894, 2020.

[20] A. Team, 2019. [Online]. Available: https://www.audacityteam.org

[21] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” *NeurIPS*, vol. 25, pp. 1097–1105, 2012.

[22] K. Choi, G. Fazekas, and M. B. Sandler, “Automatic tagging using deep convolutional neural networks,” in *ISMIR*, 2016.

[23] A. J. Fairbrass, M. Firman, C. Williams, G. J. Brostow, H. Titheridge, and K. E. Jones, “Citynet—deep learning tools for urban ecoacoustic assessment,” *Methods in Ecology and Evolution*, vol. 10, no. 2, pp. 186–197, 2019.

[24] S. Kahl, A. Navine, T. Denton, H. Klinck, P. Hart, H. Glotin, H. Goëau, W.-P. Vellinga, R. Planqué, and A. Joly, “Overview of birdclef 2022: Endangered bird species recognition in soundscape recordings,” *Working Notes of CLEF*, 2022.

[25] L. M. Chronister, T. A. Rhinehart, A. Place, and J. Kitzes, “An annotated set of audio recordings of eastern north american birds containing frequency, time, and species information,” *Ecology*, vol. 102, 2021.

[26] V. Lostanlen, J. Salamon, A. Farnsworth, S. Kelling, and J. P. Bello, “Birdvox-full-night: A dataset and benchmark for avian flight call detection,” in *ICASSP*, 2018, pp. 266–270.

[27] V. Morfi, Y. Bas, H. Pamuła, H. Glotin, and D. Stowell, “Nips4bplus: a richly annotated birdsong audio dataset,” *PeerJ Computer Science*, vol. 5, p. e223, 2019.

[28] D. Stowell and M. D. Plumbley, “An open dataset for research on audio field recording archives: freefield1010,” *arXiv preprint arXiv:1309.5275*, 2013.

[29] F. Berger, W. Freillinger, P. Primus, and W. Reisinger, “Bird audio detection-dcase 2018,” *DCASE*, 2018.

[30] J. F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, “Audio set: An ontology and human-labeled dataset for audio events,” in *ICASSP*, 2017, pp. 776–780.

[31] H. Chen, W. Xie, A. Vedaldi, and A. Zisserman, “Vggsound: A large-scale audio-visual dataset,” in *ICASSP*, 2020, pp. 721–725.

[32] M. Cartwright, J. Cramer, A. E. M. Mendez, Y. Wang, H.-H. Wu, V. Lostanlen, M. Fuentes, G. Dove, C. Mydlarz, J. Salamon, et al., “Sonyc-ust-v2: An urban sound tagging dataset with spatiotemporal context,” *arXiv preprint arXiv:2009.05188*, 2020.
# ENSEMBLE OF MULTIPLE ANOMALOUS SOUND DETECTORS

Yufeng Deng¹, Anbai Jiang¹, Yuchen Duan¹, Jitao Ma³, Xuchu Chen¹, Jia Liu¹,², Pingyi Fan¹, Cheng Lu³, Wei-Qiang Zhang¹

¹ Department of Electronic Engineering, Tsinghua University, Beijing, China

² Tsinghua AI Plus, Beijing, China

³ School of Economics and Management, North China Electric Power University, Beijing, China
dyf20@mails.tsinghua.edu.cn, {liuj, fpy, wqzhang}@tsinghua.edu.cn, lucheng1983@163.com

## ABSTRACT

This paper presents our submission to DCASE 2022 Challenge Task 2, which aims to detect anomalous machine status via sounding by using machine learning methods, where the training dataset itself does not contain any examples of anomalies. We build six subsystems, including three self-supervised classification methods, two probabilistic methods and one generative adversarial network (GAN) based method. Our final submissions are four ensemble systems, which are different combinations of the six subsystems. The best official score of the ensemble systems can achieve 86.81% on the development dataset, whereas the corresponding Autoencoder-based baseline and the MobileNetV2-based baseline are with scores of 52.61% and 56.01%, respectively. In addition, our methods rank top on the development dataset and fourth on the evaluation dataset in this challenge.

**Index Terms—** DCASE, anomaly detection, domain generalization, machine condition monitoring, machine health monitoring

## 1. INTRODUCTION

The DCASE 2022 Challenge Task 2 is concerned with detecting anomalous state of the target machine using the sounding data. Unlike the acoustic scene classification, the available training data in this task contains samples of only one class — the normal-state class, but the aim is to detect whether a test sample is in another class, refer to as anomaly class, which may include various anomalous situations. A further complication added to this challenge is that the distributions of the training data and of the test data are different. This is called as domain shift. In the literature, there are some works investigating how to solve the domain shift problem by using machine learning methods and reducing the performance gap between the training and test data [1]. Although these techniques achieved impressive performance on image classification, they did not generally gain the expected and comparable results in the machine status detection via sounding up to now.

In this paper, we present six subsystems, the first three are self-supervised classifiers trained by using the supervision information provided by the metadata, similar to the approach taken by several teams at DCASE 2021 [2, 3] and DCASE 2020 [4]. The fourth and fifth models are probabilistic models. For the fourth model, inspired by the probabilistic model in [2], we employ normalizing flows to estimate the conditional density of the feature vectors for each section where the Mel spectrograms are used as the input of the pooling layers. Those output above the defined threshold will be marked as

anomaly. The fifth model estimates the conditional density of spectrogram target frames conditioned on remaining frames using an RNN based model and a GMM loss. The sixth model is a GAN based model.

We next present the results of the baseline systems in Section 2, and then describe each of our subsystems in detail in Section 3. For each subsystem, we will describe how it is trained and present its results on the development dataset. After that, we present our ensemble systems and their detection results in Section 4. Finally, conclusions are drawn in Section 5.

## 2. BASELINE RESULTS

In order to give a clear picture of the Challenge Task 2, we include the baseline scores on Table 1 and Table 2. To present the results succinctly, the results in all tables in this paper present only the harmonic mean of the source AUC, target AUC and pAUC for each machine type on the development dataset. Here the harmonic mean is denoted as h-mean. The data used in this challenge is 16 kHz, single-channel audio. For more details, please see [5, 6, 7].

<table><caption>Table 1: baseline-AE results</caption><thead><tr><th>bearing</th><th>fan</th><th>gearbox</th><th> slider</th><th>ToyCar</th><th>ToyTrain</th><th>valve</th><th>h-mean</th></tr></thead><tbody><tr><td>54.80%</td><td>58.47%</td><td>63.07%</td><td>57.99%</td><td>51.06%</td><td>39.61%</td><td>50.59%</td><td>52.61%</td></tr></tbody></table>

<table><caption>Table 2: baseline-MobileNetV2 results</caption><thead><tr><th>bearing</th><th>fan</th><th>gearbox</th><th> slider</th><th>ToyCar</th><th>ToyTrain</th><th>valve</th><th>h-mean</th></tr></thead><tbody><tr><td>59.16%</td><td>57.21%</td><td>59.91%</td><td>50.26%</td><td>54.23%</td><td>51.18%</td><td>62.42%</td><td>56.01%</td></tr></tbody></table>

## 3. APPROACHES

The general idea of the first three approaches is to first train a neural network to extract the embeddings of the samples by classifying labels extracted from the metadata, and then use the outlier detection algorithm to score how abnormal the embeddings are. The input to the first two models is a spectrogram with or without a Mel transformation, the difference between the first two models is that the first model uses a single index loss detection, while the second model uses a multiple indices loss detection. For the third model, the inputs are the embeddings extracted by the pre-trained wav2vec [8],
<figure><img src="image_31.png" /><figcaption>Figure 1: Overview of the classification based method.</figcaption></figure>

which is trained with a partial AudioSet [9]. The loss function for the first three models is ArcFace [10]. The fourth and fifth models differ from the first three in that they are not trained using any metadata information, which makes them completely unsupervised. The fourth model attempts to learn several distributions of some feature vector bins conditioned on the remaining bins. The fifth model estimates the distribution of target frames for the spectrogram conditioned on the remaining frames. The sixth model is a GAN based model. After describing these six subsystems, we present four ensembles of these subsystems, which are our final submission.

## 3.1. Classification Based Methods

In this subsection, we describe the first three subsystems, which we call SC (Section Classification), MHCR (Multi-head Classification & Regression) and SC-wav2vec. All of the three subsystems follow the overview shown in Figure 1.

The overview shown in Figure 1 is divided into three processes. First, we use the training samples to train the embedding network, and then use the trained embedding network to extract the embeddings of the training samples. Later on, we use these embeddings to train the outlier detector. Finally, the trained embedding network is used to extract the embeddings of the test samples, and the trained outlier detector is used to score the abnormality of these embeddings. In Figure 1, the embedding is extracted from the output of the last or penultimate layer of the embedding network. Since the ArcFace [10] loss function increases the inter-class distance and decreases the intra-class distance so that the network learns a better representation of the data, an ArcFace [10] layer is usually chosen for the classification head. The regression header is a fully connected layer.

### 3.1.1. Features & Training

The input feature of the embedding network used in the SC and MHCR is STFT spectrogram with or without a Mel transformation. For the SC-wav2vec, the input feature is the embedding extracted from a pre-trained wav2vec [8], which is trained using a partial AudioSet [9]. The logarithm is taken for the Mel spectrogram but not taken for the STFT spectrogram. We select the optimal STFT frame length, hop length, and number of frames based on the results on the development dataset and decide whether to use the Mel transform for each machine type. The specific feature parameters are shown in Table 3.

For the SC, the embedding network is trained to predict the section IDs using ArcFace [10] loss function. Since only the section ID metadata is used, there is only one classification head in the SC. For the MHCR, we additionally use other tags in the filenames to design classification (factory noise, microphone position, etc.) or regression (speed, weight, etc.) tasks. In order to achieve multi-label classification or regression, multiple parallel classification or regression heads are used. Different tasks are trained simultaneously. The hyperparameter λ is used to balance multiple losses, as shown in (1).

$$ \mathcal{L} = \sum_{i} \lambda_{i} \mathcal{L}_{i} \qquad (1) $$

where $\lambda_i > 0$ and $\mathcal{L}_i$ is the loss of label i. For the SC-wav2vec, we first train wav2vec [8] model using Fairseq [11] on the balanced AudioSet [9], while the features extracted using the pre-trained wav2vec [8] are input to the embedding network. The supervised label for the SC-wav2vec is the section IDs, hence, only one classification head is used in the SC-wav2vec.

Based on various experiments using the training dataset provided in section 0-2, we observe that the architecture of the embedding network has a significant impact on the performance and the optimal network architecture is different for different machine types, so we select the best performing network from Mobile-FaceNet (MFN) [12], MFNSE, Ecapa-tdnn [13] and Cnn6 for each machine type. The MFNSE is an improved network from the MFN, the difference between the MFNSE and MFN is that we add a squeeze-excitation [14] block to the bone block of the MFN. The Cnn6 is a 6-layer convolutional network used in [15].

For the training of embedding network, we adopt the AdamW optimizer with the default learning rate of $1 \times 10^{-3}$, weight decay $1 \times 10^{-4}$, and 25 epochs for training, where all the training data is drawn from the development and evaluation datasets. When training the wav2vec [8] model on the balanced AudioSet [9], we use the initial learning rate of $1 \times 10^{-7}$ and linearly increase the learning rate to $5 \times 10^{-3}$ in the first 500 updates, then decay to $1 \times 10^{-6}$ along the cosine curve.

### 3.1.2. Anomaly Detection Algorithms

Because the training samples only contain the normal ones, outlier detection algorithms are used to detect anomalous samples. Once the training is completed, the embedding network is used to extract the embeddings of the input samples, which are used to fit the outlier detectors. For the outlier detection, we employ four well known algorithms, k-NN [16], LOF [17], cosine distance and Mahalanobis distance. For the cosine distance and Mahalanobis distance, we compute the average embedding using the embeddings of the training samples, and additionally compute the covariance matrix for the Mahalanobis distance. In the test phase, the average
embedding and covariance matrix are used to compute the cosine and Mahalanobis distances of the test embedding, and use them to present the anomaly scores.

### 3.1.3. Results

Table 3 shows the results of the SC, Table 4 shows the results of the MHCR, while Table 5 shows the results of the SC-wav2vec.

<table><caption>Table 3: The results of SC</caption><thead><tr><th></th><th>bearing</th><th>fan</th><th>gearbox</th><th>slider</th><th>ToyCar</th><th>ToyTrain</th><th>valve</th></tr></thead><tbody><tr><td>feature</td><td>STFT</td><td>STFT</td><td>STFT</td><td>STFT</td><td>logmel</td><td>STFT</td><td>STFT</td></tr><tr><td>nMels</td><td>-</td><td>-</td><td>-</td><td>-</td><td>64</td><td>-</td><td>-</td></tr><tr><td>nffts</td><td>2048</td><td>2048</td><td>2048</td><td>2048</td><td>2048</td><td>2048</td><td>2048</td></tr><tr><td>nFrames</td><td>192</td><td>192</td><td>192</td><td>192</td><td>192</td><td>192</td><td>192</td></tr><tr><td>network</td><td>MFN</td><td>MFNSE</td><td>MFN</td><td>MFNSE</td><td>Ecapa-tdnn</td><td>Cnn6</td><td>MFNSE</td></tr><tr><td>h-mean</td><td>82.20%</td><td>80.64%</td><td>83.20%</td><td>88.27%</td><td>77.50%</td><td>75.92%</td><td>96.84%</td></tr></tbody></table>

<table><caption>Table 4: The results of MHCR</caption><thead><tr><th>bearing</th><th>fan</th><th>gearbox</th><th> slider</th><th>ToyCar</th><th>ToyTrain</th><th>valve</th><th>h-mean</th></tr></thead><tbody><tr><td>72.07%</td><td>72.16%</td><td>84.01%</td><td>83.18%</td><td>71.10%</td><td>68.14%</td><td>95.28%</td><td>77.01%</td></tr></tbody></table>

## 3.2. Probabilistic Models

In this subsection, we describe the fourth and fifth subsystems, which we call WSP-NFCDEE and IMDN (Interpolation Mixture Density Network).

The WSP-NFCDEE is built on the NFCDEE proposed in [2]. The difference between the NFCDEE and the WSP-NFCDEE is that we add a Weighted Statistic Pooling (WSP) layer before the normalizing flows, which improves the performance on most machine types, especially on the slider. Hence, we call this method WSP-NFCDEE. Let $X \in \mathbf{R}^{M \times T}$ is a Mel spectrogram, where $M$ is the number of Mel bins and $T$ is the number of frames. The WSP computes the mean and standard deviation of $X$ along time axis to obtain the mean vector $y \in \mathbf{R}^M$ and the standard deviation vector $z \in \mathbf{R}^M$. The output of the WSP is $\alpha \cdot y + \beta \cdot z$, where the $\alpha$ and $\beta$ are two trainable parameters satisfying the constraints (2).

$$ \alpha + \beta = 1, \alpha, \beta > 0 \qquad (2) $$

For the IMDN, we adopt three network structures mainly based on CNN and GRU, and a special density estimation loss function that combines the IDNN structure and Gaussian mixture model. We employ three networks named IGNN, LRCGNN, and a simplified DeepFilterNet [18]. Sufficient experiments had shown that networks based on RNN structure perform well on time series inputs. Hence, we select the 3D input with this form, (batch size, nFrames, nMels), which will be later propagated forward in the time dimension by the GRU, while CNN is mainly in the frequency dimension. We first select two lightweight network architectures, IGNN and LRCGNN. These two networks are with relatively low computation complexity, but achieve significant improvements over the Autoencoder-based baseline. The architecture of IGNN is shown in Table 6. The LRCGNN additionally add a Conv1d layer after the first fully connected layer of IGNN.

<table><caption>Table 5: The results of SC-wav2vec</caption><thead><tr><th></th><th>bearing</th><th>fan</th><th>gearbox</th><th> slider</th><th>ToyCar</th><th>ToyTrain</th><th>valve</th></tr></thead><tbody><tr><td>Scoring</td><td>maha</td><td>maha</td><td>k-NN</td><td>k-NN</td><td>LOF</td><td>k-NN</td><td>cosine</td></tr><tr><td>h-mean</td><td>58.54%</td><td>57.16%</td><td>58.01%</td><td>66.77%</td><td>71.90%</td><td>65.03%</td><td>68.89%</td></tr></tbody></table>

<table><caption>Table 6: The architecture of IGNN</caption><thead><tr><th>layer_name</th><th>parameters</th></tr></thead><tbody><tr><td>Fully Connected</td><td>(nMels, 128)</td></tr><tr><td>3 × GRU</td><td>(128, 128)</td></tr><tr><td>Fully Connected</td><td>(128, 32)</td></tr><tr><td>Fully Connected</td><td>(32, 128)</td></tr><tr><td>3 × GRU</td><td>(128, 128)</td></tr><tr><td>Fully Connected</td><td>((nF-1)×128, 2×nC×nMels)</td></tr></tbody></table>

To achieve a better performance on more complex data, we employ DeepFilterNet (DFnet) [18], a more complex Unet-like network proposed in speech enhancement. The computational complexity of the original DeepFilterNet [18] is high, so we use two fully connected layers in place of the original two convolutional layers, which greatly reduces the amount of computation. Additionally, we add a fully connected layer to match the number of Mel bins.

IDNN [19] was demonstrated to achieve significant improvement on non-stationary signals, which predicted the center frame of the input Mel spectrogram. We also adopt this idea. Another modification is that the Gaussian mixture model is adopted as the loss function. Let $x_p$ is the $p$-th frame to be predicted. By mapping input features $X$ to the parameters of the GMM, we obtain the frame predictions in probabilistic form (In the test system, the component weights are selected the same value):

$$ p(x_p|X, D, E) = \sum_{m=1}^{C} p_m(x_p|X, D, E) \qquad (3) $$

where $C$ is the number of Gaussian components and $p_m$ is the density of the $m$-th Gaussian component. Since the aim is to find the maximum probability density of the predicted frame, the LSE function is as follows.

$$ \mathcal{L}_{LSE} = -\log \sum_{i=1}^{M} \exp \left( \sum_{m=1}^{C} \log p_m(x_p|\mathbf{x}, D, E) \right) \qquad (4) $$

where $M$ is the number of Mel bins. When the covariance matrix of GMM is taken as diagonal matrix, this loss function is proved to be equivalent to maximization of the log likelihood of network with the given data.

### 3.2.1. Features & Training

The input feature of the WSP-NFCDEE and the IMDN is STFT spectrogram with Mel transformation and the logarithm is taken for the Mel spectrogram. For different machine types we tune the values of nMels, nffts, nFrames.

### 3.2.2. Results

Table 7 shows the results of the WSP-NFCDEE, while Table 8 shows the results of the IMDN.
<table><caption>Table 7: The results of WSP-NFCDEE</caption><thead><tr><th></th><th>bearing</th><th>fan</th><th>gearbox</th><th>slider</th><th>ToyCar</th><th>ToyTrain</th><th>valve</th></tr></thead><tbody><tr><td>nMels</td><td>128</td><td>128</td><td>64</td><td>64</td><td>64</td><td>64</td><td>64</td></tr><tr><td>nfftS</td><td>2048</td><td>2048</td><td>2048</td><td>2048</td><td>2048</td><td>2048</td><td>1024</td></tr><tr><td>nFrames</td><td>100</td><td>60</td><td>100</td><td>100</td><td>100</td><td>100</td><td>30</td></tr><tr><td>network</td><td>LRCGNN</td><td>LRCGNN</td><td>DFnet</td><td>LRCGNN</td><td>LRCGNN</td><td>DFnet</td><td>IGNN</td></tr><tr><td>h-mean</td><td>68.12%</td><td>68.54%</td><td>79.96%</td><td>78.52%</td><td>65.23%</td><td>60.29%</td><td>68.27%</td></tr></tbody></table>

<table><caption>Table 8: The results of IMDN</caption><thead><tr><th></th><th>bearing</th><th>fan</th><th>gearbox</th><th> slider</th><th>ToyCar</th><th>ToyTrain</th><th>valve</th></tr></thead><tbody><tr><td>nMels</td><td>128</td><td>128</td><td>64</td><td>64</td><td>64</td><td>64</td><td>64</td></tr><tr><td>nfftS</td><td>2048</td><td>2048</td><td>2048</td><td>2048</td><td>2048</td><td>2048</td><td>1024</td></tr><tr><td>nFrames</td><td>100</td><td>60</td><td>100</td><td>100</td><td>100</td><td>100</td><td>30</td></tr><tr><td>network</td><td>LRCGNN</td><td>LRCGNN</td><td>DFnet</td><td>LRCGNN</td><td>LRCGNN</td><td>DFnet</td><td>IGNN</td></tr><tr><td>h-mean</td><td>60.92%</td><td>62.24%</td><td>67.73%</td><td>73.17%</td><td>68.29%</td><td>65.07%</td><td>86.19%</td></tr></tbody></table>

### 3.3. AEGAN-AD

In this subsection, we describe the sixth subsystem, which we call AEGAN-AD.

We design an autoencoder which reconstructs Mel spectrograms and complement it with a discriminator, resulting in a GAN [20] model. Inspired by [21], we adopt a DCGAN [22]-like autoencoder, with the discriminator being our encoder and the generator being our decoder. As deconvolution suffers from checker-board effect, yet this effect is somehow resulting from the periodicity of the spectrogram, which makes reconstruction better than an “upsample-conv” structure. BNs are substituted by LNs in order to promote the detection in the target domain. Since most samples in a batch are from the source domain, it is likely that the network is misled by the biased statistics and only learns the distribution of source domain, resulting in a poor performance for the target domain. LN, which normalizes each sample independently, can learn to transform spectrograms into domain-invariant features. As for ToyCar and gearbox, we pass the latent variable through an adaptive LN which does different affine transformations for different sections. This observation indicates that it could help to transform reconstructed samples to their respective styles as in [23] so that features of different sections can be better represented. The loss function is selected as the MSE. Anomaly detection is conducted not only in input space, but also in the latent space, which is done by sending the reconstructed samples back to the encoder to obtain their latent representations. L1 norm, L2 norm and cosine are utilized to measure the difference of each spectrogram and the overall anomaly score is the mean/min/max of these. We select the best performing metric among these metrics.

For gearbox and slider, a discriminator is introduced to promote the reconstruction, while the autoencoder becomes the generator. The discriminator has the similar architecture with the encoder. It is trained to do a feature level discrimination on the reconstructed samples as a complement for MSE loss. Loss function for the discriminator is WGAN-GP [24] and loss function for the generator is a combination of MSE and feature matching loss [25]. Both of them are shown as (5) and (6), respectively.

$$ \mathcal{L}_D = \mathbb{E}_{\hat{x} \sim P_g}[D(\hat{x})] - \mathbb{E}_{x \sim P_r}[D(x)] + \lambda \mathbb{E}_{\hat{x} \sim P_{\bar{x}}}[(\|\nabla_{\hat{x}} D(\tilde{x})\|_2 - 1)^2] \quad (5) $$

$$ \mathcal{L}_G = \mathbb{E}_{x \sim P_r} [\|x - G(x)\|_2^2] + \mu \|\mathbb{E}_{x \sim P_r} [f(x)] - \mathbb{E}_{x \sim P_g} [f(\hat{x})]\|_2^2 \quad (6) $$

<table><caption>Table 9: The results of AEGAN-AD</caption><thead><tr><th>bearing</th><th>fan</th><th>gearbox</th><th> slider</th><th>ToyCar</th><th>ToyTrain</th><th>valve</th><th>h-mean</th></tr></thead><tbody><tr><td>75.78%</td><td>65.83%</td><td>71.50%</td><td>75.02%</td><td>79.16%</td><td>58.71%</td><td>52.52%</td><td>67.04%</td></tr></tbody></table>

where $P_r$ and $P_g$ denote the real distribution and the reconstructed distribution respectively. $P_{\bar{x}}$ is the linear combination of $P_r$ and $P_g$. $f(x)$ is the output of the last convolution layer in $D$. This embedding is also extracted during test time and it is compared with average embedding using k-NN [16], LOF [17], cosine and Mahalanobis distances. We simply choose the best performing metric from both G-based and D-based metrics.

All input for the model is 128 × 128 Mel spectrogram computed with 2048-point FFT and 512 hop-length. Logarithm is taken first and a MinMaxScalar then scales spectrograms to [-1, 1]. We use an Adam optimizer with a learning rate of $2 \times 10^{-4}$. The batch size is set to 512. The model is trained on both development set and evaluation set. The performance is shown in Table 9.

### 4. SUBMISSION RESULTS

In this subsection, we present the results of ensembles. For the ensembles, we combine the six subsystems by first standardizing the training data scores and then searching over a grid of convex combinations, similar to [2].

The difference between submission-1 and submission-2 is that for submission-1, we additionally train domain classifiers with Cnn6 for sections with obvious domain differences to predict whether the test samples belong to the source domain or the target domain. The test scores for both domains are normalized respectively. For the submission-3, we combine the top performing two or three subsystems for each machine type. For the submission-4, we only combine the SC and WSP-NFCDEE for each machine type. Table 10 shows the results.

<table><caption>Table 10: The results of ensembles</caption><thead><tr><th>method</th><th>bearing</th><th>fan</th><th>gearbox</th><th> slider</th><th>ToyCar</th><th>ToyTrain</th><th>valve</th><th>h-mean</th></tr></thead><tbody><tr><td>submission-1</td><td>87.75%</td><td>84.98%</td><td>87.55%</td><td>89.35%</td><td>83.52%</td><td>78.81%</td><td>98.10%</td><td>86.81%</td></tr><tr><td>submission-2</td><td>87.72%</td><td>84.34%</td><td>87.55%</td><td>88.77%</td><td>83.48%</td><td>78.15%</td><td>98.06%</td><td>86.51%</td></tr><tr><td>submission-3</td><td>87.62%</td><td>84.73%</td><td>87.54%</td><td>89.33%</td><td>82.25%</td><td>77.94%</td><td>98.10%</td><td>86.40%</td></tr><tr><td>submission-4</td><td>82.76%</td><td>83.88%</td><td>84.09%</td><td>89.25%</td><td>78.81%</td><td>76.57%</td><td>97.02%</td><td>84.18%</td></tr></tbody></table>

### 5. CONCLUSION

We have outlined our submission to the DCASE 2022 Challenge Task 2, which features a domain shift between the training and test distributions.

In this challenge, we build six subsystems and four ensemble systems in which four new unsupervised models, namely WSP-NFCDEE, IGNN, LRCGNN, and AEGAN-AD are integrated. All these four new models are employed with unsupervised training. Our methods are expected to be promising because they clearly match the data sampling characteristics of practical application scenarios for machine working condition detection in Industry 4.0. Moreover, our best official score of ensembles can achieve 86.81% on the development dataset, which is 30.80% higher than the best baseline. Finally, our method ranks fourth on the evaluation dataset.
# 6. ACKNOWLEDGEMENT

This work was supported by the National Natural Science Foundation of China under Grant No. U1836219 and No. 62276153.

# 7. REFERENCES

[1] G. Wilson and D. J. Cook, "A survey of unsupervised deep domain adaptation," *ACM Transactions on Intelligent Systems and Technology (TIST)*, vol. 11, no. 5, pp. 1-46, 2020.

[2] J. A. Lopez, G. Stemmer, P. Lopez Meyer, P. Singh, J. Del Hoyo Ontiveros, and H. Cordourier, "Ensemble of complementary anomaly detectors under domain shifted conditions," in *Proceedings of the 6th Detection and Classification of Acoustic Scenes and Events 2021 Workshop (DCASE2021)*, Barcelona, Spain, November 2021, pp. 11-15.

[3] K. Morita, T. Yano, and K. Tran, "Anomalous sound detection using cnn-based features by self supervised learning," DCASE2021 Challenge, Tech. Rep., July 2021.

[4] R. Giri, S. V. Tenneti, F. Cheng, K. Helwani, U. Isik, and A. Krishnaswamy, "Self-supervised classification for detecting anomalous sounds," in *Detection and Classification of Acoustic Scenes and Events Workshop (DCASE)*, 2020, pp. 46-50.

[5] K. Dohi, T. Nishida, H. Purohit, R. Tanabe, T. Endo, M. Yamamoto, Y. Nikaido, and Y. Kawaguchi, "MIMII DG: Sound dataset for malfunctioning industrial machine investigation and inspection for domain generalization task," *In arXiv e-prints*: 2205.13879, 2022.

[6] N. Harada, D. Niizumi, D. Takeuchi, Y. Ohishi, M. Yasuda, and S. Saito, "ToyADMOS2: Another dataset of miniature-machine operating sounds for anomalous sound detection under domain shift conditions," in *Proceedings of the 6th Detection and Classification of Acoustic Scenes and Events 2021 Workshop (DCASE2021)*, Barcelona, Spain, November 2021, pp. 1-5.

[7] K. Dohi, K. Imoto, N. Harada, D. Niizumi, Y. Koizumi, T. Nishida, H. Purohit, T. Endo, M. Yamamoto, and Y. Kawaguchi, "Description and discussion on DCASE 2022 challenge task 2: Unsupervised anomalous sound detection for machine condition monitoring applying domain generalization techniques," *In arXiv e-prints*: 2206.05876, 2022.

[8] S. Schneider, A. Baevski, R. Collobert, and M. Auli, "wav2vec: Unsupervised pre-training for speech recognition," *arXiv preprint arXiv:1904.05862*, 2019.

[9] J. F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, "Audio set: An ontology and human-labeled dataset for audio events," in *2017 IEEE international conference on acoustics, speech and signal processing (ICASSP)*. IEEE, 2017, pp. 776-780.

[10] J. Deng, J. Guo, N. Xue, and S. Zafeiriou, "Arcface: Additive angular margin loss for deep face recognition," in *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, 2019, pp. 4690-4699.

[11] M. Ott, S. Edunov, A. Baevski, A. Fan, S. Gross, N. Ng, D. Grangier, and M. Auli, "fairseq: A fast, extensible toolkit for sequence modeling," in *Proceedings of NAACL-HLT 2019: Demonstrations*, 2019.

[12] S. Chen, Y. Liu, X. Gao, and Z. Han, "Mobilefacenets: Efficient cnns for accurate real-time face verification on mobile devices," in *Chinese Conference on Biometric Recognition*. Springer, 2018, pp. 428-438.

[13] B. Desplanques, J. Thienpondt, and K. Demuynck, "Ecapatdnn: Emphasized channel attention, propagation and aggregation in tdnn based speaker verification," *arXiv preprint arXiv:2005.07143*, 2020.

[14] J. Hu, L. Shen, and G. Sun, "Squeeze-and-excitation networks," in *Proceedings of the IEEE conference on computer vision and pattern recognition*, 2018, pp. 7132-7141.

[15] Q. Kong, Y. Cao, T. Iqbal, Y. Wang, W. Wang, and M. D. Plumbley, "Panns: Large-scale pretrained audio neural networks for audio pattern recognition," *IEEE/ACM Transactions on Audio, Speech, and Language Processing*, vol. 28, pp. 2880-2894, 2020.

[16] S. Ramaswamy, R. Rastogi, and K. Shim, "Efficient algorithms for mining outliers from large data sets," in *Proceedings of the 2000 ACM SIGMOD international conference on Management of data*, 2000, pp. 427-438.

[17] M. M. Breunig, H.-P. Kriegel, R. T. Ng, and J. Sander, "Lof: identifying density-based local outliers," in *Proceedings of the 2000 ACM SIGMOD international conference on Management of data*, 2000, pp. 93-104.

[18] H. Schroter, A. N. Escalante-B, T. Rosenkranz, and A. Maier, "Deepfilternet: A low complexity speech enhancement framework for full-band audio based on deep filtering," in *ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. IEEE, 2022, pp. 7407-7411.

[19] K. Suefusa, T. Nishida, H. Purohit, R. Tanabe, T. Endo, and Y. Kawaguchi, "Anomalous sound detection based on interpolation deep neural network," in *ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. IEEE, 2020, pp. 271-275.

[20] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. WardeFarley, S. Ozair, A. Courville, and Y. Bengio, "Generative adversarial nets," *Advances in neural information processing systems*, vol. 27, 2014.

[21] S. Akcay, A. Atapour-Abarghouei, and T. P. Breckon, "Ganomaly: Semi-supervised anomaly detection via adversarial training," in *Asian conference on computer vision*. Springer, 2018, pp. 622-637.

[22] A. Radford, L. Metz, and S. Chintala, "Unsupervised representation learning with deep convolutional generative adversarial networks," *arXiv preprint arXiv:1511.06434*, 2015.

[23] X. Huang and S. Belongie, "Arbitrary style transfer in real-time with adaptive instance normalization," in *Proceedings of the IEEE international conference on computer vision*, 2017, pp. 1501-1510.

[24] I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. C. Courville, "Improved training of wasserstein gans," *Advances in neural information processing systems*, vol. 30, 2017.

[25] T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen, "Improved techniques for training gans," *Advances in neural information processing systems*, vol. 29, 2016.
# MIMII DG: SOUND DATASET FOR MALFUNCTIONING INDUSTRIAL MACHINE INVESTIGATION AND INSPECTION FOR DOMAIN GENERALIZATION TASK

Kota Dohi, Tomoya Nishida, Harsh Purohit, Ryo Tanabe, Takashi Endo
Masaaki Yamamoto, Yuki Nikaido, and Yohei Kawaguchi

Research and Development Group, Hitachi, Ltd.
1-280, Higashi-koigakubo, Kokubunji, Tokyo 185-8601, Japan
{kota.dohi.gr, yohei.kawaguchi.xk}@hitachi.com

## ABSTRACT

We present a machine sound dataset to benchmark domain generalization techniques for anomalous sound detection (ASD). Domain shifts are differences in data distributions that can degrade the detection performance, and handling them is a major issue for the application of ASD systems. While currently available datasets for ASD tasks assume that occurrences of domain shifts are known, in practice, they can be difficult to detect. To handle such domain shifts, domain generalization techniques that perform well regardless of the domains should be investigated. In this paper, we present the first ASD dataset for the domain generalization techniques, called MIMII DG. The dataset consists of five machine types and three domain shift scenarios for each machine type. The dataset is dedicated to the domain generalization task with features such as multiple different values for parameters that cause domain shifts and introduction of domain shifts that can be difficult to detect, such as shifts in the background noise. Experimental results using two baseline systems indicate that the dataset reproduces domain shift scenarios and is useful for benchmarking domain generalization techniques.

**Index Terms—** Machine sound dataset, Anomalous sound detection, Unsupervised learning, Domain shift, Domain generalization

## 1. INTRODUCTION

Anomalous sound detection systems (ASD) are automatic inspection systems that identify anomalous sounds emitted from machines [1–8]. Because these systems use microphones to conduct inspections, contactless inspections of anomalies inside the machines can be realized, unlike the vibration monitoring systems [9–11].

For the widespread application of ASD systems, researchers have mainly tackled two types of challenges. First, in real-world cases, only a few anomalous samples are available or provided. anomalous samples do not cover all possible types of anomalies. Therefore, unsupervised anomaly detection methods are often adopted so that the system can detect anomalies by training with only normal samples. MIMII [12] and ToyADMOS [13] are the first datasets that contain machine sounds in real factory environments, and are used for benchmarking the performance of unsupervised ASD methods.

Second, the detection performance of the system degrades due to changes in the distribution of normal sounds (i.e., domain shifts). Domain shifts for an ASD task can be classified into two categories; operational domain shifts caused by changes in states of a machine

and environmental domain shifts caused by changes in the background noise or in the recording environment. One solution for handling domain shifts is to use domain adaptation techniques and adapt the model to the new data. MIMII DUE [14] and ToyADMOS2 [15] were developed for benchmarking domain adaptation techniques, while an unsupervised scenario was also assumed.

However, in some real-world cases, domain generalization techniques [16–18] rather than domain adaptation techniques can be preferred. For example, if the operational domain shifts occur too frequently, adaptation of the model can be difficult. This is because only a small amount of data can be used for adaptation and frequent adaptation can be too costly. For another example, if domain shifts are difficult to detect, such as the domain shifts in the background noise, adaptation of the model can also be difficult. In these cases, domain generalization can be useful for handling domain shifts. Because these techniques aim at generalizing the model to detect anomalies regardless of the domains, adaptation of the model during the operation is not necessary. Therefore, domain generalization techniques for ASD task should be investigated for handling domain shifts that are too frequent to adapt or too difficult to detect.

To benchmark the domain generalization techniques for ASD task, a new dataset dedicated to the domain generalization task should be developed. This is because the data required for domain generalization and domain adaptation can be different. For example, generalization of the model may require a larger number of sets of data recorded under different conditions. Also, because domain generalization techniques are likely to be used for domain shifts that can be difficult to detect, this type of shifts should be included in the dataset for domain generalization tasks.

In this paper, we present a new dataset for benchmarking ASD methods using domain generalization techniques. The dataset consists of five different machine types; fan, gearbox, bearing, slide rail, and valve. Each machine type includes three sections, each of which corresponds to a type of domain shift. Each section consists of the source domain data to be used for generalizing the model and the target domain data for evaluating the domain generalization performance. The source domain has at least two different sets of values that cause domain shifts to generalize the model. Also, domain shifts that can only be handled with domain generalization techniques are included in the dataset. The dataset is freely available at https://zenodo.org/record/6529888 and is a subset of the dataset for Task 2 of the DCASE 2022 Challenge.
## 2. RECORDING ENVIRONMENT AND SETUP

We prepared five types of machines (fan, gearbox, bearing, slide rail, and valve), three types of factory noise data (factory noise A, B, and C), and three different domain shift scenarios for each machine type. The types of machines and domain shift scenarios were chosen on the basis of our experiences building ASD systems for real-world commercial solutions. Here, we identify each scenario of domain shifts by **section IDs**. The details of the type of domain shift for each section and the values of the parameters that shift between domains, the domain shift parameters, are described in Table 1.

We then recorded sound data of each machine to reproduce the domain shift scenarios we assumed. We recorded both normal and anomalous sounds for each domain, where to reproduce anomalous sounds, we used deliberately damaged machines or operated machines in an incorrect manner. For recording, we used a TAMAGO-03 microphone manufactured by System In Frontier Inc. [19]. The recording was conducted either in a sound-proof room (Fan and Valve) or in an anechoic chamber (Gearbox, Bearing, Slide rail). Although the microphone has eight channels, we only used the first channel for the dataset. Recorded sound clips are 16-bit audio with a sampling rate of 16 kHz and are 10 seconds long. Examples of spectrograms for each machine type are shown in Figure 1. A short description and recording procedures of each machine type are as follows.

**Fan** An industrial fan used to keep gas or air flowing in a factory. Operational conditions were kept the same between source and target domains, since Fan was dedicated to environmental domain shifts. Anomaly types include wing damage, unbalanced, clogging, and over voltage.

**Gearbox** A gearbox that links a direct current (DC) motor to a slider-crank mechanism, transmitting the power generated by the rotation of the motor at a constant speed to the slider-crank mechanism. The slider-crank mechanism then converts the rotational motion into a linear motion and raises and lowers its weight. We changed the operation voltage and mass of the weight to cause domain shifts. Anomaly types include gear damage and over voltage.

**Bearing** Two ball-type bearings are attached to a shaft with a spindle motor, and the sound is emitted from the bearing as it supports the rotating shaft. We changed the rotation speed of the shaft and the location of the microphones to cause domain shifts. Anomaly types include eccentricity in the bearing for two different directions.

**Slide rail ( slider )** A linear slide system consisting of a moving platform and a staging base that repeats a pre-programmed operation pattern. We changed the operation velocity and acceleration to cause domain shifts. Anomaly types include cracks on the rail, removal of grease, and a loose belt for a belt-type slide rail.

**Valve** A solenoid valve that repeatedly opens and closes in accordance with a pre-programmed operating pattern and is connected to a pump to control air or water flow. We changed the operating pattern and location of the panels surrounding the valve. Anomaly types include contamination in the valve.

After recording the machine sounds, we mixed the prerecorded

<figure><img src="image_32.png" /><figcaption>Figure 1: Examples of spectrograms for each machine type.</figcaption></figure>

factory noise A, B, or C as the background noise to simulate real-world environments. The factory noise A, B, and C were recorded in different real factories and consisted of sounds of various machinery. The noise-mixed data of each section was generated by the following steps.

1. The average power over all clips in the section, *a* was calculated.

2. For each clip *i* from the section,

(a) the signal-to-noise ratio (SNR) $\gamma$ dB for the clip was set to the value shown in Table 1,

(b) a background-noise clip *j* was randomly selected, and its power $b_j$ was tuned so that $\gamma = 10 \log_{10}(a/b_j)$, and

(c) the noise-mixed data was generated by mixing the machine sound clip *i* and the power-tuned background-noise clip *j*.

Here, the background-noise clip *j* was randomly selected from pre-determined types of factory noise, depending on the domain shift scenario. For Fan section 01, Bearing section 02, and Slide rail section 02, factory noise A and B were used for source domain and factory noise C was used for target domain. For other sections, factory noise A and B were used for both source and target domain. Also, for Fan section 00, we additionally mixed sound data of pumps from MIMII DUE.

The complete dataset consists of normal and anomalous operating sounds of five different types of industrial machines, and each machine type has three sections with source and target domain samples. Table 2 lists the number of samples in each section. The training data have 990 source domain samples and ten target domain samples for each section. We prepared ten target domain samples for training data so that the users can utilize a small number of target samples for generalization if the generalization of the model was too difficult. The test data have 50 normal samples and 50 anomalous samples for both domains.
<table><caption>Table 1: Type of domain shift, values of domain shift parameter, and SNR for each section. Values of domain shift parameters represent machines the sound of which are mixed in Fan section 00, levels of noise in Fan section 02, and locations of microphone in Bearing section 01.</caption><thead><tr><th>Machine type / section ID</th><th>SNR [dB]</th><th>Type of Domain shift [Domain shift parameter]</th><th>Parameter values for source-domain</th><th>Parameter values for target-domain</th></tr></thead><tbody><tr><td rowspan="3">Fan</td><td>00</td><td>-6.0 Mixing of different machine sound [machine sound index]</td><td>W, X</td><td>Y, Z</td></tr><tr><td>01</td><td>-12.0 Mixing of different factory noise [factory noise index]</td><td>A, B</td><td>C</td></tr><tr><td>02</td><td>N/A Different levels of noise [noise level (SNR [dB])]</td><td>L1 (3), L2 (-9)</td><td>L3 (-3), L4 (-15)</td></tr><tr><td rowspan="3">Gearbox</td><td>00</td><td>-6.0 Different operation voltage [V]</td><td>1.0, 1.5, 2.0, 2.5, 3.0</td><td>0.6, 0.8, 1.3, 1.8, 2.3, 2.3, 3.3, 3.5</td></tr><tr><td>01</td><td>-12.0 Different weight attached to the gearbox [g]</td><td>0, 50, 100, 150, 200</td><td>30, 80, 130, 180, 230, 250</td></tr><tr><td>02</td><td>-12.0 Different gearbox ID [machine ID]</td><td>05, 08, 13</td><td>00, 02, 11</td></tr><tr><td rowspan="3">Bearing</td><td>00</td><td>12.0 Different rotation speed [krpm]</td><td>6, 10, 14, 18, 22</td><td>2, 4, 8, 12, 16, 20, 24, 26</td></tr><tr><td>01</td><td>12.0 Different microphone location [location of the mic.]</td><td>A, B, C, D</td><td>E, F, G, H</td></tr><tr><td>02</td><td>12.0 Mixing of different factory noise [factory noise index]</td><td>A, B</td><td>C</td></tr><tr><td rowspan="3">Slide rail</td><td>00</td><td>-6.0 Different operation velocity [mm/s]</td><td>300, 500, 700, 900, 1100</td><td>100, 200, 400, 600, 800, 1000, 1200, 1300</td></tr><tr><td>01</td><td>-3.0 Different acceleration [m/s²]</td><td>0.03, 0.05, 0.07, 0.09, 0.11</td><td>0.01, 0.02, 0.04, 0.06, 0.08, 0.10, 0.12, 0.14</td></tr><tr><td>02</td><td>-12.0 Mixing of different factory noise [factory noise index]</td><td>A, B</td><td>C</td></tr><tr><td rowspan="3">Valve</td><td>00</td><td>0.0 Different open/close operation patterns [pattern index]</td><td>00, 01</td><td>02, 03</td></tr><tr><td>01</td><td>0.0 Different number and location of panels [panel locations]</td><td>open (no panels), bs-c (back-side closed)</td><td>b-c (back closed), s-c (side closed)</td></tr><tr><td>02</td><td>0.0 Different number of valves [(valve1 pattern index, valve2 pattern index)]</td><td>(v1 04), (v1 05), (v2 04), (v2 05)</td><td>(v1 04, v2 04), (v1 04, v2 05), (v1 05, v2 04), (v1 05, v2 05)</td></tr></tbody></table>

<table><caption>Table 2: Number of samples in each section</caption><thead><tr><th rowspan="2"></th><th colspan="2">Source domain</th><th colspan="2">Target domain</th></tr><tr><th>normal</th><th>anomaly</th><th>normal</th><th>anomaly</th></tr></thead><tbody><tr><th>Train</th><td>990</td><td>0</td><td>10</td><td>0</td></tr><tr><th>Test</th><td>50</td><td>50</td><td>50</td><td>50</td></tr></tbody></table>

### 3. RELATION TO MIMII DUE AND TOYADMOS2

While MIMII DUE and ToyADMOS2 were developed for domain adaptation tasks, MIMII DG in this paper is for domain generalization tasks. As described in Sec. 1, the domain generalization techniques are promised for handling domain shifts that domain adaptation techniques may not be applicable. We created a new dataset dedicated to the domain generalization tasks because the dataset for domain generalization tasks and domain adaptation tasks should be different in some points. We included these points as three main features that characterize differences from MIMII DUE and ToyADMOS2.

* The number of values the domain shift parameter (a parameter that causes domain shift) takes has increased to at least three for each type of domain shift. This change is crucial because domain generalization techniques may require multiple sets of data obtained from different domain shift parameter values to generalize the model [20]. For example, for the velocity shift in Slide rail, we increased the number of values of the velocity from four in MIMII DUE to 13 in MIMII DG. Also, with the increased number of sets, users can adjust the difficulty of the generalization task.

* Domain shifts that can be difficult to detect are introduced. As described in Sec. 1, domain generalization techniques are preferred for domain shifts that can be unnoticed. Therefore, we introduced difficult-to-detect domain shifts such as differences in states of a machine operating in the background.

* Domain shift parameters become easier to access and utilize. To generalize the model, not only the sound data but additional information such as the domain shift parameters and other attributes can be useful. Therefore, easy access to these additional information is crucial. Unlike MIMII DUE, we specified domain shift parameters in file names and attribute files for both the source and target domain. With domain shift parameters in the target domain, users can evaluate the detection performance for each value of the domain shift parameter.
# 4. EXPERIMENT

In this section, we use MIMII DG to benchmark the domain generalization performance of two baseline systems.

## 4.1. Baseline systems

We used two ASD systems for benchmarking; an autoencoder-based system and a MobileNetV2-based system. These systems are provided as the baseline systems in Task 2 of the DCASE 2022 Challenge, and Python implementations of the systems are available at https://github.com/Kota-Dohi/dcase2022_task2_basename_ae for the autoencoder-based system and https://github.com/Kota-Dohi/dcase2022_task2_basename_mobile_net_v2 for the MobileNetV2-based system.

The autoencoder-based system is often used as an unsupervised ASD system. Sound data were first converted to log-Mel spectrogram with a frame size of 1024, a hop size of 512, and 128 Mel bins. Five frames with four overlappings were successively concatenated to generate 640-dimensional input feature vectors. The model had four linear layers with 128 dimensions for the encoder, one bottleneck layer with eight dimensions, and four linear layers with 128 dimensions for the decoder. The model was trained to minimize the error between the input feature vector x and the reconstruction x'. We trained the model for 100 epochs using the Adam optimizer [21] with a learning rate of 0.0001 and a batch size of 128. The anomaly scores were calculated by the averaged reconstruction error.

The MobileNetV2-based system uses an auxiliary task to improve the detection performance of an unsupervised ASD system [22, 23]. 64 frames with 48 overlappings were successively concatenated to generate input feature vectors. For the model, we used a MobileNetV2 [24] with a multiplier parameter of 0.5. The model was trained to classify section IDs for each machine type. We trained the model for 20 epochs using the Adam optimizer with a learning rate of 0.0001 and a batch size of 128. The anomaly scores were calculated by the averaged negative logit of the predicted probabilities for the correct section.

## 4.2. Metric

We used the area under the receiver operating characteristic curve (AUC) for evaluation. Because the domain generalization task requires detecting anomalies even when the occurrence of domain shifts can be difficult to detect, the anomaly detector is expected to work with the same threshold regardless of the domain. Therefore, we calculated the AUC using both the source and target domain data. Also, to evaluate the anomaly detection performance for each domain, the AUC was computed for each domain. The AUC for each domain, section, and machine type was calculated as

$$ \text{AUC} = \frac{1}{N_d^- N_n^+} \sum_{i=1}^{N_d^-} \sum_{j=1}^{N_n^+} \mathcal{H}(A_{\theta}(x_j^+) - A_{\theta}(x_i^-)), \quad (1) $$

where n represents the index of a section, d ∈ {source, target} represents a domain, and $\mathcal{H}(x)$ returns 1 when x > 0 and 0 otherwise. $A_{\theta}(x)$ is the anomalous score of a sound clip x, where θ is the parameters of the system. Here, {$x_i^-$}$_{i=1}^{N_d^-}$ is normal test clips in the domain d in the section n and {$x_j^+$}$_{j=1}^{N_n^+}$ is anomalous test clips

<table><caption>Table 3: AUC (%) of each domain for each section.</caption><thead><tr><th rowspan="2">Machine type / section ID</th><th colspan="2">Autoencoder</th><th colspan="2">MobileNetV2</th></tr><tr><th>source</th><th>target</th><th>source</th><th>target</th></tr></thead><tbody><tr><td>00</td><td>84.69</td><td>39.35</td><td>71.07</td><td>62.13</td></tr><tr><td>Fan</td><td>01</td><td>71.69</td><td>44.74</td><td>76.26</td><td>35.12</td></tr><tr><td></td><td>02</td><td>80.54</td><td>63.49</td><td>67.29</td><td>58.02</td></tr><tr><td></td><td>00</td><td>64.63</td><td>64.79</td><td>63.54</td><td>67.02</td></tr><tr><td>Gearbox</td><td>01</td><td>67.66</td><td>58.12</td><td>66.68</td><td>66.96</td></tr><tr><td></td><td>02</td><td>75.38</td><td>65.57</td><td>80.87</td><td>43.15</td></tr><tr><td></td><td>00</td><td>57.48</td><td>63.07</td><td>67.85</td><td>60.17</td></tr><tr><td>Bearing</td><td>01</td><td>71.03</td><td>61.04</td><td>59.67</td><td>64.65</td></tr><tr><td></td><td>02</td><td>42.34</td><td>52.91</td><td>61.71</td><td>60.55</td></tr><tr><td></td><td>00</td><td>81.92</td><td>58.04</td><td>87.15</td><td>80.77</td></tr><tr><td>Slide rail</td><td>01</td><td>67.85</td><td>50.30</td><td>49.66</td><td>32.07</td></tr><tr><td></td><td>02</td><td>86.66</td><td>38.78</td><td>72.70</td><td>32.94</td></tr><tr><td></td><td>00</td><td>54.24</td><td>52.73</td><td>75.26</td><td>43.60</td></tr><tr><td>Valve</td><td>01</td><td>50.45</td><td>53.01</td><td>54.78</td><td>60.43</td></tr><tr><td></td><td>02</td><td>51.56</td><td>43.84</td><td>76.26</td><td>78.74</td></tr><tr><td>Average</td><td></td><td>67.21</td><td>53.99</td><td>68.72</td><td>56.42</td></tr></tbody></table>

in the section n in the machine type m. $N_d^-$ is the number of normal test clips in the domain d and $N_n^+$ is the number of anomalous test clips in the section n.

## 4.3. Results

Baseline results are shown in Table 3. On average, the AUC for the target domain data was lower than the source domain data at 13.2% with the autoencoder-based system and 12.3% with the MobileNetV2-based system. In some sections, the AUC of the target domain was slightly higher than that of the source domain. This could be because the target domain data happened to be similar to the source domain data of other sections. Overall, the fact that models trained with the source-domain tended to show lower performance for the target data indicate that there is a significant difference between the source-domain data and the target-domain data. This suggests that domain shift scenarios have been successfully reproduced. Thus, the dataset is useful for benchmarking the performance of domain generalization techniques.

# 5. CONCLUSION

We presented a new dataset, MIMII DG, which was developed for benchmarking domain generalization techniques for ASD. The dataset has normal and anomalous operating sounds of five different types of industrial machines with domain shifts. Experimental results using two ASD systems demonstrate that the detection performance significantly degrades for the target domain.

# 6. REFERENCES

[1] Y. Koizumi, S. Saito, H. Uematsu, and N. Harada, "Optimizing acoustic feature extractor for anomalous sound detection based on Neyman-Pearson lemma," in Proc. 25th European Signal Processing Conference (EUSIPCO), 2017, pp. 698–702.
[2] Y. Kawaguchi and T. Endo, “How can we detect anomalies from subsampled audio signals?” in Proc. 27th IEEE International Workshop on Machine Learning for Signal Processing (MLSP), 2017.

[3] Y. Koizumi, S. Saito, H. Uematsu, Y. Kawachi, and N. Harada, “Unsupervised detection of anomalous sound based on deep learning and the Neyman-Pearson lemma,” *IEEE/ACM Transactions on Audio, Speech, and Language Processing*, vol. 27, no. 1, pp. 212–224, Jan. 2019.

[4] Y. Kawaguchi, R. Tanabe, T. Endo, K. Ichige, and K. Hamada, “Anomaly detection based on an ensemble of dereverberation and anomalous sound extraction,” in Proc. 44th IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2019, pp. 865–869.

[5] Y. Koizumi, S. Saito, M. Yamaguchi, S. Murata, and N. Harada, “Batch uniformization for minimizing maximum anomaly score of DNN-based anomaly detection in sounds,” in Proc. IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), 2019, pp. 6–10.

[6] K. Suefusa, T. Nishida, H. Purohit, R. Tanabe, T. Endo, and Y. Kawaguchi, “Anomalous sound detection based on interpolation deep neural network,” in Proc. 45th IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020, pp. 271–275.

[7] H. Purohit, R. Tanabe, T. Endo, K. Suefusa, Y. Nikaido, and Y. Kawaguchi, “Deep autoencoding GMM-based unsupervised anomaly detection in acoustic signals and its hyperparameter optimization,” in Proc. 5th Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE), 2020, pp. 175–179.

[8] K. Dohi, T. Endo, H. Purohit, R. Tanabe, and Y. Kawaguchi, “Flow-based self-supervised density estimation for anomalous sound detection,” in Proc. 46th IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021, pp. 336–340.

[9] E. P. Carden and P. Fanning, “Vibration based condition monitoring: a review,” *Structural health monitoring*, vol. 3, no. 4, pp. 355–377, 2004.

[10] G. Toh and J. Park, “Review of vibration-based structural health monitoring using deep learning,” *Applied Sciences*, vol. 10, no. 5, p. 1680, 2020.

[11] A. Khademi, F. Raji, and M. Sadeghi, “Iot enabled vibration monitoring toward smart maintenance,” in *2019 3rd International Conference on Internet of Things and Applications (IoT)*, 2019, pp. 1–6.

[12] H. Purohit, R. Tanabe, T. Ichige, T. Endo, Y. Nikaido, K. Suefusa, and Y. Kawaguchi, “MIMII dataset: Sound dataset for malfunctioning industrial machine investigation and inspection,” in Proc. 4th Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE), 2019, pp. 209–213.

[13] Y. Koizumi, S. Saito, H. Uematsu, N. Harada, and K. Imoto, “ToyADMOS: A dataset of miniature-machine operating sounds for anomalous sound detection,” in Proc. IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), 2019, pp. 313–317.

[14] R. Tanabe, H. Purohit, K. Dohi, T. Endo, Y. Nikaido, T. Nakamura, and Y. Kawaguchi, “MIMII DUE: Sound dataset for malfunctioning industrial machine investigation and inspection with domain shifts due to changes in operational and environmental conditions,” in Proc. IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), 2021.

[15] N. Harada, D. Niizumi, D. Takeuchi, Y. Ohishi, M. Yasuda, and S. Saito, “ToyADMOS2: Another dataset of miniature-machine operating sounds for anomalous sound detection under domain shift conditions,” in Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE), 2021.

[16] K. Zhou, Z. Liu, Y. Qiao, T. Xiang, and C. Change Loy, “Domain generalization: A survey,” *arXiv e-prints*, pp. arXiv:2103.2021.

[17] Y. Wang, H. Li, and A. C. Kot, “Heterogeneous domain generalization via domain mixup,” in ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020, pp. 3622–3626.

[18] T. Dissanayake, T. Fernando, S. Denman, H. Ghaemmaghami, S. Sridharan, and C. Fookes, “Domain generalization in biosignal classification,” *IEEE Transactions on Biomedical Engineering*, vol. 68, no. 6, pp. 1978–1989, 2021.

[19] System In Frontier Inc. (https://www.sifi.co.jp/product/microphone-array/).

[20] K. Dohi, T. Endo, and Y. Kawaguchi, “Disentangling physical parameters for anomalous sound detection under domain shifts,” *arXiv preprint arXiv:2111.06539*, 2021.

[21] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in Proc. 3rd International Conference on Learning Representations (ICLR), 2015.

[22] R. Giri, S. V. Tenneti, F. Cheng, K. Helwani, U. Isik, and A. Krishnaswamy, “Self-supervised classification for detecting anomalous sounds,” in Proc. 5th Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE), 2020, pp. 46–50.

[23] P. Primus, V. Haunschmid, P. Praher, and G. Widmer, “Anomalous sound detection as a simple binary classification problem with careful selection of proxy outlier examples,” in Proc. 5th Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE), 2020, pp. 170–174.

[24] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, “MobileNetV2: Inverted residuals and linear bottlenecks,” in Proc. 31st IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 4510–4520.
# DESCRIPTION AND DISCUSSION ON DCASE 2022 CHALLENGE TASK 2: UNSUPERVISED ANOMALOUS SOUND DETECTION FOR MACHINE CONDITION MONITORING APPLYING DOMAIN GENERALIZATION TECHNIQUES

Kota Dohi¹, Keisuke Imoto², Noboru Harada³, Daisuke Niizumi³, Yuma Koizumi⁴, Tomoya Nishida¹, Harsh Purohit¹, Takashi Endo¹, Masaaki Yamamoto¹, and Yohei Kawaguchi¹

¹ Hitachi, Ltd., Japan, kota.dohi.gr@hitachi.com

² Doshisha University, Japan, keisuke.imoto@ieee.org

³ NTT Corporation, Japan, noboru.harada.pv@hco.ntt.co.jp

⁴ Google, Japan, koizumiyuma@google.com

## ABSTRACT

We present the task description and discussion on the results of the DCASE 2022 Challenge Task 2: “Unsupervised anomalous sound detection (ASD) for machine condition monitoring applying domain generalization techniques”. Domain shifts are a critical problem for the application of ASD systems. Because domain shifts can change the acoustic characteristics of data, a model trained in a source domain performs poorly for a target domain. In DCASE 2021 Challenge Task 2, we organized an ASD task for handling domain shifts. In this task, it was assumed that the occurrences of domain shifts are known. However, in practice, the domain of each sample may not be given, and the domain shifts can occur implicitly. In 2022 Task 2, we focus on domain generalization techniques that detects anomalies regardless of the domain shifts. Specifically, the domain of each sample is not given in the test data and only one threshold is allowed for all domains. Analysis of 81 submissions from 31 teams revealed two remarkable types of domain generalization techniques: 1) domain-mixing-based approach that obtains generalized representations and 2) domain-classification-based approach that explicitly or implicitly classifies different domains to improve detection performance for each domain.

**Index Terms**— anomaly detection, acoustic condition monitoring, domain shift, domain generalization, DCASE Challenge,

## 1. INTRODUCTION

Anomalous sound detection (ASD) [1–7] is the task of identifying whether the sound emitted from a target machine is normal or anomalous. Automatic detection of mechanical failure is essential in the fourth industrial revolution, which involves artificial intelligence (AI)–based factory automation. Prompt detection of machine anomalies by observing sounds is useful for machine condition monitoring.

One challenge regarding the application scope of ASD systems is that anomalous samples for training can be insufficient both in number and type. In 2020, we organized the fundamental ASD task in Detection and Classification of Acoustic Scenes and Event (DCASE) Challenge 2020 Task 2 [8]; “*unsupervised ASD*” that was aimed to detect unknown anomalous sounds using only normal sound samples as the training data [1–7]. For the wide spread application of ASD systems, advanced tasks such as handling of domain shifts should be tackled. Domain shifts are differences in acoustic

characteristics between the source and target domain data caused by differences in a machine’s operational conditions or environmental noise. Because these shifts are caused by factors other than anomalies, the detection performance of models trained with the source domain data can degrade for the target domain data. Therefore, in 2021, we organized DCASE Challenge 2021 Task 2 [9], “*unsupervised ASD under domain shifted conditions*” that focused on handling domain shifts using domain adaptation techniques.

The task in 2021 involved the use of domain adaptation techniques under two assumptions. First, all domain shifts have been detected in advance, and the domain of each sample is known. Second, the domain shifts do not occur too frequently for the model to adapt. However, these assumptions may not hold for certain real-world scenarios. For example, a machine’s background sound can be affected by various sound sources surrounding the machine, and it can be difficult to identify the cause of changes and attribute the changes to the domain shift. Also, because the operational conditions of the machine can change within a short period, adapting the model every time can be too costly. Therefore, methods have to be investigated such that the detection of domain shifts is unnecessary and frequent occurrences of domain shifts can be handled.

To solve the problem described above, we designed DCASE challenge 2022 Task 2 “*Unsupervised Detection of Anomalous Sounds for Machine Condition Monitoring Applying Domain Generalization Techniques*”. This task is aimed at developing domain generalization techniques to handle domain shifts. The task involves the use of domain generalization techniques so that the developed ASD systems do not require detection of the domain shifts or adaptation of the model. Specifically, to evaluate the generalization performance, the domain of each sample is not provided in the test data. To enhance generalization of the model, attributes that caused domain shifts are also provided in the training data.

We received 81 submissions from 31 teams. By analyzing these submissions, we found two types of domain generalization techniques: 1) domain-mixing-based approach and 2) domain-classification-based approach. The domain-mixing-based approach aims at obtaining generalized representations across domains by mixing data from different domains. In contrast, the domain-classification-based approach differentiates different domains so that the model can be specialized for each domain.
## 2. UNSUPERVISED ANOMALOUS SOUND DETECTION APPLYING DOMAIN GENERALIZATION TECHNIQUES

Let the L-sample time-domain observation $x \in \mathbb{R}^L$ be an audio clip that includes a sound emitted from a machine. The ASD task is a task to determine whether a machine is in a normal or anomalous state using an anomaly score $A_\theta(x)$ calculated by an anomaly score calculator $\mathcal{A}: \mathbb{R}^L \to \mathbb{R}$ with parameters $\theta$. The machine is determined to be anomalous when $A_\theta(x)$ exceeds a pre-defined threshold $\phi$ as

$$ \text{Decision} = \begin{cases} \text{Anomaly} & (A_{\theta}(x) > \phi) \\ \text{Normal} & (\text{otherwise}). \end{cases} \quad (1) $$

The primary difficulty in this task is to train $\mathcal{A}$ with only normal sounds. This is because anomalies are rarely obtained in practice.

Domain-shift is another major issue in real-world applications. Domain shifts mean a difference in conditions between training and testing. The conditions are machine's operational conditions such as its speed, load, and temperature, or the environmental conditions such as the type of environmental noise, level of the noise, and location of the microphone. Differences in these conditions change the distribution of data and degrades the detection performance. Let us define two domains: **source domain** and **target domain**, where the source domain is the original condition with enough training clips and the target domain is another condition with zero or a few training clips. Also, let $\mathcal{D}_S$, $\mathcal{D}_T$, $\mathcal{D}_{SA}$, and $\mathcal{D}_{TA}$ be the distributions of $x$ under the normal condition in the source domain, normal condition in the target domain, anomalous condition in the source domain, and anomalous condition in the target domain, respectively.

The task in DCASE 2021 Task 2 involved two tasks. One was to detect anomalies in the source domain: determine whether $x_s$ is from $\mathcal{D}_S$ or $\mathcal{D}_{SA}$ using an anomaly score calculator $A_{\theta_s}(x)$ and a threshold $\phi_s$. The other was detection in the target domain: whether $x_t$ is generated from $\mathcal{D}_T$ or $\mathcal{D}_{TA}$ using an anomaly score calculator $A_{\theta_t}(x)$ and a threshold $\phi_t$. The task was set to develop domain adaptation techniques so that the detection performance on the target domain can be improved by adaptation on the model trained with the source domain data. Although this problem setting assumes that the domain (source/target) of each sample is known, in practice, the detection of domain shifts can be difficult and the domain may not be available. Also, the use of domain adaptation techniques can be too costly if the domain shifts occur too frequently.

We show four types of real-world scenarios for these problems.

### Domain shifts due to differences in machine's conditions

Characteristics of a machine sound can change due to changes in the machine's operational conditions. Although these shifts can be detected, if these conditions change within a short period of time, it can be too costly to adapt the model every time.

### Domain shifts due to differences in environmental conditions

Because characteristics of background noise can be affected by various factors, it is difficult to detect these shifts. Therefore, a model that is unaffected by these shifts is desirable.

### Domain shifts due to maintenance

Characteristics of a machine sound can change after maintenance or parts replacement. Though these shifts can be detected, adapting the model every time can be costly.

### Domain shifts due to differences in recording devices

In real-world scenarios, many microphones are installed at different locations, and these microphones may be from different manufacturers. Although these shifts can be detected, adapting the model

for each location or microphone can be too costly.

As a possible solution to handle these problems, domain generalization techniques should be investigated. Domain generalization techniques for ASD aims at detecting anomalies from different domains with a single threshold. These techniques, unlike domain adaptation techniques, do not require detection of domain shifts or adaptation of the model in the testing phase. Therefore, domain generalization techniques can be used for handling domain shifts that are difficult to detect or too costly to adapt.

The DCASE 2022 Task 2 is set to develop domain generalization techniques for ASD. Because the domain generalization techniques are expected to work regardless of the domains, the domain of each sample is not given in the test data. The task is to determine if $x$ is from the normal condition $\mathcal{D}_S \cup \mathcal{D}_T$ or anomalous condition $\mathcal{D}_{SA} \cup \mathcal{D}_{TA}$ using an anomaly score calculator $A_\theta(x)$ and $\phi$. Because the differences in operational or environmental conditions make $\mathcal{D}_S \neq \mathcal{D}_T$, the decision must be executed without being affected by the differences between different domains.

## 3. TASK SETUP

### 3.1. Dataset

We used ToyADMOS2 [10] and MIMII DG [11] to generate the dataset. The dataset consists of normal/anomalous operating sounds from seven types of toy/real machines (ToyCar, ToyTrain, fan, gearbox, bearing, slide rail, and valve).

Each recording is a single-channel and 10-sec-long audio with a sampling rate of 16 kHz. We mixed machine sounds recorded at laboratories and the environmental noise recorded at real-world factories to create the training/test data. Details of the recording procedure can be found in [10] and [11].

In this dataset, **Machine type** means the type of machine. **Section** is defined as a subset of the data within a machine type and corresponds to a type of domain shift scenario.

We provide three datasets: **development dataset**, **additional training dataset**, and **evaluation dataset**. The **development dataset** consists of three sections (Sections 00, 01, and 02), which are sets of the training and test data. Each section provides (i) 990 normal clips from a source domain for training, (ii) 10 normal clips from a target domain for training, (iii) 100 normal clips and 100 anomalous clips from both domains for the test. We provided domain information (source/target) in the test data for the convenience of participants. Attributes represent the operational or environmental conditions, e.g. velocity of slide rail and level of noise (SNR) mixed in fan data. The **additional training dataset** provides training clips for three sections (Sections 03, 04, and 05). Each section consists of (i) 990 normal clips in a source domain for training and (ii) 10 normal clips in a target domain for training. Attributes are also provided. The **evaluation dataset** provides test clips for three sections (Sections 03, 04, and 05). Each section consists of 200 test clips, none of which have a condition label (i.e., normal or anomaly) or the domain information. Attributes are not provided. The main difference from our task in 2021 is that the domain information is not given in the evaluation dataset. Thus, the participants have to develop a system that performs well regardless of the domains.

### 3.2. Evaluation metrics

This task is evaluated with the area under the receiver operating characteristic (ROC) curve (AUC) and the partial AUC (pAUC).
The pAUC is calculated as the AUC over a low false-positive-rate (FPR) range [0, p]. In this task, we used $p = 0.1$.

Because the domain generalization task requires detecting anomalies using the same threshold between domains, the pAUC has to be calculated for each section, not for each domain. We calculated the AUC for each domain and pAUC for each section as

$$ \text{AUC}_{m,n,d} = \frac{1}{N_d^- N_n^+} \sum_{i=1}^{N_d^-} \sum_{j=1}^{N_n^+} \mathcal{H}(A_{\theta}(x_j^+) - A_{\theta}(x_i^-)), \quad (2) $$

$$ \text{pAUC}_{m,n} = \frac{1}{P_n^- N_n^+} \sum_{i=1}^{P_n^-} \sum_{j=1}^{N_n^+} \mathcal{H}(A_{\theta}(x_j^+) - A_{\theta}(x_i^-)), \quad (3) $$

where $P_n^- = \lfloor pN_n^- \rfloor$, $m$ represents the index of a machine type, $n$ represents the index of a section, $d = \{\text{source, target}\}$ represents a domain, $\lfloor \cdot \rfloor$ is the flooring function, and $\mathcal{H}(x)$ returns 1 when $x > 0$ and 0 otherwise. Here, $\{A_{\theta}(x_i^-)\}$ and $\{A_{\theta}(x_j^+)\}$ are sets of anomaly scores of normal and anomalous test clips, ordered in descending power, respectively. $N_d^-$ is the number of normal test clips in domain $d$, $N_n^+$ and $N_n^-$ are the number of normal and anomalous test clips in section $n$, respectively. We calculated AUC$_{m,n,d}$ to evaluate the contribution of each domain to AUC$_{m,n}$, as it holds that $\text{AUC}_{m,n} = \sum_d \text{AUC}_{m,n,d}$ if $N_{\text{source}}^- = N_{\text{target}}^-$.

The official score $\Omega$ for ranking submitted systems is given by the harmonic mean of the AUC and pAUC scores over all machine types and sections as follows:

$$ \Omega = h\{\text{AUC}_{m,n,d}, \text{pAUC}_{m,n} \mid m \in \mathcal{M}, n \in \mathcal{S}(m), d \in \{\text{source, target}\}\}, \quad (4) $$

where $h\{\cdot\}$ represents the harmonic mean (over all machine types, sections, and domains), $\mathcal{M}$ represents the set of machine types, and $\mathcal{S}(m)$ represents the set of sections for machine type $m$.

Participants are required to submit the anomaly score and normal/anomaly decision result of each test clip. Even though the official score can be calculated with only the anomaly scores, decision results are also required because we must determine the threshold in real-world applications.

## 3.3. Baseline systems and results

The task organizers provide an autoencoder (AE)-based and a MobileNetV2-based baseline systems.

The AE-based system calculates the anomaly score as the reconstruction error of the sound. To determine the threshold, we assume that anomaly scores of normal sound follows a gamma distribution. The parameters of the gamma distribution are estimated from the anomaly scores of normal sound in the training data, and the threshold is calculated by the 90th percentile of the gamma distribution. A test clip is determined to be anomalous if its anomaly score exceeds the threshold.

In the MobileNetV2-based system [12–14], classifiers such as the MobileNetV2 [15] are trained to identify from which section the observed signal was generated. The anomaly score is calculated as the averaged negative logit of the predicted probabilities for the correct section. The threshold is calculated in the same manner as in the AE-based baseline.

Tables 1 and 2 show the AUC and pAUC for the two baselines. Because the results produced with a GPU are generally non-deterministic, the average and standard deviations from five independent trials are also shown in the tables.

<table><caption>Table 1: Results of the AE-based baseline</caption><thead><tr><th rowspan="2">Section</th><th colspan="2">AUC [%]</th><th rowspan="2">pAUC [%]</th></tr><tr><th>Source</th><th>Target</th></tr></thead><tbody><tr><td>00</td><td>86.42 ± 1.10</td><td>41.48 ± 6.11</td><td>51.31 ± 1.34</td></tr><tr><td>01</td><td>89.85 ± 1.39</td><td>41.93 ± 5.36</td><td>54.08 ± 1.84</td></tr><tr><td>02</td><td>98.84 ± 0.52</td><td>26.50 ± 13.52</td><td>52.79 ± 1.04</td></tr><tr><td>00</td><td>67.54 ± 0.97</td><td>33.68 ± 3.12</td><td>52.72 ± 1.63</td></tr><tr><td>01</td><td>79.32 ± 0.82</td><td>29.87 ± 5.62</td><td>50.64 ± 2.33</td></tr><tr><td>02</td><td>84.08 ± 0.38</td><td>15.52 ± 14.90</td><td>48.33 ± 2.33</td></tr><tr><td>00</td><td>67.85 ± 19.61</td><td>60.17 ± 7.24</td><td>54.41 ± 5.72</td></tr><tr><td>01</td><td>59.67 ± 12.67</td><td>64.65 ± 12.63</td><td>55.09 ± 3.36</td></tr><tr><td>02</td><td>61.71 ± 33.52</td><td>60.55 ± 35.10</td><td>64.18 ± 19.79</td></tr><tr><td>00</td><td>84.69 ± 1.74</td><td>39.35 ± 9.35</td><td>59.95 ± 2.00</td></tr><tr><td>01</td><td>71.69 ± 0.69</td><td>44.74 ± 1.79</td><td>51.12 ± 0.55</td></tr><tr><td>02</td><td>80.54 ± 1.42</td><td>63.49 ± 2.36</td><td>62.88 ± 1.55</td></tr><tr><td>00</td><td>64.63 ± 0.88</td><td>64.79 ± 1.06</td><td>60.93 ± 2.31</td></tr><tr><td>01</td><td>67.66 ± 0.51</td><td>58.12 ± 0.38</td><td>53.74 ± 0.56</td></tr><tr><td>02</td><td>75.38 ± 0.75</td><td>65.57 ± 0.82</td><td>61.51 ± 0.69</td></tr><tr><td>00</td><td>81.92 ± 0.81</td><td>58.04 ± 1.22</td><td>61.65 ± 1.22</td></tr><tr><td>01</td><td>67.85 ± 0.53</td><td>50.30 ± 1.25</td><td>53.06 ± 0.53</td></tr><tr><td>02</td><td>86.66 ± 0.39</td><td>38.78 ± 5.13</td><td>53.44 ± 1.18</td></tr><tr><td>00</td><td>54.24 ± 0.68</td><td>52.73 ± 1.93</td><td>52.15 ± 0.25</td></tr><tr><td>01</td><td>50.45 ± 3.67</td><td>53.01 ± 1.73</td><td>49.78 ± 0.19</td></tr><tr><td>02</td><td>51.56 ± 2.89</td><td>43.84 ± 1.11</td><td>49.24 ± 0.65</td></tr></tbody></table>

<table><caption>Table 2: Results of the MobileNetV2-based baseline</caption><thead><tr><th rowspan="2">Section</th><th colspan="2">AUC [%]</th><th rowspan="2">pAUC [%]</th></tr><tr><th>Source</th><th>Target</th></tr></thead><tbody><tr><td>00</td><td>47.40 ± 7.22</td><td>56.40 ± 4.11</td><td>49.96 ± 2.56</td></tr><tr><td>01</td><td>62.02 ± 11.07</td><td>56.38 ± 11.31</td><td>50.92 ± 2.52</td></tr><tr><td>02</td><td>74.19 ± 7.94</td><td>45.64 ± 11.32</td><td>56.51 ± 6.07</td></tr><tr><td>00</td><td>46.02 ± 12.21</td><td>49.41 ± 15.14</td><td>50.25 ± 1.49</td></tr><tr><td>01</td><td>71.96 ± 5.72</td><td>45.14 ± 13.66</td><td>52.97 ± 4.61</td></tr><tr><td>02</td><td>63.23 ± 25.60</td><td>44.34 ± 21.50</td><td>51.54 ± 4.34</td></tr><tr><td>00</td><td>67.85 ± 19.61</td><td>60.17 ± 7.24</td><td>54.41 ± 5.72</td></tr><tr><td>01</td><td>59.67 ± 12.67</td><td>64.65 ± 12.63</td><td>55.09 ± 3.36</td></tr><tr><td>02</td><td>61.71 ± 33.52</td><td>60.55 ± 35.10</td><td>64.18 ± 19.79</td></tr><tr><td>00</td><td>71.07 ± 19.84</td><td>62.13 ± 12.50</td><td>55.40 ± 11.29</td></tr><tr><td>01</td><td>76.26 ± 4.95</td><td>35.12 ± 13.38</td><td>52.14 ± 4.08</td></tr><tr><td>02</td><td>67.29 ± 10.34</td><td>58.02 ± 7.46</td><td>65.14 ± 1.09</td></tr><tr><td>00</td><td>63.54 ± 9.46</td><td>67.02 ± 13.50</td><td>62.12 ± 11.66</td></tr><tr><td>01</td><td>66.68 ± 12.29</td><td>66.96 ± 8.92</td><td>56.85 ± 4.47</td></tr><tr><td>02</td><td>80.87 ± 7.85</td><td>43.15 ± 16.12</td><td>50.62 ± 7.73</td></tr><tr><td>00</td><td>87.15 ± 2.71</td><td>80.77 ± 4.53</td><td>71.57 ± 5.28</td></tr><tr><td>01</td><td>49.66 ± 30.46</td><td>32.07 ± 46.84</td><td>48.21 ± 2.73</td></tr><tr><td>02</td><td>72.70 ± 11.67</td><td>32.94 ± 19.77</td><td>49.69 ± 1.63</td></tr><tr><td>00</td><td>75.26 ± 4.84</td><td>43.60 ± 14.38</td><td>55.37 ± 5.86</td></tr><tr><td>01</td><td>54.78 ± 5.37</td><td>60.43 ± 5.08</td><td>54.69 ± 3.87</td></tr><tr><td>02</td><td>76.26 ± 1.02</td><td>78.74 ± 2.64</td><td>85.74 ± 0.08</td></tr></tbody></table>

# 4. CHALLENGE RESULTS

## 4.1. Results for evaluation dataset

We received 81 submissions from 31 teams, and 22 teams outperformed the MobileNetV2-based baseline in the official score. In Figure 1, the harmonic means of the AUCs are shown for top 10 teams [16–25]. Although the AUCs change drastically between different machine types and teams, these highly ranked teams outperformed the baselines for most of the machine types. It is worth noting that, for these teams, the source-domain AUC did not correlate with the official rank (correlation coefficient was −0.033) while the target-domain AUC did (correlation coefficient was −0.862). This indicates that handling domain shifts and generalizing the model was the key to better ranks among highly ranked teams.

We find that domain generalization approaches adopted by the participants can be categorized into two types: domain-mixing-based approach and domain-classification-based approach. These methods achieved the aim of the task by generalizing the model using data with different attributes. Figure 2 shows the average source-domain and target-domain AUC of the top 20 teams.
<figure><img src="image_33.png" /><figcaption>Figure 1: Evaluation results of top 10 teams in the ranking. Average source-domain AUC (Top) and target-domain AUC (bottom) for each machine type. Label “A” and “M” on the x-axis denote AE-based and MobileNetV2-based baselines, respectively.</figcaption></figure>

<figure><img src="image_34.png" /><figcaption>Figure 2: Average source-domain AUC and target-domain AUC of the top 20 teams. “classification” denotes teams that used domain-classification-based approaches, “mix-up” denotes teams that used domain-mixing-based approaches, and “none” denotes teams that did not use particular domain generalization techniques.</figcaption></figure>

Domain-classification-based approaches outperformed other approaches especially for the target domain. However, these approaches may be specialized for the types of target domain data provided in both the training and test data, and thus may not perform well for those not included in the train data. We describe the details in the following.

## 4.2. Domain-mixing-based approach

Domain-mixing-based approach extracts common representations between domains. These include batch mixing that use data from both domains in a batch to train a model [17, 23], Mixup [26] that synthesizes data from both domains to obtain intermediate representations [17, 23, 25, 27], and data augmentation techniques to obtain robust representations [24]. These techniques use the target domain data to expand the normal conditions for the model so that the model can be generalized to better handle domain shifts. However, as shown in Figure 2, they have been outperformed by domain-classification-based approaches. This can be that, for the Mixup and data augmentation, synthesized data was not useful for representing the target domain data. One future direction can be on obtaining meaningful synthetic representations with the aid of external information such as the attribute information.

## 4.3. Domain-classification-based approach

Domain-classification-based approach distinguishes the source and target domain data to obtain better detection performance for each domain. The 1st and 6th place teams [16, 21] used distances between the embedding of a domain and that of the test data to calculate anomaly scores. Because the domain of each sample can be estimated by the domain with shorter distance, this approach can be regarded as implicitly classifying the domain of each sample. The 2nd, 3rd, 4th, and 5th place teams [17–20] explicitly trained a classifier to distinguish the attributes or the domains. The 5th place teams trained an attribute classifier and a section classifier so that both the domain-wise information from the attribute classifier and the domain-independent information from the section classifier can be obtained.

As shown in Figure 2, the domain-classification-based approach outperformed the domain-mixing-based approach. This can be because the normal conditions are defined for each specific domain, unlike the domain-mixing-based approach that defines normal conditions over all domains. However, this approach assumes that the target domain data in the train data includes all types of the target domain data in the test data. If the target domain data in the test data contains too many types of data not included in the train data, the classifier may fail to distinguish domains, which can degrade the detection performance. Therefore, further investigation is needed to examine the ability of this approach to handle completely unseen target domain data.

# 5. CONCLUSION

This paper presented an overview of the task and analysis of the solutions submitted to DCASE 2022 Challenge Task 2. To handle domain shifts that occur implicitly, the task was dedicated to developing domain generalization techniques. The organization of the task revealed two approaches that can be useful for domain generalization task: domain-mixing-based approach and domain-classification-based approach. For the former approach, obtaining more meaningful synthetic representations from multiple domains is left for future works. For the latter approach, future works can focus on analyzing the effect of this approach on completely unseen types of target domain data.