# SOUND EVENT LOCALIZATION AND DETECTION WITH PRE-TRAINED AUDIO SPECTrogram TRANSFORMER AND MULTICHANNEL SEPARATION NETWORK

Robin Scheibler, Tatsuya Komatsu, Yusuke Fujita, Michael Hentschel

LINE Corporation, Tokyo, Japan

## ABSTRACT

We propose a sound event localization and detection system based on a CNN-Conformer base network. Our main contribution is to evaluate the use of pre-trained elements in this system. First, a pre-trained multichannel separation network allows to separate overlapping events. Second, a fine-tuned self-supervised audio spectrogram transformer provides a priori classification of sound events in the mixture and separated channels. We propose three different architectures combining these extra features into the base network. We first train on the STARSS22 dataset extended by simulation using events from FSD50K and room impulse responses from previous challenges. To bridge the gap between the simulated dataset and the STARSS22 dataset, we fine-tune the models on the training part of the STARSS22 development dataset only before the final evaluation. Experiments reveal that both the pre-trained separation and classification models enhance the final performance, but the extent depends on the adopted network architecture.

**Index Terms**— SELD, 3D CNN, Conformer, Audio Spectrogram Transformer, Separation

## 1. INTRODUCTION

Sound event localization and detection (SELD) combines both sound event detection (SED) and direction of arrival (DOA) estimation from multichannel recordings into a single task [1]. The task has been part of the DCASE challenge<fnref n="1" /> since 2019. While both tasks are fairly well understood, their combination is made challenging by event polyphony, moving sources, imbalance in the duration of events, interfering events, and an overall training data scarcity. Two types of multichannel recordings have been made available, both derived from 32 channel recordings done with the Eigenmike rigid spherical microphone array<fnref n="2" />. A tetrahedral subset of four channels of the Eigenmike (MIC), and the first order ambisonics (FOA) coefficients derived from all 32 channels. Both formats have four channels and can be used separately or together in the challenge.

Due to the difficulty of the task, all neural solutions have been broadly adopted in the DCASE challenge submissions. A variety of input features have been proposed: generalized cross-correlation, inter-aural level and time differences, intensity vectors [2], per-channel energy normalization [3], SALSA-lite [4], to mention a few. For a detailed list, see [5]. SELD is a data-poor task and augmentations have been a crucial component of past winning systems. One successful strategy is to create new data by convolution of sound event samples with real [6] and simulated [7] impulse responses.

To prevent the network overfitting to some directions, the symmetries of the recording system have been used to increase the diversity of DOA angles by artificially rotating the data [8]. A variety of network architectures have been proposed. Many are derived from the convolutional recurrent networks used in the challenge baselines [5]. Many recent solutions have replaced or complemented the recurrent layers by self-attention [9]. The event-independent network architecture [10] proposes to decouple the SED and DOA tasks using separate networks with soft stitching between layers. The 2022 challenge is characterized by the new STARSS22 dataset of real sound scenes played by actors [11]. Due to the high quality of the data set, only 4.9 h of recordings are available. One of the avowed objective of this year's challenge is to explore the use of external resources to compensate the lack of training data.

Our submission to the DCASE Challenge Task 3 relies on two unrelated advancements in the field. First, high quality pre-trained and self-supervised audio representation have become recently widely available. As examples, see wav2vec [12], PANN [13], and the self supervised audio spectrogram transformer (SSAST) [14]. Second, multichannel source separation based on independent vector analysis (IVA) has been shown to improve sound event detection [15]. Our proposed solution uses the FOA format as it is free of spatial aliasing up to 9 kHz. We use a multichannel separation model trained in advance to coarsely separate the input signal by directions. The separation algorithm is independent vector analysis with a neural source model [16]. Then, we use a pre-trained SSAST [14] fine-tuned on the Task 3 dataset to predict events in the FOA omni channel and the four separation output channels. However, these predictions lack spatial information so we combine them with a dedicated CNN-Conformer network. The inputs of this network are the log-mel spectrograms of the FOA channels and the intensity vector [2]. The CNN creates useful feature maps which are further processed by an eight layer conformer-encoder. We explore three different architectures introducing the SSAST predictions at different points of this network. We find that it is most effective to introduce the SSAST features both at the input and output of the conformer block. The proposed systems are illustrated in Fig. 1.

## 2. PROPOSED SELD NETWORK

### 2.1. Features

The input data to our SELD network are the four channels first order ambisonics (FOA) signals. First, to help with recognition of events, we run the FOA into a separation network that roughly separates the different events. The separation network is based on independent vector analysis [17] with a neural source model [16] described in Section 2.1.1. We obtain four tracks out of the separation network. Second, these four tracks as well as the omni channel of the FOA are run through a fine-tuned Self-Supervised Audio Spectrogram

<footnote n="1">¹https://dcase community</footnote>

<footnote n="2">²https://mhacoustics.com/products</footnote>
<figure><img src="image_103.png" /><figcaption>Figure 1: Structure of the proposed systems. Blue blocks have trainable parameters. Green blocks have been pre-trained. Gray blocks are not trainable. The (a) base system and the three extensions proposed (b), (c), and (d). HEAD is either a linear or MLP layer.</figcaption></figure>

Transformer (SSAST) described in Section 2.1.2. Third, the logmel-spectrograms of the four FOA channels as well as the intensity vector (IV) [2], as used in many SELD systems, provide the spatial information needed for DOA estimation. We use 128 bands for the mel-spectrogram analysis.

### 2.1.1. Separation Network

The multichannel separation network consists of a blind dereverberation part using weighted prediction error (WPE) [18], followed by independent vector analysis (IVA) [19, 20]. For WPE, the STFT uses an FFT size of 512 with 3/4-overlap and a Hann window. The number of iterations, delays, and taps is 3, 3, and 10, respectively. For IVA, the STFT uses an FFT size of 2048 with 3/4-overlap and a Hann window. The IVA algorithm used is iterative source steering [17] with a neural source model [16]. The number of IVA iterations is 20 and we use demixing matrix checkpointing [21] to save memory. The neural source model uses three 1D convolutional layers with GLU non-linearities and batch normalization with four groups. The hidden dimension is 128 which we map back to the STFT size by a 1D transposed convolution layer. Finally, a sigmoid non-linearity produces a mask-like signal from the network's output. A system description of the IVA separation and neural source models are shown in Fig. 2.

Since we do not have access to the ground-truth separated signals for the SELD datasets, we cannot use the conventional source separation loss functions, e.g., SI-SDR or CI-SDR. However, we have access to the direction of arrival of the events so that we can use a recently proposed spatial loss [22]. To train the network, we cut the input data into blocks of 5 s and use the median DOA as target because IVA assumes the sources to be static in this interval.

### 2.1.2. Self-Supervised Audio Spectrogram Transformer

The Self-Supervised Audio Spectrogram Transformer (SSAST) [14] is an all-attention model that has been extensively pre-trained by self-supervision on Audioset [23]. We

fine-tune a pre-trained version of SSAST [24] on the STARSS22 dataset and the baseline extended dataset prepared by the organizers of Task 3. The fine tuning is done for the SED part of the task only. To this end, the DOA information is stripped from the targets and multiple events of the same class are merged together when they appear simultaneously. The SSAST model operates on 5 s blocks and produces class presence prediction vectors (13-dimensional) for each of the 497 frames (approx. 10 ms per frame).

## 2.2. SELD Network

Our proposed SELD system combines a base network with the extra predictions obtained from the separation network and the SSAST. The different system variants are shown in Fig. 1 and their number of parameters given in Table 1.

**Base Network:** The base network is a fairly conventional CNN-Conformer network for SELD using FOA features. We feed the logmel-spectrograms of the four FOA channels and the IV channels into a convolutional network with two layers (total of 7 channels). The first is a 3D convolutional layer where the three dimensions are channels, mel-frequency bands, and time, respectively. We expect that such 3D filters can better capture the directional information present in the input signal. The kernels are of size 7 × 3 × 3 and the padding is (0, 1, 1), which results in a 2D output signal. Thus, the second layer is a 2D convolutional layer with 3 × 3 kernels. Strides of size 2 are used in the frequency and time dimension to reduce the size of the input signal. The number of channels after the 3D convolution is 128. Group normalization with four groups and ReLU activations are used after each layer. After the two strided convolutions, the remaining 32 frequency dimensions are merged with the 128 channels and projected to dimension 128 by a linear layer before the output. The output of this stage is an embedding signal with 128 dimensions and a frame interval of 40 ms. This output is fed into a conformer-encoder [25] with eight layers and convolution kernel size 7. The embedding vectors so created are then projected by an output head. We explore both a simple linear projection and a two layer multi-layer perceptron (MLP). The MLP uses a GeLU non-
<figure><img src="image_104.png" /><figcaption>Figure 2: The structure of the separation network used to obtain the input features.</figcaption></figure>

<table><caption>Table 1: List of models and their number of parameters. Numbers reported include the MLP output head.</caption><thead><tr><th>Model</th><th>SSAST</th><th>IVA</th><th>Base</th><th>I</th><th>II</th><th>III</th></tr></thead><tbody><tr><td>Params.</td><td>87.2 M</td><td>2.4 M</td><td>3.8 M</td><td>3.8 M</td><td>3.9 M</td><td>15.3 M</td></tr></tbody></table>

linearity. The final non-linearity is a hyperbolic tangent to limit the output to the [-1, 1] range. The output is in the Multi-ACCDOA format [26] with 4 tracks, thus the output size is 4 tracks × 3 dimension of Cartesian DOA vectors × 13 classes, a total of 156 outputs per time frame. The event presence probability is given by the length of the 3D vector for each track/class slot. The base network is shown in Fig. 1a.

**Architecture I:** Our first variant combines all the features at the input of the CNN (Fig. 1b). The 5 channels of SSAST predictions are aggregated by taking the max. Then, they are broadcasted to match the 128 bands of the log-mel spectrogram and IV features to which they are concatenated. Thus, the input of the CNN has 20 channels and 128 mel-frequency bands. The rest of the network is similar to the base network.

**Architecture II:** Here we concatenate the SSAST predictions to the output of the conformer, before the output head (Fig. 1c). We project the SSAST prediction vectors of the omni FOA channel and the 4 IVA output channels (see Section 2.1.1) from 13 to 128 dimensions by a linear projection followed by ReLU activations. After this, these five channels are averaged into one. The frame rate is adjusted to that of the spatial feature extraction network by average pooling of size four along the time axis. The embedding obtained is concatenated to the output of the conformer to obtain an embedding of size 256. Finally, a linear layer projects this concatenated embedding to the output size.

**Architecture III:** This variant combines the SSAST predictions to both the input and the output of the conformer (Fig. 1d). The 5 channels and 13 classes dimensions are reshaped to size 65 and projected to size 51 by a linear layer. The output of the CNN is linearly projected to size 205 and concatenated to obtain an embedding of size 256 which is fed to the conformer-encoder. Another linear layer projects the 65-dimensional pre-trained features to size 256, which is concatenated to the output of the conformer layers to obtain a 512-dimensional embedding fed to the output head. This variant uses an embedding size of 256 both for the CNN and conformer (unlike 128 for all other networks), resulting in a larger network.

<table><caption>Table 2: The datasets used. Columns “Ov.” and “Inter.” indicate the maximum number of overlapping event, and the number of interfering out-of-classes events. Rec. and Sim. stand for “recorded” and “simulated”, respectively.</caption><thead><tr><th>Name</th><th>Ref</th><th>Type</th><th>Ov.</th><th>Inter.</th><th>Train</th><th>Val.</th></tr></thead><tbody><tr><td>STARSS22</td><td>[11]</td><td>Rec.</td><td>5</td><td>✓</td><td>2.9 h</td><td>2.0 h</td></tr><tr><td>Synth1</td><td>[27]</td><td>Sim.</td><td>2</td><td>0</td><td>20 h</td><td>—</td></tr><tr><td>Synth2</td><td></td><td>Sim.</td><td>4</td><td>1</td><td>20 h</td><td>—</td></tr></tbody></table>

<table><caption>Table 3: List of external datasets and models used</caption><thead><tr><th>Name</th><th>Ref.</th><th>Type</th></tr></thead><tbody><tr><td>STARSS22</td><td>[11]</td><td>DCASE2022 task 3 dataset</td></tr><tr><td>FSD50K</td><td>[28, 29]</td><td>audio dataset</td></tr><tr><td>TAU-SRIR DB</td><td>[6, 30]</td><td>RIR dataset</td></tr><tr><td>SSAST</td><td>[14, 24]</td><td>pre-trained pytorch model</td></tr></tbody></table>

## 2.3. Post-processing

The post-processing works in two steps. Let $q_{ntc}$ be the output of the nth frame, th track, and cth class. The event probability is taken to be $p_{ntc} = ||q_{ntc}||$. First, events are detected if $p_{ntc} \ge \sigma_c$ at the output framerate of the network. We run a de-duplication procedure to remove duplicate events produced by the Multi-ACCDOA format. Events from different tracks of the same class with directions closer than $\theta_c$, a class specific threshold, are merged together. Second, all the events from the same output frame are aggregated together. Because the output frames of the network are 40 ms and the target frames are 100 ms, there are 2 or 3 events per output frame, track, and class. For every output frame and class, we find the event with largest $p_{ntc}$ and count all events within $\theta_c$. If the count is larger than $\eta_c$, we declare an event with direction given by the average of all aggregated events, weighted by their probability. By default, we use $\sigma_c = 1/2$, $\theta_c = 15^{\circ}$, and $\eta_c = 1$. To maximize performance, we use a post-processing calibration procedure where $\sigma_c$, $\theta_c$, and $\eta_c$ are chosen per class to minimize the SELD score on the validation dataset of STARSS22.

## 2.4. Differences with the Challenge Submission

Our challenge submission was based on Architecture II with a linear output head. However, after the end of the challenge, we found a mistake in our use of the pre-trained SSAST, namely, the output layers were not initialized to the correct size (527 instead of 13 dimensional vectors) and weights. Furthermore the weights used on the development and evaluation datasets were different. We have corrected and retrained all the architectures for this paper.

# 3. DATASET AND TRAINING

## 3.1. Datasets

We use the three datasets described in Table 2 with a total of 42.9 h and 2.0 h of training and validation data, respectively. From the DCASE2022 task 3 dataset, STARSS22 [11], fold3 (2.9 h) is used for training and fold4 (2.0 h) for validation, as suggested. Since this is not sufficient, we also use the baseline training synthetic dataset
(Synth1) provided by the task organizers [27]. This dataset is cre-
ated by remixing sound events from the FSD50K dataset [28, 29]
with the measured RIR from the TAU-SRIR database [30, 6]. How-
ever, the dataset Synth1 only contains up to two overlapping events,
and no interfering events. Thus, we use the original recipe pro-
vided for Synth1 [31] to create an extended training set, Synth2.
We change the recipe in the following ways. First, increase the
maximum number of overlapping events from 2 to 4. Second,
we add interfering sound events not included in the classifica-
tion task. For the interference, we select clips from the following
categories of FSD50K: Cutlery, silverware, Computer,
keyboard, Chewing, mastication, Buzz, Crumpling,
crinkling, Typing, Clock, Meow, Breathing, Glass,
Writing, Chink, clink. The base external datasets and pre-
trained models used are summarized in Table 3 and the training
datasets in Table 2, respectively.

3.2. Data Augmentations

*SpecAugment:* We apply SpecAugment [32] using the same mask to all FOA channels prior to computation of mel-spectrogram and IV during training. The maximum time masking is 2% of the total length, while frequency masking is up to 10%.

*Random Rotations:* To avoid the network over-fitting to specific directions, we apply random rotations to the FOA input, as has been successfully used for SELD networks in previous challenges [8]. By applying the same rotation to the targets, we are able to simulate large spatial variations in the input dataset. This augmentation is applied to input examples with probability 1/2.

3.3. Training

We train the network with the recently proposed Multi-ACCDOA
loss [26]. The optimizer is Adam [33] with learning rate 0.001. We
do learning rate warm-up over the first 10000 steps. The network
is trained for 1000 epochs on STARSS22, Synth1, and Synth2
datasets. The progress of the optimization is monitored on the vali-
dation set of STARSS22 using the SELD score,

$$
\mathrm{SELD}=0.25\left(\mathrm{ER}+(1-\mathrm{F})+\mathrm{LE} / 180+(1-\mathrm{LR})\right),
$$

where ER, F, LE, LR, are the official SELD metrics [1]. Af-
ter training finishes, we fine-tune the network on the training part
(fold3) of STARSS22 only. We freeze all layers except the output
head. Training is restarted from the average of the 10 checkpoints
with lowest SELD score with learning rate 0.0001. We proceed for
1000 epochs. Finally, we select the 10 checkpoints with the lowest
validation score and average their weights.

4. EXPERIMENTS

We do an ablation study to assess the contribution of the different
components. Our reference is the base network with linear output
head, trained without fine tuning (Fig. 1a). For each architecture we
add in order SSAST predictions of FOA omni channel (+AST), and
four separation outputs (+IVA), MLP output head (+MLP), fine-
tuning (+FINE), and post-processing calibration (+POST).

Table 4 shows the results on the validation part of the devel-
opment set of STARSS22 (fold4) compared to that of the baseline
system [34]. Our first observation is that all the proposed architec-
tures improve significantly over the baseline [34]. In addition, we
see that how we insert the SSAST/IVA features into the network

<table><caption>Table 4: SELD metrics of proposed architectures and baseline [34].</caption>
  <thead>
    <tr>
      <th>Model</th>
      <th>ER↓</th>
      <th>F↑</th>
      <th>LE↓</th>
      <th>LR↑</th>
      <th>SELD ↓</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Baseline (FOA) [34]</td>
      <td>0.71</td>
      <td>0.21</td>
      <td>29.3</td>
      <td>0.46</td>
      <td>0.5507</td>
    </tr>
    <tr>
      <td>Base Network</td>
      <td></td>
      <td></td>
      <td></td>
      <td></td>
      <td></td>
    </tr>
    <tr>
      <td>+MLP</td>
      <td>0.578</td>
      <td>0.421</td>
      <td>19.083</td>
      <td>0.602</td>
      <td>0.4154</td>
    </tr>
    <tr>
      <td>+FINE</td>
      <td>0.561</td>
      <td>0.451</td>
      <td>16.314</td>
      <td>0.563</td>
      <td>0.4094</td>
    </tr>
    <tr>
      <td>+POST</td>
      <td>0.535</td>
      <td>0.464</td>
      <td><strong>15.869</strong></td>
      <td>0.562</td>
      <td>0.3994</td>
    </tr>
    <tr>
      <td>Architecture I</td>
      <td></td>
      <td></td>
      <td></td>
      <td></td>
      <td></td>
    </tr>
    <tr>
      <td>+AST</td>
      <td>0.575</td>
      <td>0.423</td>
      <td>18.752</td>
      <td>0.591</td>
      <td>0.4164</td>
    </tr>
    <tr>
      <td>+IVA</td>
      <td>0.574</td>
      <td>0.418</td>
      <td>17.809</td>
      <td>0.582</td>
      <td>0.4182</td>
    </tr>
    <tr>
      <td>+MLP</td>
      <td>0.584</td>
      <td>0.455</td>
      <td>17.331</td>
      <td>0.606</td>
      <td>0.4050</td>
    </tr>
    <tr>
      <td>+FINE</td>
      <td>0.562</td>
      <td>0.469</td>
      <td>16.881</td>
      <td>0.616</td>
      <td>0.3928</td>
    </tr>
    <tr>
      <td>+POST</td>
      <td>0.519</td>
      <td>0.480</td>
      <td>16.375</td>
      <td>0.598</td>
      <td>0.3830</td>
    </tr>
    <tr>
      <td>Architecture II</td>
      <td></td>
      <td></td>
      <td></td>
      <td></td>
      <td></td>
    </tr>
    <tr>
      <td>+AST</td>
      <td>0.572</td>
      <td>0.424</td>
      <td>18.130</td>
      <td>0.604</td>
      <td>0.4111</td>
    </tr>
    <tr>
      <td>+IVA</td>
      <td>0.589</td>
      <td>0.414</td>
      <td>18.016</td>
      <td>0.611</td>
      <td>0.4160</td>
    </tr>
    <tr>
      <td>+MLP</td>
      <td>0.592</td>
      <td>0.445</td>
      <td>18.156</td>
      <td><strong>0.641</strong></td>
      <td>0.4020</td>
    </tr>
    <tr>
      <td>+FINE</td>
      <td>0.534</td>
      <td>0.478</td>
      <td>17.163</td>
      <td>0.595</td>
      <td>0.3891</td>
    </tr>
    <tr>
      <td>+POST</td>
      <td>0.516</td>
      <td>0.497</td>
      <td>16.551</td>
      <td>0.603</td>
      <td>0.3768</td>
    </tr>
    <tr>
      <td>Architecture III</td>
      <td></td>
      <td></td>
      <td></td>
      <td></td>
      <td></td>
    </tr>
    <tr>
      <td>+AST</td>
      <td>0.579</td>
      <td>0.417</td>
      <td>18.785</td>
      <td>0.607</td>
      <td>0.4147</td>
    </tr>
    <tr>
      <td>+IVA</td>
      <td>0.572</td>
      <td>0.437</td>
      <td>17.957</td>
      <td>0.621</td>
      <td>0.4037</td>
    </tr>
    <tr>
      <td>+MLP</td>
      <td>0.567</td>
      <td>0.460</td>
      <td>18.294</td>
      <td>0.616</td>
      <td>0.3980</td>
    </tr>
    <tr>
      <td>+FINE</td>
      <td>0.551</td>
      <td>0.493</td>
      <td>17.505</td>
      <td>0.639</td>
      <td>0.3792</td>
    </tr>
    <tr>
      <td>+POST</td>
      <td><strong>0.500</strong></td>
      <td><strong>0.514</strong></td>
      <td>17.131</td>
      <td>0.624</td>
      <td><strong>0.3644</strong></td>
    </tr>
  </tbody>
</table>

matters. Architecture I concatenates the extra features at the input
of the network, which might make training the entire network more
difficult. The MLP head is important here to obtain the best per-
formance. For architecture II, where the features are only inserted
at the end, the MLP is also required to fully take advantage of the
extra information. Architecture III performs best by considering
features both in the conformer, and again in the output head. There,
the benefits of SSAST and IVA features are clearly visible. Of the
three, it performs best, but is also the largest model. However, the
better performance is not due to size only since it does not outper-
form I and II for +AST. In all cases, fine-tuning and post-processing
calibration are necessary to maximize performance.

5. CONCLUSION

We have presented three ways of using a pre-trained SSAST and separation system to improve SELD. An ablation study demonstrated the effectiveness of the different components. We found that inserting the pre-trained predictions both before and after the conformer encoder, combined with an MLP output classification head is most effective.

6. REFERENCES

[1] A. Politis, A. Mesaros, S. Adavanne, T. Heittola, and T. Virenan, “Overview and evaluation of sound event localization and detection in dcase 2019,” *IEEE/ACM Trans. Audio Speech Lang. Process.*, vol. 29, pp. 684–698, 2020.
[2] K. Lopatka, J. Kotus, and A. Czyzewski, "Detection, classification and localization of acoustic events in the presence of background noise for acoustic surveillance of hazardous situations," Multimedia Tools and Applications, vol. 75, no. 17, pp. 10407-10439, 2016.

[3] V. Lostanlen et al., "Per-Channel Energy Normalization: Why and How," IEEE Signal Process. Lett., vol. 26, no. 1, pp. 39-43, Jan. 2019.

[4] T. N. Tho Nguyen, D. L. Jones, K. N. Watcharasupat, H. Phan, and W.-S. Gan, "SALSA-Lite: A fast and effective feature for polyphonic sound event localization and detection with microphone arrays," in Proc. IEEE ICASSP, Singapore, SG, May 2022.

[5] S. Adavanne, A. Politis, J. Nikunen, and T. Virtanen, "Sound event localization and detection of overlapping sources using convolutional recurrent neural networks," IEEE J. Sel. Top. Signal Process., vol. 13, no. 1, pp. 34-48, Apr. 2019.

[6] A. Politis, S. Adavanne, and T. Virtanen, "A dataset of reverberant spatial sound scenes with moving sources for sound event localization and detection," in Proc. DCASE, Tokyo, JP, Nov. 2020.

[7] Y. Koyama et al., "Spatial data augmentation with simulated room impulse responses for sound event localization and detection," in Proc. IEEE ICASSP, Singapore, SG, May 2022, pp. 8872-8876.

[8] F. Ronchini, D. Arteaga, and A. Pérez-López, "Sound event localization and detection based on crnn using rectangular filters and channel rotation data augmentation," in Proc. DCASE2020, Tokyo, JP, Nov. 2020.

[9] K. Shimada et al., "Ensemble of ACCDOA- and EINV2-based Systems with D3Nets and Impulse Response Simulation for Sound Event Localization and Detection," DCASE Challenge, Tech. Rep., June 2021.

[10] Y. Cao, T. Iqbal, Q. Kong, F. An, W. Wang, and M. D. Plumbley, "An Improved Event-Independent Network for Polyphonic Sound Event Localization and Detection," in Proc. IEEE ICASSP, Toronto, CA, June 2021, pp. 885-889.

[11] A. Politis et al., "STARSS22: A dataset of spatial recordings of real scenes with spatiotemporal annotations of sound events," arXiv preprint arXiv:2206.01948, 2022.

[12] S. Schneider, A. Baevski, R. Collobert, and M. Auli, "wav2vec: Unsupervised pre-training for speech recognition," in Proc. Interspeech, Graz, AU, Sept. 2019, pp. 3465-3469.

[13] Q. Kong, Y. Cao, T. Iqbal, Y. Wang, W. Wang, and M. D. Plumbley, "PANNs: Large-scale pretrained audio neural networks for audio pattern recognition," IEEE/ACM Trans. Audio Speech Lang. Process., vol. 28, pp. 2880-2894, Oct. 2020.

[14] Y. Gong, C.-I. J. Lai, Y.-A. Chung, and J. Glass, "SSAST: Self-supervised audio spectrogram transformer," arXiv preprint arXiv:2110.09784, 2021.

[15] R. Scheibler, T. Komatsu, and M. Togami, "Multichannel separation and classification of sound events," in Proc. EURIPCO, Dublin, IE, Aug. 2021, pp. 1035-1039.

[16] R. Scheibler and M. Togami, "Surrogate source model learning for determined source separation," in Proc. IEEE ICASSP, Toronto, CA, June 2021, pp. 176-180.

[17] R. Scheibler and N. Ono, "Fast and stable blind source separation with rank-1 updates," in Proc. IEEE ICASSP, Barcelona, ES, May 2020, pp. 236-240.

[18] T. Yoshioka and T. Nakatani, "Generalization of Multi-Channel Linear Prediction Methods for Blind MIMO Impulse Response Shortening," IEEE Trans. Audio Speech Lang. Process., vol. 20, no. 10, pp. 2707-2720, 2012.

[19] A. Hiroe, "Solution of permutation problem in frequency domain ica, using multivariate probability density functions," in ASIACRYPT 2016. Springer Berlin Heidelberg, Jan. 2006, vol. 3889, pp. 601-608.

[20] T. Kim, T. Eltoft, and T.-W. Lee, "Independent vector analysis: An extension of ICA to multivariate components," in ASIACRYPT 2016. Springer Berlin Heidelberg, Jan. 2006, vol. 3889, pp. 165-172.

[21] K. Saijo and R. Scheibler, "Independence-based joint dereverberation and separation with neural source model," in Proc. Interspeech, Incheon, KR, Sept. 2022.

[22] ——, "Spatial loss for unsupervised multi-channel source separation," in Proc. Interspeech, Incheon, KR, Sept. 2022.

[23] J. F. Gemmeke et al., "Audio Set: An ontology and human-labeled dataset for audio events," in Proc. IEEE ICASSP, New Orleans, LA, USA, Mar. 2017, pp. 776-780.

[24] Y. Gong et al., "Yuangongnd/ssast." [Online]. Available: https://github.com/YuanGongND/ssast

[25] A. Gulati et al., "Conformer: Convolution-augmented transformer for speech recognition," in Proc. Interspeech, 2020, pp. 5036-5040.

[26] K. Shimada et al., "MULTI-ACCDOA: Localizing and detecting overlapping sounds from the same class with auxiliary duplicating permutation invariant training," in Proc. IEEE ICASSP, Singapore, SG, pp. 316-320.

[27] A. Politis, "[DCASE2022 Task 3] Synthetic SELD mixtures for baseline training," Apr. 2022. [Online]. Available: https://doi.org/10.5281/zenodo.6406873

[28] E. Fonseca et al., "FSD50K: an open dataset of human-labeled sound events," IEEE/ACM Trans. Audio Speech Lang. Process., vol. 30, pp. 829-852, 2022.

[29] E. Fonseca, X. Favory, J. Pons, F. Font, and X. Serra, "FSD50K," Oct. 2020. [Online]. Available: https://doi.org/10.5281/zenodo.4060432

[30] A. Politis, S. Adavanne, and T. Virtanen, "TAU Spatial Room Impulse Response Database (TAU- SRIR DB)," Apr. 2022. [Online]. Available: https://doi.org/10.5281/zenodo.6408611

[31] D. Krause and A. Politis, "danielkrause/dcase2022-data-generator." [Online]. Available: https://github.com/danielkrause/DCASE2022-data-generator

[32] D. S. Park et al., "SpecAugment: A simple data augmentation method for automatic speech recognition," in Proc. Interspeech. ISCA, Sept. 2019, pp. 2613-2617.

[33] D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization," in Proc. ICLR, San Diego, CA, USA, May 2015.

[34] S. Adavanne, "sharathadavanne/seld-dcase2022." [Online]. Available: https://github.com/sharathadavanne/seld-dcase2022
# KNOWLEDGE DISTILLATION FROM TRANSFORMERS FOR LOW-COMPLEXITY ACOUSTIC SCENE CLASSIFICATION

Florian Schmid<fnref n="1" />,², Shahed Masoudian², Khaled Koutini², Gerhard Widmer<fnref n="1" />,²

¹Institute of Computational Perception (CP-JKU),²LIT Artificial Intelligence Lab, Johannes Kepler University Linz, Austria
florian.schmid@jku.at, shahed.masoudian@jku.at, khaled.koutini@jku.at

## ABSTRACT

Knowledge Distillation (KD) is known for its ability to compress large models into low-complexity solutions while preserving high predictive performance. In Acoustic Scene Classification (ASC), this ability has recently been exploited successfully, as underlined by three of the top four systems in the low-complexity ASC task of the DCASE'21 challenge [1] relying on KD. Current KD solutions for ASC mainly use large-scale CNNs or specialist ensembles to derive superior teacher predictions. In this work, we use the *Audio Spectrogram Transformer* model *PaSST*, pre-trained on Audioset, as a teacher model. We show how the pre-trained *PaSST* model can be properly trained downstream on the *TAU Urban Acoustic Scenes 2022 Mobile development dataset* [2] and how to distill the knowledge into a low-complexity CNN student. We study the effect of using teacher ensembles, using teacher predictions on extended audio sequences, and using Audioset as an additional dataset for knowledge transfer. Additionally, we compare the effectiveness of Mixup and Freq-MixStyle to improve performance and enhance device generalization. The described system achieved rank 1 in the Low-complexity ASC Task of the DCASE'22 challenge [3]<fnref n="1" />.

**Index Terms—** Patchout Spectrogram Transformer, Mixup, Freq-MixStyle, Knowledge Distillation

## 1. INTRODUCTION

Acoustic Scene Classification (ASC) aims to build a model that accurately predicts a scene label given an audio recording. Convolutional Neural Networks (CNNs) are well-established models that dominate the field of ASC [1, 4–6]. Low-complexity solutions are of increasing interest, making the ASC systems runnable on edge devices. Model compression techniques to reduce complexity include *Parameter Pruning* [7–9], *designing efficient network architectures* [10, 11] and *Knowledge Distillation (KD)* [12–14]. KD recently showed promising results for ASC, with three out of the top four systems applying KD in the low-complexity ASC task of the DCASE’21 challenge [1].

This paper focuses on distilling knowledge from Patchout FaSt Spectrogram Transformers (PaSST) [15] into low-complexity CNNs, the winning approach of the DCASE’22 low-complexity ASC task [3]. The *TAU Urban Acoustic Scenes 2022 Mobile development dataset* [2] and the low-complexity constraints of the DCASE’22 challenge define an ASC scenario involving various challenges:

C1 Limits on the number of parameters and the number of multiply-accumulate operations.

C2 Scenes are recorded using different recording devices, resulting in a domain shift.

C3 1-second audio snippets are provided compared to 10-second snippets in the ASC task of the DCASE’21 challenge [1]. Shorter audio files contain less information and are more challenging to classify.

The system studied in the following tackles C1 by using KD with PaSST [15] models as a teacher and a low-complexity CNN as a student. Regarding C2, the domain shift caused by different recording devices is counteracted by *Freq-MixStyle* [16], a modified version of *MixStyle* [17], operating on frequency statistics. The 1-second audio snippets described in C3 result from splitting the 10-second snippets of the *TAU Urban Acoustic Scenes 2020 Mobile development dataset* [2] into ten pieces. The provided segment identifiers allow for reassembling the complete 10-second files. We train all models on randomly cropped 1-second pieces from the reassembled 10-second files to increase the diversity in the training data. Reassembling the 10-second files additionally allows for obtaining superior teacher predictions for KD, as described later.

The related work regarding the system’s building blocks is described in Section 2. Section 3 introduces student and teacher architectures and the PaSST downstream training. Section 4 describes how student and teacher are combined in a KD setup and sets the stage for the results presented in Section 5. The paper is concluded in Section 6.

## 2. RELATED WORK

### 2.1. ASC Architectures

In the past years, Convolutional Neural Networks (CNNs) became the most prominent solution to process spectrograms and dominated previous DCASE challenges [1, 4–6]. Restricting the receptive field of CNNs, known as Receptive Field Regularization, was shown to be particularly well suited for ASC tasks [18, 19].

Inspired by Vision Transformers (ViT) [20], transformers capable of processing spectrograms to solve audio tasks have been proposed recently. In this regard, Audio Spectrogram Transformers (AST) [21], pre-trained on computer vision tasks, have been adapted to the audio domain and achieved state-of-the-art results on Audioset [22]. Image [20, 23] and spectrogram [15, 21] transformers extract overlapping patches with a certain stride from the input image and add a positional encoding. Patchout FaSt Spectrogram Transformer (PaSST) [15] disentangles frequency- and time-encodings, simplifying downstream training on shorter audio clips. PaSST additionally introduces *Patchout*, a mechanism that drops parts of the input sequence to improve generalization and reduce

<footnote n="1">¹Source Code: https://github.com/CPJKU/cpjku_dcase22</footnote>
memory and computational complexity in the quadratically scaling attention layers.

## 2.2. Knowledge Distillation in ASC

In KD [12–14], the aim is to compress the knowledge of a possibly large number of complex teacher models into a low-complexity student model without significant performance loss. The student minimizes a weighted sum of hard label loss and distillation loss, as shown in Eq. 1. The distillation loss matches student and teacher predictions based on the Kullback-Leibler divergence between soft targets: $L_{DIST} = D_{KL}(q_{teacher}||q_{student})$. By minimizing the distillation loss, the student learns to mimic the teacher.

$$L_{\text{TOTAL}} = L_{\text{LABEL}} + \lambda L_{\text{DIST}} \quad (1)$$

The soft targets $q$ are created from the logits $z$ by computing $q = \text{softmax}(z/T)$ with a specific temperature $T$. Raising the temperature $T$ creates a softer distribution and allows the student to exploit the rich similarity information between classes predicted by the teacher compared to the hard labels [12].

In the low-complexity task of the DCASE’21 challenge [1], the top two systems included KD based on a pre-trained CNN teacher network [4] and a large two-stage fusion CNN teacher model [5].

Jung et al. [24] showed that learning from soft targets in a teacher-student setup has a beneficial effect as one-hot labels do not reflect the blurred decision boundaries between different acoustic scenes. Teacher superiority is achieved by using multiple audio segments of the same scene recorded at different locations to generate teacher predictions.

Another popular idea to generate superior teacher predictions is to use ensembles of specialist models. In this regard, ensembles of device-experts [25], ensembles of models trained on different audio representations [26], and ensembles of specialist models for confusing pairs of acoustic scenes [27] have been studied and successfully applied.

## 2.3. Mixup and Freq-MixStyle

Mixup [28] constructs virtual training samples by linearly mixing two existing samples and their labels. In particular, the coefficients of the convex sample and label combinations are drawn randomly from a Beta distribution, its shape specified by a parameter $\alpha$. Mixup has been shown to improve generalization on ASC tasks before [29].

MixStyle [17] is introduced to enhance domain generalization by mixing channel-wise statistics of images. However, the device-style in spectrograms primarily resides in the frequency-wise statistics. To enhance generalization across recording devices, an adapted version of MixStyle, *Freq-MixStyle*, is proposed in [16]. It proceeds by normalizing each frequency band and denormalizing it with mixed frequency coefficients of two different samples. *Freq-MixStyle* is guided by two parameters: $\alpha$ determines the shape of the Beta distribution used to randomly draw mixing coefficients, and $p$ specifies the probability of whether it is applied to a batch or not.

# 3. MODEL SPECIFICATIONS

## 3.1. Student Model: Compact RFR-CNN

The student model is a Receptive Field Regularized [18, 19] Convolutional Neural Network (RFR-CNN) and is based on CP_ResNet

which performed well in previous editions of the DCASE ASC challenge [1, 2, 6, 29, 30]. The initial width is reduced to $W = 32$ channels, and a grouping of 2 is applied to the penultimate block to obtain a compact model. The max-pooling layers (indicated by $P_f$) are adapted to perform pooling only over the frequency dimension to account for the shorter audio clips (1-second). This allows us to downscale spectrogram dimensionality while preserving temporal information. Finally, $C = 36$ channels are cut from the final residual block of the network to conform to the low-complexity limitations. Table 1 summarizes the architecture of the CNN used as the student.

<table><caption>Table 1: Low-complexity Student Model realized by a compact CP-ResNet Architecture.</caption><thead><tr><th>WIDTH</th><th>GROUPING</th><th>BLOCK</th><th>CONFIG</th></tr></thead><tbody><tr><td>W</td><td></td><td>INPUT</td><td>5 × 5, P</td></tr><tr><td>W</td><td>1</td><td>R</td><td>3 × 3, 1 × 1, P<sub>f</sub></td></tr><tr><td>W</td><td>1</td><td>R</td><td>3 × 3, 3 × 3, P<sub>f</sub></td></tr><tr><td>2 × W</td><td>2</td><td>LINEAR<br>R</td><td>W → 2W<br>3 × 3, 3 × 3</td></tr><tr><td>4 × W - C</td><td>1</td><td>LINEAR<br>R</td><td>2W → 4W<br>3 × 3, 1 × 1</td></tr></tbody></table><p>CLASSIFIER 4 × W - C → 10 CLASSES<br>GLOBAL MEAN POOLING</p><p>P: 2 × 2 MAX POOLING.</p><p>P<sub>f</sub>: 2 × 1 MAX POOLING OVER THE FREQUENCY DIMENSION.</p><p>R: RESIDUAL, THE INPUT IS ADDED TO THE OUTPUT</p>

## 3.2. Teacher Model: PaSST

The main criterion for selecting teacher models is the accuracy of the predictions to reflect class similarity structures. The choice of PaSST transformer models as teachers is motivated by their high performance on downstream tasks – for instance, achieving an accuracy of 76.3% [15] on the *TAU Urban Acoustic Scenes 2020 Mobile development dataset* [2] –, and their ability to recognize fine-grained acoustic events after being pre-trained on the 527 classes of Audioset [22]. The PaSST models selected for downstream training on the *TAU Urban Acoustic Scenes 2022 Mobile development dataset* [2] are pre-trained on Audioset and extract patches of size 16x16 with a stride of 10. We use a *structured Patchout* of 6 only on the frequency dimension, which means that 6 frequency bands are dropped at random during training. This is an important countermeasure to prevent overfitting on the downstream dataset [3]. Due to the short length of the audio clips, we do not apply Patchout across the time dimension. The experimental setup is the same as provided in the system’s technical report [31].

Table 2 compares a PaSST Baseline model with *Mixup* and *Freq-MixStyle* augmentation techniques. *Freq-MixStyle* is only applied to the raw spectrograms. While the effect on real devices is limited, *Mixup* and *Freq-MixStyle* substantially improve over the Baseline on simulated and unseen devices. In particular, on unseen devices, *Freq-MixStyle* significantly outperforms *Mixup*, underlining its superior device generalization capabilities and leading to an overall performance gain compared to *Mixup*. *Ensemble* denotes averaged logits of five PaSST models trained with different *Freq-MixStyle* configurations. The most significant performance gain of
<table><caption>Table 2: PaSST downstream training on the *TAU Urban Acoustic Scenes 2022 Mobile development dataset* [2] following the official test split: Device-wise comparison between Baseline, Mixup ($\alpha$ = 0.3), Freq-MixStyle ($\alpha$ = 0.4, $p$ = 0.4) and an ensemble of five PaSST models trained with different Freq-MixStyle configurations. The provided accuracies (%) are averaged over three runs and the last 10 epochs of training. The devices are grouped according to real devices (Real: A, B, C), the seen, simulated devices (Sim: S1, S2, S3) and the unseen, simulated devices (Unseen: S4, S5, S6).</caption><thead><tr><th rowspan="2">Method</th><th colspan="3">Real Devices</th><th colspan="4">Simulated Devices</th><th colspan="4">Unseen Devices</th><th rowspan="2">Overall</th></tr><tr><th>A</th><th>B</th><th>C</th><th>Real</th><th>S1</th><th>S2</th><th>S3</th><th>Sim</th><th>S4</th><th>S5</th><th>S6</th><th>Unseen</th></tr></thead><tbody><tr><td>PaSST Baseline</td><td>72.00</td><td>63.29</td><td>67.59</td><td>67.63</td><td>58.13</td><td>56.61</td><td>58.25</td><td>57.66</td><td>57.05</td><td>57.62</td><td>53.67</td><td>56.11</td><td>60.46</td></tr><tr><td>+ Mixup</td><td><strong>72.65</strong></td><td>62.86</td><td><strong>68.04</strong></td><td><strong>67.85</strong></td><td>59.24</td><td><strong>57.30</strong></td><td>58.81</td><td>58.45</td><td>57.99</td><td>58.24</td><td>55.07</td><td>57.10</td><td>61.13</td></tr><tr><td>+ Freq-MixStyle</td><td>72.14</td><td><strong>63.55</strong></td><td>67.32</td><td>67.68</td><td><strong>59.69</strong></td><td>57.24</td><td><strong>59.99</strong></td><td><strong>58.97</strong></td><td><strong>59.03</strong></td><td><strong>58.65</strong></td><td><strong>56.98</strong></td><td><strong>58.22</strong></td><td><strong>61.64</strong></td></tr><tr><td>Ensemble</td><td>73.70</td><td>64.07</td><td>68.09</td><td>68.62</td><td>61.30</td><td>58.24</td><td>60.79</td><td>60.11</td><td>60.39</td><td>60.03</td><td>58.76</td><td>59.73</td><td>62.82</td></tr></tbody></table>

the ensemble occurs with unseen devices, underlining the ensemble's robustness.

## 4. KNOWLEDGE DISTILLATION FROM PASST TO CNN

Knowledge Distillation (KD) [12–14] is the concept used to transfer the knowledge of the well-performing, large PaSST transformer from Section 3.2 to the low-complexity RFR-CNN introduced in Section 3.1. PaSST derives its superiority from its size and the pre-training on Audioset [22]. Hence, in a KD setup, PaSST implicitly passes its semantically rich understanding of acoustic scenes gained from the 527 classes of Audioset to the low-complexity student model. We limit the knowledge transfer to the logs and leave experiments on mimicking embeddings as an interesting future research direction. In Section 5, we experiment with KD and probe for a positive performance impact of the following three variations:

* **Teacher Ensemble:** We ensemble five PaSST models with different Freq-MixStyle configurations by averaging their logits. Freq-MixStyle trained models with different values for $\alpha$ and $p$ tend to have different strengths on real, simulated, and unseen devices, leading to a robust and well-performing ensemble, as shown in Table 2.

* **Superior Teacher:** Given that the full 10-second audio files can be reassembled from the 1-second pieces, the distillation loss can be additionally based on teacher predictions for the full 10 seconds. This way, the student has to match superior teacher predictions while having only access to one-tenth of the input sequence. Eq. 2 presents the adapted loss calculation, adding the new distillation loss $L_{DIST\_SUP}$ with its corresponding weight $\lambda_{SUP}$.

$$L_{TOTAL} = L_{LABEL} + \lambda L_{DIST} + \lambda_{SUP} L_{DIST\_SUP} \quad (2)$$

* **Distillation on Out-of-Domain Dataset:** In addition to KD on the *TAU Urban Acoustic Scenes 2022 Mobile development dataset* [2], we experiment with transferring the knowledge using Audioset [22]. For each batch, we sample a batch of Audioset samples of the same size, generate teacher predictions, and calculate the distillation loss. The procedure results in a total loss calculated as shown in Eq. 3.

$$L_{TOTAL} = L_{LABEL} + \lambda(L_{DIST} + L_{DIST\_AUDIOSET}) \quad (3)$$

When computing student and teacher predictions in the context of Mixup, the same audio snippets are mixed using the same mixing coefficients. Freq-MixStyle is applied to student and teacher spectrograms independently, which forces the student to match the teacher's soft targets in the context of different device-styles.

### 4.1. Experimental Setup

**Preprocessing:** The raw audio signal is down-sampled using a sampling rate of 32 kHz. Spectrograms are generated by applying Short Time Fourier Transformation with a window size of 2048 and an overlap of 744 (approximately 36%) in case of the students and a window size of 800 with an overlap of 320 (40%) for the teachers. A Mel-scaled filter bank is applied to create spectrograms with 256 and 128 mel bins for students and teachers, respectively. The applied preprocessing matches the teacher's pre-training, while, for the student, a higher frequency resolution proved beneficial.

**Training:** All students in the KD framework are trained with a batch size of 64 for a total amount of 750 epochs, where the models only process one-tenth of the available data each epoch because of the random 1-second cropping. Adam optimizer with a specific learning rate schedule is applied. The learning rate is exponentially increasing to $1 \times 10^{-3}$ until epoch 150 and linearly decreasing from epoch 250 until epoch 650, dropping to a value of $5 \times 10^{-6}$.

## 5. RESULTS

A summary of the results for the RFR-CNN student model, as presented in Section 3.1, is shown in Table 3. The results are categorized into *Student Baseline* (no KD), *KD Baseline* (PaSST + Freq-MixStyle as teacher) and the three KD variations presented in Section 4. In the following, we describe the effect of Freq-MixStyle compared to Mixup, the effect of KD, and the impact of the KD variations.

### 5.1. Mixup vs. Freq-MixStyle

We investigated different Mixup and Freq-MixStyle configurations and observed that Mixup with $\alpha$ = 0.4 and Freq-MixStyle with $\alpha$ = 0.3 and $p$ = 0.4 yield robust results across a variety of configurations. For *Student Baseline*, Freq-MixStyle outperforms Mixup significantly, achieving the highest performance gains on the unseen device category. In combination with KD, Freq-MixStyle is still slightly superior to Mixup in terms of accuracy but Mixup leads to lower log losses. Freq-MixStyle generalizes much better to unseen devices than Mixup but weakens the performance on real devices.

### 5.2. Effectiveness of KD

We experiment with three different temperature configurations and adapt the distillation loss weight $\lambda$ for each temperature. We select the best of the three settings **High (H)** (T=8, $\lambda$=800), **Medium (M)** (T=3, $\lambda$=100) and **Low (L)** (T=1, $\lambda$=50) to be listed in each row of Table 3. KD with Mixup requires high temperatures, while Freq-MixStyle tends to favour low temperature settings.
<table><caption>Table 3: Results of the low-complexity RFR-CNN student model on the official test split: Accuracies (%) for the device groups **Real** (A, B, C), **Sim** (S1, S2, S3) and **Unseen** (S4, S5, S6), overall accuracy and overall log loss are compared between a baseline using no KD, a baseline using KD and KD variations. Configuration **Temp** is referring to **High** (H) (T=8), **Medium** (M) (T=3) and **Low** (L) (T=1) temperature when calculating teacher and student soft targets, and **AS** indicates the use of Audioset [22] for KD. All results presented are averages of 3 independent runs averaged over the last 10 epochs of training.</caption><thead><tr><th rowspan="2">Method</th><th colspan="5">Configuration</th><th colspan="3">Test Accuracy (%)</th><th colspan="2">Log Loss</th></tr><tr><th>Mixup</th><th>Freq-MixStyle</th><th>Temp</th><th>Teach. Type</th><th>AS</th><th>Real</th><th>Sim</th><th>Unseen</th><th>Overall</th><th>Overall</th></tr></thead><tbody><tr><td rowspan="3">Student Baseline</td><td>✗</td><td>✗</td><td>-</td><td>No</td><td>✗</td><td>61.97</td><td>50.10</td><td>40.71</td><td>50.92</td><td>1.5822</td></tr><tr><td>✓</td><td>✗</td><td>-</td><td>No</td><td>✗</td><td>62.70</td><td>52.48</td><td>42.99</td><td>52.72</td><td>1.4161</td></tr><tr><td>✗</td><td>✓</td><td>-</td><td>No</td><td>✗</td><td>63.89</td><td>56.00</td><td>49.98</td><td>56.62</td><td>1.2344</td></tr><tr><td rowspan="3">KD Baseline</td><td>✗</td><td>✗</td><td>H</td><td>Single</td><td>✗</td><td>66.21</td><td>57.35</td><td>50.14</td><td>57.89</td><td>1.1316</td></tr><tr><td>✓</td><td>✗</td><td>H</td><td>Single</td><td>✗</td><td>66.43</td><td>58.31</td><td>51.32</td><td>58.68</td><td>1.1063</td></tr><tr><td>✗</td><td>✓</td><td>L</td><td>Single</td><td>✗</td><td>64.36</td><td>58.36</td><td>55.12</td><td>59.28</td><td>1.1431</td></tr><tr><td rowspan="2">KD Ensemble</td><td>✓</td><td>✗</td><td>H</td><td>Ensemble</td><td>✗</td><td>66.30</td><td>58.65</td><td>52.06</td><td>59.00</td><td>1.0888</td></tr><tr><td>✗</td><td>✓</td><td>L</td><td>Ensemble</td><td>✗</td><td>64.74</td><td>58.59</td><td>55.14</td><td>59.49</td><td>1.1322</td></tr><tr><td rowspan="2">KD Superior Teacher</td><td>✓</td><td>✗</td><td>H</td><td>Superior</td><td>✗</td><td>66.53</td><td>58.54</td><td>51.89</td><td>58.98</td><td>1.1033</td></tr><tr><td>✗</td><td>✓</td><td>L</td><td>Superior</td><td>✗</td><td>64.73</td><td>58.60</td><td><strong>55.15</strong></td><td>59.49</td><td>1.1313</td></tr><tr><td rowspan="3">KD Audioset</td><td>✓</td><td>✗</td><td>M</td><td>Single</td><td>✓</td><td><strong>66.54</strong></td><td>59.09</td><td>52.49</td><td>59.37</td><td>1.0906</td></tr><tr><td>✗</td><td>✓</td><td>M</td><td>Single</td><td>✓</td><td>64.99</td><td>58.50</td><td>54.43</td><td>59.30</td><td>1.0939</td></tr><tr><td>✓</td><td>✗</td><td>M</td><td>Ensemble</td><td>✓</td><td>66.35</td><td><strong>59.95</strong></td><td>52.99</td><td><strong>59.76</strong></td><td><strong>1.0794</strong></td></tr></tbody></table>

KD Baseline outperforms Student Baseline, leading to an overall accuracy improvement of 2.66% when comparing the Freq-MixStyle configurations. While the accuracy on real devices improves only slightly, the accuracy on unseen devices increases by 5.14%.

## 5.3. Effectiveness of KD variations

Using the PaSST ensemble from Table 2 as the teacher improves the results slightly but consistently for both Freq-MixStyle and Mixup, in terms of overall accuracies and log losses. However, the performance gain of more than 1 percentage point in terms of overall accuracy that the ensemble gives compared to a single PaSST model cannot be transferred to the student.

We investigate a range of $\lambda_{SUP}$ values for each temperature setting for the superior teacher variation. The superior teacher predictions on the 10-second snippets are computed offline. In case of Mixup, the superior predictions are mixed accordingly and no Freq-MixStyle is applied to infer the predictions. KD Superior Teacher performs similar to KD Ensemble using $\lambda_{SUP} = 3.0$ for the Mixup configuration and $\lambda_{SUP} = 1.0$ for the Freq-MixStyle configuration.

With KD Audioset, Freq-MixStyle is applied to samples from both datasets, while Mixup is not applied to the samples from Audioset since no hard labels are available. The Mixup configuration shows the best results in terms of accuracy across all Mixup settings, while the Freq-MixStyle configuration only leads to a minor accuracy improvement over KD Baseline. However, the Freq-MixStyle configuration achieves the lowest log loss across all Freq-MixStyle experiments. Combining Audioset with KD Ensemble and using Mixup leads to the highest overall accuracy and the lowest log loss.

As a final investigation, Figure 1 compares the impact of Freq-MixStyle, Mixup and the KD variations on the overall and the unseen device accuracies. The dominating factor for enhanced generalization to unseen devices is Freq-MixStyle, which clearly outperforms Mixup. The KD variations slightly improve over the KD baseline in terms of overall accuracy but have no significant impact on unseen device performance.

<figure><img src="image_105.png" /><figcaption>Figure 1: Comparison of the different KD variations and augmentation techniques and their effect on the overall accuracy and the performance on unseen devices. Each line depicts an average over three runs and is colored according to the performance on unseen devices. In addition to the configurations shown in the figure, experiments differ in terms of temperature setting (High, Medium, Low).</figcaption></figure>

## 6. CONCLUSION

In this paper, we distilled the knowledge of a PaSST transformer model into a low-complexity CNN. We showed how the pre-trained PaSST model can be effectively adapted to a downstream task. CNN students that are taught by PaSST models perform significantly better than CNNs learning only from the hard class labels. Based on this, we experiment with three KD variations, including a PaSST teacher ensemble, a superior teacher and KD on Audioset, that show promising performance compared to the KD baseline. To enhance generalization to unseen devices, we compared Mixup with Freq-MixStyle and observed that Freq-MixStyle leads to high accuracy improvements on unseen devices for both PaSST teacher and CNN student models.

## 7. ACKNOWLEDGMENT

The LIT AI Lab is financed by the Federal State of Upper Austria.
## 8. REFERENCES

[1] I. Martín-Morató, T. Heittola, A. Mesaros, and T. Virtanen, "Low-complexity acoustic scene classification for multi-device audio: Analysis of DCASE 2021 challenge systems," in DCASE 2021 Workshop, 2021.

[2] T. Heittola, A. Mesaros, and T. Virtanen, "Acoustic scene classification in dcase 2020 challenge: generalization across devices and low complexity solutions," in DCASE 2020 Workshop, 2020, pp. 56–60.

[3] I. Martín-Morató, F. Paissan, A. Ancilotto, T. Heittola, A. Mesaros, E. Farella, A. Brutti, and T. Virtanen, "Low-complexity acoustic scene classification in dcase 2022 challenge," 2022.

[4] B. Kim, S. Yang, J. Kim, and S. Chang, "QTI submission to DCASE 2021: Residual normalization for device-imbalanced acoustic scene classification with efficient design," DCASE2021 Challenge, Tech. Rep., 2021.

[5] C.-H. H. Yang, H. Hu, S. M. Siniscalchi, Q. Wang, W. Yuyang, X. Xia, Y. Zhao, Y. Wu, Y. Wang, J. Du, and C.-H. Lee, "A lottery ticket hypothesis framework for low-complexity device-robust neural acoustic scene classification," DCASE2021 Challenge, Tech. Rep., 2021.

[6] K. Koutini, F. Henkel, H. Eghbal-zadeh, and G. Widmer, "CP-JKU Submissions to DCASE'20: Low-Complexity Cross-Device Acoustic Scene Classification with RF-Regularized CNNs," DCASE2020 Challenge, Tech. Rep., 2020.

[7] S. A. Janowsky, "Pruning versus clipping in neural networks," Physical Review A, vol. 39, no. 12, p. 6600, 1989.

[8] J. Frankle, G. K. Dziugaite, D. Roy, and M. Carbin, "Pruning neural networks at initialization: Why are we missing the mark?" in ICLR, 2021.

[9] C. Liu, Z. Zhang, and D. Wang, "Pruning deep neural networks by optimal brain damage," in INTERSPEECH, 2014, pp. 1092–1095.

[10] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, "Mobilenets: Efficient convolutional neural networks for mobile vision applications," CoRR, vol. abs/1704.04861, 2017.

[11] M. Tan and Q. V. Le, "Efficientnet: Rethinking model scaling for convolutional neural networks," in ICML, vol. 97, 2019, pp. 6105–6114.

[12] G. E. Hinton, O. Vinyals, and J. Dean, "Distilling the knowledge in a neural network," CoRR, vol. abs/1503.02531, 2015.

[13] J. Ba and R. Caruana, "Do deep nets really need to be deep?" in NIPS, 2014, pp. 2654–2662.

[14] C. Bucila, R. Caruana, and A. Niculescu-Mizil, "Model compression," in ACM SIGKDD, 2006, pp. 535–541.

[15] K. Koutini, J. Schlüter, H. Eghbal-zadeh, and G. Widmer, "Efficient training of audio transformers with patchout," CoRR, vol. abs/2110.05069, 2021.

[16] B. Kim, S. Yang, J. Kim, H. Park, J. Lee, and S. Chang, "Domain generalization with relaxed instance frequency-wise normalization for multi-device acoustic scene classification," CoRR, vol. abs/2206.12513, 2022.

[17] K. Zhou, Y. Yang, Y. Qiao, and T. Xiang, "Domain generalization with mixstyle," in ICLR, 2021.

[18] K. Koutini, H. Eghbal-zadeh, and G. Widmer, "Receptive field regularization techniques for audio classification and tagging with deep convolutional neural networks," IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 1987–2000, 2021.

[19] K. Koutini, H. Eghbal-zadeh, M. Dorfer, and G. Widmer, "The Receptive Field as a Regularizer in Deep Convolutional Neural Networks for Acoustic Scene Classification," in EUSIPCO, 2019.

[20] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, "An image is worth 16x16 words: Transformers for image recognition at scale," in ICLR, 2021.

[21] Y. Gong, Y. Chung, and J. R. Glass, "AST: audio spectrogram transformer," in Interspeech, 2021, pp. 571–575.

[22] J. F. Gemmeke, D. P. W. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, "Audio set: An ontology and human-labeled dataset for audio events," in ICASSP, 2017.

[23] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou, "Training data-efficient image transformers & distillation through attention," in ICML 2021, vol. 139, 2021, pp. 10347–10357.

[24] H. Heo, J. Jung, H. Shim, and H. Yu, "Acoustic scene classification using teacher-student learning with soft-labels," in Interspeech, 2019, pp. 614–618.

[25] S. Takeyama, T. Komatsu, K. Miyazaki, M. Togami, and S. Ono, "Robust acoustic scene classification to multiple devices using maximum classifier discrepancy and knowledge distillation," in EUSIPCO, 2020, pp. 36–40.

[26] L. Gao, K. Xu, H. Wang, and Y. Peng, "Multi-representation knowledge distillation for audio classification," Multim. Tools Appl., vol. 81, no. 4, pp. 5089–5112, 2022.

[27] J. Jung, H. Heo, H. Shim, and H. Yu, "Knowledge distillation in acoustic scene classification," IEEE Access, vol. 8, pp. 166870–166879, 2020.

[28] H. Zhang, M. Cissé, Y. N. Dauphin, and D. Lopez-Paz, "mixup: Beyond empirical risk minimization," in ICLR, 2018.

[29] K. Koutini, H. Eghbal-zadeh, and G. Widmer, "CP-JKU submissions to DCASE'19: Acoustic scene classification and audio tagging with receptive-field-regularized CNNs," DCASE2019 Challenge, Tech. Rep., 2019.

[30] K. Koutini, S. Jan, and G. Widmer, "CPJKU Submission to DCASE21: Cross-Device Audio Scene Classification with Wide Sparse Frequency-Damped CNNs," DCASE2021 Challenge, Tech. Rep., 2021.

[31] F. Schmid, S. Masoudian, K. Koutini, and G. Widmer, "CP-JKU submission to dcase22: Distilling knowledge for low-complexity convolutional neural networks from a patchout audio transformer," DCASE2022 Challenge, Tech. Rep., 2022.
# FEATURE SELECTION USING ALTERNATING DIRECTION METHOD OF MULTIPLIER FOR LOW-COMPLEXITY ACOUSTIC SCENE CLASSIFICATION

Lorenz P. Schmidt, Beran Kiliç, Nils Peters

Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany
International Audio Laboratories, Erlangen, Germany
{lopa.schmidt, beran.kilic, nils.peters}@fau.de

## ABSTRACT

Acoustic Scene Classification (ASC) is a common task for many resource-constrained devices, e.g., mobile phones or hearing aids. Limiting the complexity and memory footprint of the classifier is crucial. The number of input features directly relates to these two metrics. In this contribution, we evaluate a feature selection algorithm which we also used in this year's challenge [1].

We propose binary search with hard constraints on the feature set and solve the optimization problem with Alternating Direction Method of Multipliers (ADMM). With minimal impact on accuracy and log loss, results show the model complexity is halved by masking 50% of the Mel input features. Further, we found that training convergence is more stable across random seeds. This also facilitates the hyperparameter search. Finally, the remaining Mel features provide an insight into the properties of the DCASE ASC data set.

**Index Terms**— ASC, feature selection, hard thresholding

## 1. INTRODUCTION

An essential part of the training and inference in machine learning is feature extraction. For Acoustic Scene Classification (ASC), feature extraction is often combined with domain knowledge to make features invariant to pitch changes, amplitude differences, and other modifications [2]. For instance, a Mel-scaled filterbank with additional log transformation and normalization is commonly used. In our DCASE 2022 challenge contribution [1], we saw benefits of using automated features selection to meet the low-complexity model constraint [3]. Here, we explain and further examine this approach.

Studying features and performing subset selection [4] is an experimental step that happens before the actual model training. Several benefits make feature selection interesting to a model developers: First, redundant features increase the computational complexity of the model without benefit to the classification accuracy. For example, in convolutional models, the number of MACs (Multiply-Accumulate) operations scale linearly with the number of features [5]. Second, feature selection makes the model more amenable to pruning as the dimensionality of data is reduced. Third, noisy or weakly correlated features can make the training unstable and increase overfitting. Reducing the number of features avoids modelling the noise of the input and therefore decreases variance in the generalization error. This means that a good model can be found with fewer runs of different seeds. Finally, from a research perspective, selecting features allow us to draw conclusions on the genera-

tion process and to interpret patterns of the feature sets. This may give a clue about deficiencies and better feature parametrization.

Sparse estimation methods and the concept of parsimonious representation [6] is an established field for linear models and found recent application for DNNs [7, 8]. Unstructured pruning and feature selection is expressed in linear models as zeroing out model parameters, which also removes correlation between feature and target. In (non-linear) DNNs, this relation is more complex because parameters' magnitudes are not linearly connected to outcomes. Further solutions may differ for each run, making comparisons and drawing conclusions on feature importance difficult. Applying a proximal operator to the iterate directly gives magnitude pruning [9], often seen in DNN methods such as deep compression [10] or lottery ticket [11]. But an abundance of methods exist and are used for compression, including reweighted $l_2$ penalized techniques [6], log barriers [12], active sets [13] and proximal methods [9].

In Section 2, we state the problem of optimizing with feature cardinality constraint $C$ and derive an algorithm with the Alternating Direction Method of Multipliers (ADMM) [14] and proximal operators [9]. We also explain how binary search is used for finding a bound on $C$ adhering to a threshold on performance loss. We conduct experiments in Section 3, discuss results in Sec. 4, and conclude this study with an outlook in Sec. 5.

### 1.1. Study findings

* Masking half of the features results in minimal loss of accuracy; discarding them saves half of the DNN model complexity

* Feature selection reduces training variance and improves convergence speed; enabling efficient single-run hyperparameter searches

* Different model architectures vary in selected feature sets

* Mel features above 19 kHz seem to provide unexpected benefits to ASC, with fewer information contained between 9 kHz and 19 kHz

## 2. FEATURE SELECTION PROBLEM

In this section, we will derive a feature selection method using ADMM. Instead of using an explicit penalty (e.g. $l_1$ norm on the feature support), we chose to have hard constraint on the feature cardinality. We do this to make the application applicable to binary search as we are interested in the feature sub-set and the amount of features we can mask. The proximal operator makes our method applicable to a number of different constraints, including matrix constraints [15] or mixed $l_1/l_2$ with elastic net regularization [9].

<footnote>The International Audio Laboratories Erlangen are a joint institution of the Friedrich-Alexander-Universität Erlangen-Nürnberg and Fraunhofer IIS.</footnote>
## 2.1. Feature selection using ADMM

We split the DNN loss function and non-smooth constraint with dual decomposition and solve the sub-problem of constraining the feature support with an proximal operator.

Our objective is to minimize the loss function of a model while fulfilling constraints on the support of the feature set. The optimization problems reads then as

$$ \min_{\theta, \mathbf{w}} f_{\theta}(\mathbf{w} \circ \mathbf{X}) \quad \text{s.t.} \quad \text{card}(\mathbf{w}) \leq C, \qquad (1) $$

where $\mathbf{X}$ is the input spectrogram, $\mathbf{w}$ the feature support vector (masking by rows with $\circ$), $\theta$ the model weights, $\text{card}(\mathbf{w})$ the number of non-zero elements, and $C$ is the hard constraint on the feature set. We move the hard constraint into the loss function with indicator

$$ I_C(\mathbf{w}) = \begin{cases} 0 & \text{if } \text{card}(\mathbf{w}) \leq C \\ +\infty & \text{otherwise} \end{cases} \qquad (2) $$

and split along the feature mask $\mathbf{w}$ by introducing equality constraints

$$ \min_{\theta, \mathbf{w}} f_{\theta}(\mathbf{w} \circ \mathbf{X}) + I_C(\hat{\mathbf{w}}) \quad \text{s.t.} \quad \mathbf{w} = \hat{\mathbf{w}}. \qquad (3) $$

The optimization problem contains the smooth and differentiable term of the DNN loss function and the non-smooth indicator function $I_C(\hat{\mathbf{w}})$. While this cannot be solved directly via gradient descent, we can relax the problem by introducing a Lagrangian multiplier for the equality constraint $\mathbf{w} = \hat{\mathbf{w}}$ and optimizing those via gradient ascent. We can derive the augmented Lagrangian as

$$ \mathcal{L}_p(\theta, \mathbf{w}, \hat{\mathbf{w}}, \mu) = f_{\theta}(\mathbf{w} \circ \mathbf{X}) + I_C(\hat{\mathbf{w}}) + \langle \mu, \mathbf{w} - \hat{\mathbf{w}} \rangle + \frac{\rho}{2} \|\mathbf{w} - \hat{\mathbf{w}}\|_2^2 \qquad (4) $$

with $\mu$ the dual variable and $\rho$ smoothing factor of a Moreau envelope [16]. A stable saddle point does not have to exist for non-linear DNNs. In practise, treating $\rho$ as a hyperparameter, we observe convergence. The scaled version with dual variables $\mathbf{u} = \frac{1}{\rho}\mu$ is then

$$ \mathcal{L}_p(\theta, \mathbf{w}, \hat{\mathbf{w}}, \mu) = f_{\theta}(\mathbf{w} \circ \mathbf{X}) + I_C(\hat{\mathbf{w}}) + \frac{\rho}{2} (\|\mathbf{w} - \hat{\mathbf{w}} + \mathbf{u}\|_2^2 - \|\mathbf{u}\|_2^2) \qquad (5) $$

and we can optimize over $(\theta, \mathbf{w}), \hat{\mathbf{w}}$ and $\mu$. In each cycle $k = 0, 1, \dots$ we update the variables separately and repeat until convergence. We use the iteration number $k$ for variables, which are fixed in the following section. For $(\theta, \mathbf{w})$ we can see that

$$ \nabla \mathcal{L}_p(\theta, \mathbf{w}, \hat{\mathbf{w}}^k, \mu^k) = \nabla f_{\theta}(\mathbf{w} \circ \mathbf{X}) + (0, \rho(\mathbf{w} - \hat{\mathbf{w}} + \mathbf{u})) \qquad (6) $$

and therefore the gradient of $\theta$ is unchanged, but $\mathbf{w}$ is influenced by the regularization term. In practice the PyTorch loss function [17] is extended with the regularization term, while derivatives are generated with automatic differentiation.

The sub-problem of $\hat{\mathbf{w}}$ has a closed-form solution. The closest projection onto the support constraint

$$ \min_{\hat{\mathbf{w}}} I_C(\hat{\mathbf{w}}) + \frac{\rho}{2} \|\mathbf{w} - \hat{\mathbf{w}} + \mathbf{u}\|_2^2, \qquad (7) $$

gives a closed-form solution $\hat{\mathbf{w}} = \Pi_C(\mathbf{w} + \mathbf{u})$ [9]. Basically, we keep the $C$ largest elements of $\mathbf{w} + \mathbf{u}$, while setting the rest to zero. Finally, optimizing for $\mathbf{u}$ results in gradient ascent of the dual

variable and in total we have

$$ \begin{aligned} (\theta^{k+1}, \mathbf{w}^{k+1}) &= (\theta^k, \mathbf{w}^k) - \gamma_k \nabla f_{\theta}(\mathbf{w} \circ \mathbf{X}) - (0, \rho(\mathbf{w} - \hat{\mathbf{w}}^k + \mathbf{u}^k)) \\ \hat{\mathbf{w}}^{k+1} &= \Pi_C(\mathbf{w}^{k+1} + \mathbf{u}^k) \\ \mathbf{u}^{k+1} &= \mathbf{u}^k + \mathbf{w}^{k+1} - \hat{\mathbf{w}}^{k+1}, \end{aligned} \qquad (8) $$

with $\gamma_k$ the learning rate of gradient descent. Observe that we keep the penalty $\rho$ fixed, which in turn means that we have a fixed learning rate for the duals.

## 2.2. Learning phase

In Section 2.1, we use ADMM to derive an optimization routine for our hard constraint. Because we make no assumption about the model and we cannot ensure that a saddle point even exists for our Lagrangian, we have to take care of the optimization process. First, we pre-train the model with all features and use those weights as initialization for our feature search. Second, we then apply Eq. 8 for a number of epochs. The learning rate of gradient ascent is fixed, while that of gradient descent anneals to zero over the learning cycle. This ensures that the model loss dominates at the beginning, reducing to zero those features unnecessary for good performance. At the end of the training cycle, the penalty of the ADMM algorithm dominates that of the model, forcing the constraint on feature support to be fulfilled, even though the performance may suffer. Finally, after finding the optimal feature mask, we prune unused features and re-train for a small number of epochs (less than 10) to fine-tune the model.

# 3. EXPERIMENTS

Our experiments use the DCASE 2022 Task 1 [3] split with samples for training (139619), validation/development (29680), and testing (29680). Each sample belongs to one of 10 different acoustic scenes, captured at 44.1 kHz in one of 12 European cities by 3 real (A, B, C) or 6 simulated recording devices (S1-S6).

## 3.1. BC-ResNet model

As a state-of-the-art ASC model, we use a BC-ResNet [18] architecture, specifically the parametrization of BC-ResNet Mod-8 [19] which performed excellent in the DCASE 2021 challenge.

### 3.1.1. Pre-training

Mel-scaled spectrograms with 512 bands are extracted from full-band audio input. We use a frame length of 93 ms (4096 samples), an overlap of 30 ms (1323 samples) and Hamming weighting. We apply a logarithmic transformation and residual normalization [19] with $\lambda = 0.1$ to generalize across different devices.

The input features are augmented to avoid overfitting. We use a random roll of 40% of the signal length. We also use Specaugment [20] in frequency domain with mask parameter of 20 and Mixup [21] with $\alpha = 0.2$. We apply stochastic gradient descent (SGD) to the model and train for 60 epochs. The learning rate increases to 0.01 in a warmup phase (3 epochs) and then decreased to 0.0001 in the remaining 57 epochs. We use momentum of 0.9, weight decay of 0.001, and a mini-batch size of 64.

We train the BC-ResNet with eight different random seeds. The accuracy during training is depicted in Fig. 1 and the final performance can be seen in Table 1. With 512 Mel bands and 1 sec audio
per sample the model has a computational complexity of 300 k pa- rameters and 102 MMACs. We select the best performing model (in terms of the log loss on the testing data) as the initial model weights to use in the following experiments.

3.1.2. *Optimizing with feature constraints*

After pre-training a BC-ResNet model, we are applying Eq. 8 with a given feature constrain *C*. The primal **w**<sup>0</sup> is initialized with ones and projected; and dual variables **w**<sup>0</sup>, **u**<sup>0</sup> with zeros. This penalizes all features in the first step; the second step proceeds normally by masking smallest magnitudes. We update the model weights with an initial learning rate of 4 × 10<sup>-3</sup> and cosine annealing for five epochs. After finding a set of features and masking them, we fine-tune the model for another five epochs, resetting the optimizer to learning rate 4 × 10<sup>-3</sup> first.

The convergence of ADMM highly depends on choice of ρ. We treat ρ as a hyperparameter and found ρ = 3 × 10<sup>-4</sup> to be a good trade-off between fast convergence and stability. Unfortunately, the convergence of the variables depends not only on ρ but also on the number of masked features. As illustrated in Fig. 1, we observe that when masking more features, the initial convergence seems to be faster. When masking more features than realizable (without losing performance), the convergence continues only once the model’s learning rate is annealed enough. As we use cosine annealing, this happens in the later phase of training, where the regularization term introduced by ADMM dominates the update step of θ and **w**. This ensures that the cardinality constraint in Eq. 1 is fulfilled, though we may lose performance. We finally see no difference in convergence of primal and dual variables: both converge in similar rates and to similar residuals (see Fig. 2).

3.1.3. *Finding a good constraint on the feature number*

Choosing an appropriate *C* prior running feature selection is diffi-
cult. If too high, then the model remains too complex and train-
ing speed suffers; if too low, then additional error is introduced

<figure><img src="image_106.png" /><figcaption>Figure 1: Variance and error during training for validation accuracy. Max, min, and geometric means are depicted for eight runs with different random seeds are shown. Reducing number of features from 512 to 256 improves stability and reduces variance, selecting only 128 features affects performance.</figcaption></figure>

<figure><img src="image_107.png" /><figcaption>Figure 2: Convergence of primal (left) and dual (right) variables for masking 256 and 384 features. Masking more features initially increases the influence of the regularization term, but requires more iterations to reach low residuals.</figcaption></figure>

(see also Sec. 4.1). Therefore we apply the feature selection in context of a meta heuristic by introducing a maximal degradation $\epsilon = 3 \times 10^{-2}$ of log loss for our pre-trained model. The heuristic performs binary search, halving upwards if the log loss is under our threshold $\epsilon$; downwards, if not. For reliable results, we run the experiment with 8 different random seeds and select the best performance. Once the step size is below 8, we step alternate to stay below the threshold and explore different log loss in that region. Alternatively, (in a more conservative search) the average over all runs could be compared to the absolute threshold. In the end, we conclude with $C = 274$ masked features (see also Fig. 3).

**3.2. Linear model**

For comparison with the non-linear BC-ResNet model, we also train a linear Support Vector Classifier (SVC) [22]. The SVC performs inference with temporally global features. We reduce the time axis of our Mel spectrogram with moments up to fifth order (mean, variance, skewness, kurtosis, and hyperskewness for each band), resulting in 2560 features per data sample.

Similar to the previous experiment, we apply $l_1$ regularization
during feature selection. The penalty is treated as a hyperparameter.
A search found that the model gives the best accuracy on the testing
data for $\nu = 100$. We have seen almost no variance for the final

<figure><img src="image_108.png" /><figcaption>Figure 3: Log loss on validation set during binary search. Max, min, and geometric means of eight runs with different random seeds are shown. The dashed line indicates the threshold $\epsilon = 0.03$ of the log loss to the performance of the pre-trained model (see Sec. 3.1.1).</figcaption></figure>
<table><caption>Table 1: MACs and performance on development set for best models. The number of features is indicated in brackets behind the model.</caption><thead><tr><th>Model</th><th>MMACs</th><th>Acc. (%)</th><th>Log Loss</th></tr></thead><tbody><tr><td>Baseline [3]</td><td>29.23</td><td>42.90</td><td>1.575</td></tr><tr><td>SVC (ν = 100)</td><td>0.026</td><td>43.79</td><td>1.780</td></tr><tr><td>BC-ResNet (512)</td><td>101.89</td><td>54.24</td><td>1.370</td></tr><tr><td>BC-ResNet (256)</td><td>51.53</td><td>54.10</td><td>1.496</td></tr><tr><td>BC-ResNet (128)</td><td>26.35</td><td>52.80</td><td>1.613</td></tr></tbody></table>

testing results, making the need to do multiple runs unnecessary. Compared to the DCASE 2022 baseline [3], this SVC has a better model accuracy and a worse log loss, but requires significantly fewer MACs (see Table 1).

# 4. DISCUSSION

## 4.1. Training variance and model reduction

Having many noisy or weakly correlated features to the targets increases the risk of overfitting. We see this trend in the variance of our validation results. As can be seen in Fig. 1 for the BC-ResNet without feature selection (512 features), the model uses noisy features. Even at the end of training, its accuracy differs by several percentages. Consequently, finding a good solution requires multiple training runs, which is time- and energy-consuming. Halving the number of features to 256 has several effects. First, in the initial training phase, the model converges much faster on average as the variance is reduced drastically. Then the accuracy approximately stays the same until the final phase. All runs with different seeds give a good final result. Thus, hyperparameter searches can be executed much more efficiently. This reduces experimentation time and improves sustainability of our model search.

When reducing the number of Mel features to 128, variance slightly decreases between epochs 30 to 50, but the final model performance was compromised. We decrease variance, but underfit the dataset by using too few features. This behavior is also supported by our binary search, which suggested 278 masked features, i.e., 234 remaining features (see Fig. 3).

The BC-ResNet model uses only convolutional layers [18]. Consequently, the total number of MACs scales linearly with the number of features. By selecting only 50% of the features, we also halve the number of operations from 101.89 MMACs to 51.53 MMACs without significant change in performance. When further reducing the feature count, the log loss increases significantly (see Table 1).

## 4.2. Interpretation of feature selection

Selection allows making interpretation about the feature set. This is different to feature extraction, where all features are retained and only their representation is changed. Fig. 4 depicts the histogram of the selected feature in our experiments. For the BC-ResNet models, most of the information is contained below 8 kHz. This matches with the literature that 16 kHz sampling rate (leading to a 8 kHz lowpass filtering) gives good ASC results. However, we also tested pre-training of the BC-ResNet with a naïve selection of all features below 8 kHz, but model performance was below that of BC-ResNet (128).

The feature importance around ≈19.8 kHz for all BC-ResNet models is unexpected. It may be related to some recording artifacts by some device types. Masking those bands may improve generalization. Other features are more plausible, e.g., we find that features below 500 Hz are always selected (as they relate to low frequency or harmonics) and the region between 2.5 kHz and 8 kHz contain a lot of information as well, e.g., for consonants in speech.

Also visible in Fig. 4, one can see that feature selection sub samples the frequency bins, suggesting that the band width of ≈5.4 Hz is too small. However, features of very low frequencies (below 60 Hz) are fond to be also relevant. With increasing masking, the feature selection in regions above 500 Hz starts to subsample even more. This is most prominent when examining the 128 remaining features. In this case, only a handful of features are selected.

When comparing the BC-ResNet results to the linear SVC model (upper row of Fig. 4), we see that the regions below 500 Hz and between 1.5 kHz to 8 kHz contain most of the important information. On the other hand, the linear model uses higher frequencies above 9 kHz, but not the prominent peak at 19.8 kHz selected by BC-ResNet. This may suggest that the moments of the SVC may model different information than the convolutional layers of BC-ResNet. This indicates that feature selection is partially model-dependent. More work is needed to better understand this relationship.

<figure><img src="image_109.png" /><figcaption>Figure 4: Selected frequency bands for SVC (upper row) and BC-ResNet models across input feature sizes. The selection patterns are averaged across all 8 runs. The color indicates the number of occurrences a feature was selected; with yellow color representing features that are used in all runs. For the SVC model a frequency band counts when at least one moment is selected.</figcaption></figure>

# 5. CONCLUSION

We showed that feature selection is an important experimental step when developing efficient machine learning models. In addition to providing insights into the working of signal processing, it also improves stability and makes hyperparameter search more efficient. We applied this method in the DCASE 2022 challenge to reduce the total number of MACs [1]. In future, we aim to further examine how features map to different scenes, devices, and cities. This would allow for optimizing methods and models.
## 6. REFERENCES

[1] L. Schmidt, B. Kiliç, and N. Peters, "Structured filter pruning and feature selection for low complexity acoustic scene classification," DCASE2022 Challenge, Tech. Rep., June 2022.

[2] J. Andén and S. Mallat, "Deep scattering spectrum," *IEEE Transactions on Signal Processing*, vol. 62, 04 2013.

[3] I. Martín-Morató, F. Paissan, A. Ancilotto, T. Heittola, A. Mesaros, E. Farella, A. Brutti, and T. Virtanen, "Low-complexity acoustic scene classification in DCASE 2022 challenge," 2022. [Online]. Available: https://arxiv.org/abs/2206.03835

[4] R. Kohavi and G. H. John, "Wrappers for feature subset selection," *Artificial Intelligence*, vol. 97, no. 1, pp. 273-324, 1997, relevance. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S000437029700043X

[5] M. Taghavi and M. Shoaran, "Hardware complexity analysis of deep neural networks and decision tree ensembles for real-time neural data classification," in *2019 9th International IEEE/EMBS Conference on Neural Engineering (NER)*, 2019, pp. 407-410.

[6] F. Bach, R. Jenatton, J. Mairal, and G. Obozinski, "Optimization with sparsity-inducing penalties," *Found. Trends Mach. Learn.*, vol. 4, no. 1, p. 1-106, jan 2012. [Online]. Available: https://doi.org/10.1561/2200000015

[7] Y. Li, C.-Y. Chen, and W. Wasserman, "Deep feature selection: Theory and application to identify enhancers and promoters," in *2015 Annual International Conference on Research in Computational Molecular Biology (RECOMB)*, 04 2015.

[8] M. Wojtas and K. Chen, "Feature importance ranking for deep learning," in *Advances in Neural Information Processing Systems*, H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., vol. 33. Curran Associates, Inc., 2020, pp. 5105-5114. [Online]. Available: https://proceedings.neurips.cc/paper/2020/file/36ac8e558ac7690b6f44e2cb5ef93322-Paper.pdf

[9] N. Parikh, S. Boyd, et al., "Proximal algorithms," *Foundations and trends® in Optimization*, vol. 1, no. 3, pp. 127-239, 2014.

[10] S. Han, H. Mao, and W. J. Dally, "Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding," 2015. [Online]. Available: https://arxiv.org/abs/1510.00149

[11] J. Frankle and M. Carbin, "The lottery ticket hypothesis: Finding sparse, trainable neural networks," 2018. [Online]. Available: https://arxiv.org/abs/1803.03635

[12] H. Kervadec, J. Dolz, J. Yuan, C. Desrosiers, E. Granger, and I. B. Ayed, "Constrained deep networks: Lagrangian optimization via log-barrier extensions," 2019. [Online]. Available: https://arxiv.org/abs/1904.04205

[13] J. Kim and H. Park, "Fast active-set-type algorithms for 11-regularized linear regression," in *Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics*, ser. Proceedings of Machine Learning Research, Y. W. Teh and M. Titterington, Eds., vol. 9. Chia Laguna Resort, Sardinia, Italy: PMLR, 13-15 May 2010, pp. 397-404. [Online]. Available: https://proceedings.mlr.press/v9/kim10a.html

[14] S. Boyd, N. Parikh, E. Chu, B. Peleato, J. Eckstein, et al., "Distributed optimization and statistical learning via the alternating direction method of multipliers," *Foundations and Trends® in Machine learning*, vol. 3, no. 1, pp. 1-122, 2011.

[15] J. E. Cohen, "Computing the proximal operator of the $\ell_1$ induced matrix norm," 2020. [Online]. Available: https://arxiv.org/abs/2005.06804

[16] C. Planiden and X. Wang, *Proximal Mappings and Moreau Envelopes of Single-Variable Convex Piecewise Cubic Functions and Multivariable Gauge Functions*. Cham: Springer International Publishing, 2019, pp. 89-130. [Online]. Available: https://doi.org/10.1007/978-3-030-11370-4_5

[17] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, "Pytorch: An imperative style, high-performance deep learning library," in *Advances in Neural Information Processing Systems 32*, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, Eds. Curran Associates, Inc., 2019, pp. 8024-8035. [Online]. Available: http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf

[18] B. Kim, S. Chang, J. Lee, and D. Sung, "Broadcasted residual learning for efficient keyword spotting," 2021. [Online]. Available: https://arxiv.org/abs/2106.04140

[19] B. Kim, S. Yang, J. Kim, and S. Chang, "QTI submission to DCASE 2021: Residual normalization for device-imbalanced acoustic scene classification with efficient design," DCASE2021 Challenge, Tech. Rep., June 2021.

[20] D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le, "SpecAugment: A simple data augmentation method for automatic speech recognition," in *Interspeech 2019*. ISCA, sep 2019. [Online]. Available: https://doi.org/10.21437%2Finterspeech.2019-2680

[21] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, "mixup: Beyond empirical risk minimization," 2017. [Online]. Available: https://arxiv.org/abs/1710.09412

[22] C. Cortes and V. Vapnik, "Support-vector networks," *Machine learning*, vol. 20, no. 3, pp. 273-297, 1995.
# LOW-COMPLEXITY CNNS FOR ACOUSTIC SCENE CLASSIFICATION

Arshdeep Singh, Mark D. Plumbley

Centre for Vision, Speech and Signal Processing (CVSSP)
University of Surrey, UK

Email: {arshdeep.singh, m.plumbley}@surrey.ac.uk

## ABSTRACT

This paper presents a low-complexity framework for acoustic scene classification (ASC). Most of the frameworks designed for ASC use convolutional neural networks (CNNs) due to their learning ability and improved performance compared to hand-engineered features. However, CNNs are resource hungry due to their large size and high computational complexity. Therefore, CNNs are difficult to deploy on resource constrained devices. This paper addresses the problem of reducing the computational complexity and memory requirement in CNNs. We propose a low-complexity CNN architecture, and apply pruning and quantization to further reduce the parameters and memory. We then propose an ensemble framework that combines various low-complexity CNNs to improve the overall performance. An experimental evaluation of the proposed framework is performed on the publicly available DCASE 2022 Task 1 that focuses on ASC. The proposed ensemble framework has approximately 60K parameters, requires 19M multiply-accumulate operations and improves the performance by approximately 2-4 percentage points compared to the DCASE 2022 Task 1 baseline network.

**Index Terms—** Acoustic scene classification, Low-complexity, Pruning, Quantization. Convolution neural network.

## 1. INTRODUCTION

Convolutional neural networks (CNNs) have shown state-of-the-art performance in comparison to traditional hand-crafted methods in various domains [1]. However, CNNs are resource hungry due to their large size and computational complexity [2, 3], and hence it is difficult to deploy CNNs on resource constrained devices. For example, Cortex-M4 devices (STM32L496@80MHz or Arduino Nano 33@64MHz) have a maximum allowed limit of 128K parameters and 30M multiply-accumulate operations (MACs) per second during inference. Thus, the issue of reducing the size and the computational cost of CNNs has drawn a significant amount of attention in the detection and classification of acoustic scenes and events (DCASE) research community.

In the literature, some CNN parameters such as filters or weights may be redundant, and contribute to extra memory and computational complexity only [4, 5]. For example, Li et al. [6] found that 64% of the parameters which contribute approximately 34% of computation time are redundant. Removing the redundant parameters from CNNs gives similar performance with an advantage of reduced memory and less computational cost.

The majority of the methods applied to eliminate redundant parameters are on filter pruning [7, 8, 9, 10], where a redundant filter from the network is being eliminated. Filter pruning methods have been widely employed in the computer vision. However, only a few

works [5, 11] have applied filter pruning methods in the audio domain, and the issue of designing CNNs for resource constrained devices with constraints on both memory and MACs have not yet been fully explored. Recently, the DCASE challenge Task 1 focuses on designing low-complexity frameworks for ASC with constraints on both memory and MACs. The DCASE challenge 2022 Task 1 [12] provides a baseline CNN, which has 46512 parameters and 29.24M MACs, and achieves an ASC accuracy and log-loss approximately 43% and 1.575 respectively.

This paper aims to design “low-complexity CNNs” which have a maximum number of parameters less than 128K and a maximum number of MACs per seconds less than 30M and performance better than that of the DCASE 2022 Task 1 baseline network for ASC.

The major contributions of the paper is summarized below,

(a) We design a “low-complexity” CNN that has fewer parameters, fewer MACs and better classification performance than the DCASE 2022 Task 1 baseline CNN.

(b) A filter pruning method is applied to compress the “low-complexity CNN” of (a) further. Subsequently, we quantize each parameter from float32 to INT8 data type, reducing networks memory by four times.

(c) An ensemble approach is proposed which combines predictions obtained from several (b) low-complexity CNNs.

(d) Experimental evaluation is undertaken to compare performance of proposed framework with classical methods such as Gaussian Mixture Model (GMMs) and random forest (RF) classifiers and learning based methods such as dictionary learning and a pre-trained high complexity CNN, VGGish with the proposed framework.

The rest of the paper is organized as follows. In Section 2, a brief overview of dataset used and features used for experimentation is described. In Section 3, a procedure to obtain low-complexity CNN and an ensemble framework is described. Section 4 presents experimental analysis. Finally, discussion and conclusion is presented in Section 5 and 6 respectively.

## 2. EXPERIMENTAL DATASET AND FEATURE EXTRACTION

DCASE 2022 Task 1 uses the TAU Urban Acoustic Scenes 2022 Mobile, development and evaluation datasets [13]. The dataset contains recordings from 12 European cities in 10 different acoustic scenes using 4 different recording devices. Each audio recording has 1 second length. The development dataset is divided in training and validation sets. The training dataset consists of 139620 audio examples and the validation dataset consists of 29680 audio examples. The evaluation dataset consists of only audio examples
without any groundtruth available publicly, and the evaluation is
performed by the DCASE challenge community.

**Feature extraction:** For time-frequency representations, log-mel band energies of size (40 × 51) corresponding to an audio signal of 1 second length are extracted. A Hamming asymmetric window of length 40ms, and a hop length of 20ms is used to extract magnitude spectrogram. Next, log-mel spectrogram is computed using 40 mel bands.

### 3. OBTAINING LOW-COMPLEXITY CNNS

**Low-complexity optimal CNN architecture:** We design a simple low-complexity CNN architecture which consists of three convolutional layers (C1, C2 & C3), two pooling layers (P1, P2), a dense layer (D) and a classification layer. We perform an empirical analysis to select an appropriate size of CNN filters and activation functions across the different layers. The filter size for each convolutional layer is chosen from a set, {1 × 1, 3 × 3, 5 × 5, 7 × 7}. We choose hyperbolic tangent (tanh) or rectified linear unit (ReLU) activation function for different layers. The low-complexity CNN is trained using the training dataset with a batch size of 64 with an Adam optimizer for 1000 epochs. A categorical cross-entropy loss function is used during the training process. We apply an early stopping criterion to yield the best network that gives the minimum log-loss for the validation dataset.

The optimal architecture obtained after performing empirical analysis is given in Table 1. The details of the experiments are given in Section 4. The proposed architecture requires approximately 5M MACs to produce an output corresponding to an input of size (40 x 51), and has 14886 parameters. The performance of the trained network is measured in terms of accuracy and log-loss, averaged over 5 different iterations.

**Reducing redundancy in the low-complexity optimal CNN via filter pruning:** To eliminate redundant filters from the low-complexity optimal architecture as given in Table 1, we apply a filter pruning strategy. For each convolutional layer, we identify filter pairs which are similar. Our hypothesis is that similar filters produce similar output or feature maps and hence, contribute to redundancy only. Therefore, one of the similar filters can be eliminated. The similarity between the filters is measured using a cosine distance. We identify the closest filter pairs for each layer separately. A filter from each pair is deemed redundant and eliminated from the network. More information about the similarity based filter pruning method can be found at [14].

The number of redundant filters obtained after performing similarity-based filter pruning for C1 layer is 4 out of 16, C2 layer is 4 out of 16 and C3 layer is 10 out of 32. We obtain 6 different pruned networks that are obtained after pruning C1 layer only, C2 layer only, C3 layer only, C1 and C2 layers, C2 and C3 layers, C1 and C2 and C3 layers. The number of MACs and the number of parameters for each pruned network are given in Table 2.

To regain the loss in performance due to pruning, the pruned networks are fine-tuned in a training similar to that of the unpruned low-complexity optimal network.

**Reducing memory requirement via quantization:** To reduce size of the network, we perform quantization on parameters of each pruned network using Tensorflow-Lite (TFLite). TFLite [15] is an open-source framework to quantizes a pre-trained full-precision network for embedded devices. We quantize the network parameters from 32-bit floating point to 8-bit integers. This leads to reduce the network size 4x.

<table><caption>Table 1: Low-complexity optimal CNN architecture. Here, tanh is a hyperbolic tangent activation function and ReLU is a rectified linear unit activation function.</caption><thead><tr><th>Layer name</th><th>Description</th></tr></thead><tbody><tr><td>C1</td><td>Convolution 16@(3 × 3) + Batch Normalization + tanh</td></tr><tr><td>C2</td><td>Convolution 16@(3 × 3) + Batch Normalization + ReLU</td></tr><tr><td>P1</td><td>Average Pooling (5 × 5)</td></tr><tr><td>C3</td><td>Convolution 32@(3 × 3) + Batch Normalization + tanh</td></tr><tr><td>P2</td><td>Average Pooling (4 × 10)</td></tr><tr><td>D</td><td>Dense (100) + tanh</td></tr><tr><td>Classification</td><td>softmax (10)</td></tr></tbody></table>

**An ensemble framework:** Next, the predictions obtained from various pruned networks are aggregated together in an ensemble framework. The total number of parameters in the ensemble framework that aggregates predictions from all 6 pruned networks are 70.97K and the total number of MACs are 23.84M.

The codes and the pruned networks can be found online ¹².

### 4. PERFORMANCE ANALYSIS

<figure><img src="image_110.png" /><figcaption>Figure 1: Accuracy and log loss obtained for DCASE 2022 Task 1 development validation dataset using the low-complexity CNNs at different filter size and activation function.</figcaption></figure>

<figure><img src="image_111.png" /><figcaption>Figure 2: Accuracy and log loss obtained for DCASE 2022 Task 1 development validation dataset using the low-complexity CNN with (3 x 3) filter size when different combinations of activation functions are applied across “C1”, “C2”, “C3” and “D” layers. Here, “T” represents tanh activation function and “R” represents ReLU activation function.</figcaption></figure>

The accuracy and the log-loss obtained at various filter size and different activation functions in the low-complexity CNN is shown in Figure 1. The low-complexity CNN with (3 x 3) filter size outperforms networks with other filter size.

Using “ReLU” activation function across all layers results in smaller log-loss in comparison to that of the “tanh” activation

<footnote n="1">¹Link: Pruned Quantized models, confusion matrices, evaluation scripts.</footnote>

<footnote n="2">²Link: Model size and complexity calculation.</footnote>
<table><caption>Table 2: Various low-complexity CNNs obtained after pruning and applying quantization (INT8).</caption><thead><tr><th>Sr No.</th><th>Network Name</th><th>Pruned layer</th><th>Architecture (C1-C2-C3-Dense)</th><th>Parameters</th><th>Size (KB)</th><th>MACs (millions)</th></tr></thead><tbody><tr><td>1</td><td>Unpruned optimal low-complexity</td><td>NA</td><td>16-16-32-100</td><td>14886</td><td>18.59</td><td>5.41</td></tr><tr><td>2</td><td>Pruned_C1</td><td>C1</td><td>12-16-32-100</td><td>14254</td><td>17.86</td><td>4.16</td></tr><tr><td>3</td><td>Pruned_C2</td><td>C2</td><td>16-12-32-100</td><td>13138</td><td>16.85</td><td>4.13</td></tr><tr><td>4</td><td>Pruned_C3</td><td>C3</td><td>16-16-22-100</td><td>11396</td><td>15.11</td><td>5.29</td></tr><tr><td>5</td><td>Pruned_C12</td><td>C1 + C2</td><td>12-12-32-100</td><td>12650</td><td>16.26</td><td>3.18</td></tr><tr><td>6</td><td>Pruned_C23</td><td>C2 + C3</td><td>16-12-22-100</td><td>10008</td><td>13.73</td><td>4.04</td></tr><tr><td>7</td><td>Pruned_C123</td><td>C1 + C2 + C3</td><td>12-12-22-100</td><td>9520</td><td>13.14</td><td>3.08</td></tr></tbody></table>

function. On the other hand, combining “tanh” and “ReLU” activation functions across different layers improves the log-loss further as shown in Figure 2. We find that the “tanh” activation function in the first layer and “ReLU” for other layers improves the performance over that of using “ReLU” for all layers. Finally, we choose the low-complexity optimal CNN architecture which performs better than that of architectures with different combinations of activation function. The optimal low-complexity CNN has “tanh” activation function in all layers except C2 and the filter size is (3 x 3) in each convolutional layers.

<figure><img src="image_112.png" /><figcaption>Figure 3: (a) Accuracy and (b) log-loss obtained after pruning different intermediate layers (C1, C2, C3, C1+C2, C2+C3, C1+C2+C3) in the unpruned low-complexity optimal CNN for DCASE 2022 Task 1 development validation dataset. The accuracy and the log-loss is obtained with and without performing the fine-tuning of the pruned network.</figcaption></figure>

Next, we analyse the performance obtained using the low-complexity optimal CNN without performing any pruning and after pruning it. Figure 3 shows the accuracy and the log-loss obtained for the unpruned low-complexity optimal network and its various pruned networks. The unpruned low-complexity optimal network gives 1.475 log-loss and 45.92% accuracy. Eliminating filters from the unpruned network results in a significant reduction in performance, but this is almost entirely restored after fine-tuning.

The performance obtained after aggregating predictions from various pruned networks is given in Table 3. The ensemble framework improves the performance in comparison to that of individual pruned networks.

**Performance comparisons:** The proposed ensemble framework

improves performance as compared to the DCASE 2022 Task 1 baseline network for development and evaluation datasets as given in Table 4. We also compare the performance with the following methods,

* **GMM:** We train Gaussian Mixture Models (GMMs) for each scene class separately, and perform classification using maximum likelihood estimates. The log-mel spectrogram of size (40 x 51) is averaged along temporal dimension to yield (40 x 1) vector, which is given as an input to train the GMMs. The number of mixtures (n) per class are chosen from {5,10,15,20,30,50}.

* **RF:** A random forest (RF) classifier is trained using the log-mel spectrogram averaged along the temporal dimension. The number of estimators are set to 100 and the depth (d) is chosen from {5,20,40,50,100,200}.

* **Dictionary learning:** A dictionary learning framework with structured incoherence and shared features (DLSI) [16] is trained. DLSI framework learns the dictionary for each class by minimizing the reconstruction error and reduces the redundant dictionary atoms in the learning process itself. The number of dictionary atoms per class (k) are chosen from {5,10,15,20,40}.

* **MLP:** A multi-layer perceptron (MLP) network with 2 dense layers each having 100 and 50 units is trained on averaged log-mel spectrogram along the temporal dimension.

* **Pre-trained CNN:** We use pre-trained convolutional layers of VGGish [17, 18] followed by a dense and a classification layer to yield an end-to-end VGGish network. The input to the end-to-end VGGish is log-mel spectrogram of size (40 x 51). We train the end-to-end VGGish for 1000 epochs with similar training settings as used to train the low-complexity CNNs.

The proposed ensemble framework outperforms the other methods as shown in Figure 4. It is interesting to note that the proposed low-complexity optimal CNN performs better than that of the large-scale pre-trained CNN which has 4.5M parameters and has 1077M MACs.

**Similarity analysis among various pruned models:** We analyse similarity between the different pruned models as given in Table 2. For this, we use 10k audio examples from the validation dataset, and generated outputs from the various filters (feature maps). The similarity is measured using the aggregated mean square error (MSE) across 10k examples, computed between the corresponding feature maps of two different pruned models.

We find that a few of the feature maps generated by the two different pruned models are similar with MSE $\le 10^{-4}$. To show
<table><caption>Table 3: Performance obtained for DCASE 2022 Task 1 development validation dataset using the ensemble framework that combines various low-complexity optimal CNNs obtained after pruning and quantization.</caption><thead><tr><th>Sr No.</th><th>Ensemble framework</th><th>Number of parameters</th><th>MACs (millions)</th><th>Size (KB)</th><th>Accuracy (%)</th><th>Log-loss</th></tr></thead><tbody><tr><td>1</td><td>All pruned networks except Pruned_C1</td><td>56712</td><td>19.72</td><td>75.09</td><td>47.14</td><td>1.394</td></tr><tr><td>2</td><td>All pruned networks except Pruned_C2</td><td>57828</td><td>19.75</td><td>76.10</td><td>47.10</td><td>1.396</td></tr><tr><td>3</td><td>All pruned networks except Pruned_C3</td><td>59570</td><td>18.60</td><td>77.84</td><td>47.26</td><td>1.392</td></tr><tr><td>4</td><td>All pruned networks except Pruned_C12</td><td>58316</td><td>20.70</td><td>76.69</td><td>47.45</td><td>1.394</td></tr><tr><td>5</td><td>All pruned networks except Pruned_C23</td><td>60958</td><td>19.84</td><td>79.22</td><td>47.52</td><td>1.389</td></tr><tr><td>6</td><td>All pruned networks except Pruned_C123</td><td>61446</td><td>20.80</td><td>79.81</td><td>47.35</td><td>1.392</td></tr><tr><td>7</td><td>Ensemble on all pruned networks</td><td>70966</td><td>23.84</td><td>92.95</td><td>47.45</td><td>1.389</td></tr></tbody></table>

<table><caption>Table 4: Performance comparison</caption><thead><tr><th rowspan="3">Framework</th><th colspan="4">Dataset</th></tr><tr><th colspan="2">Development</th><th colspan="2">Evaluation</th></tr><tr><th>Accuracy (%)</th><th>log-loss</th><th>Accuracy (%)</th><th>log-loss</th></tr></thead><tbody><tr><td>DCASE baseline network [12]</td><td>42.9</td><td>1.575</td><td>44.2</td><td>1.532</td></tr><tr><td>Proposed ensemble (#3 in Table 3)</td><td>47.26</td><td>1.392</td><td>45.9</td><td>1.492</td></tr></tbody></table>

<figure><img src="image_113.png" /><figcaption>Figure 4: Accuracy comparison of various methods for DCASE 2022 Task 1 development validation set. Here, we show the best accuracy obtained from the DLSI, GMM and RF framework.</figcaption></figure>

<figure><img src="image_114.png" /><figcaption>Figure 5: (a) Aggregated MSE computed between the feature maps generated using Pruned_C2 and Pruned_C3 network from the C1 layer. (b) shows the different and the similar feature maps corresponding to 4th and 6th feature map index.</figcaption></figure>

this, the aggregated MSE computed across corresponding feature maps generated by the Pruned_C2 and the Pruned_C3 network is plotted in Figure 5(a). A few of the feature maps have MSE close to zero, and hence these feature maps are similar. We also show the different and the similar feature maps obtained from the Pruned_C2 network and the Pruned_C3 network corresponding to 4th and 6th index for a given input in Figure 5(b). This suggests that similar feature maps across different pruned networks are redundant, and are not required to compute again for the other network.

We find that the Pruned_C1 network shares 7 feature maps with the Pruned_C12 network, and 3 feature maps with the Pruned_C123 network across C1 layer. Similarly, the Pruned_C2 network shares 7 feature maps with the Pruned_C3, network and 6 feature maps with the Pruned_C23.

Ignoring similar feature maps across different pruned models, the MACs could be further reduced by 6M points, and the number of total parameters are reduced by approximately 3.4k in the ensemble framework that combines all pruned networks.

## 5. DISCUSSION

We observe that small size network obtained after pruning a relatively large size network gives better accuracy compared to that of the similar size network obtained from scratch. For example, the Pruned_C123 network having 9.6k parameters and (3 x 3) filter size

gives approximately 16 percentage points more accuracy compared to that of the similar size low-complexity CNN having (1 x 1) filter size and 8.6k parameters as shown in Figure 1.

We find that the ensemble on various CNNs improves the performance as compared to the individual CNN. However, the ensemble on the various CNNs consumes more resources. Therefore, pruning individual CNNs provide an advantage to use the ensemble framework efficiently. To improve the efficiency of the ensemble further, the shared feature maps (similarity) across various CNNs can be ignored.

## 6. CONCLUSION

This paper focuses on designing a low-complexity system for acoustic scene classification. A filter pruning and quantization is applied to obtain compressed, accelerated, and low-size CNN. Further, various low-size CNNs are combined in the ensemble framework to improve classification performance. The proposed framework shows promising results in terms of reduction in parameters and improved performance. In future, our aim is to improve the performance of the low-complexity framework further.
# 7. ACKNOWLEDGEMENTS

This work was partly supported by Engineering and Physical Sciences Research Council (EPSRC) Grant EP/T019751/1 “AI for Sound (AI4S)”. For the purpose of open access, the authors have applied a Creative Commons Attribution (CC BY) licence to any Author Accepted Manuscript version arising. Thanks to James A. King and Xubo Liu, PhD students at University of Surrey, UK for the discussions.

# 8. REFERENCES

[1] J. Gu, Z. Wang, J. Kuen, L. Ma, A. Shahroudy, B. Shuai, T. Liu, X. Wang, G. Wang, J. Cai, et al., “Recent advances in convolutional neural networks,” *Pattern Recognition*, vol. 77, pp. 354–377, 2018.

[2] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” *Proceedings of International Conference on Learning Representations, ICLR (arXiv preprint arXiv:1409.1556)*, 2015.

[3] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet classification with deep convolutional neural networks,” *Advances in Neural Information Processing Systems*, pp. 1097–1105, 2012.

[4] Y. Wen and D. Gregg, “Exploiting weight redundancy in CNNs: Beyond pruning and quantization,” *arXiv preprint arXiv:2006.11967*, 2020.

[5] A. Singh, P. Rajan, and A. Bhavsar, “SVD-based redundancy removal in 1-D CNNs for acoustic scene classification,” *Pattern Recognition Letters*, vol. 131, pp. 383–389, 2020.

[6] H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf, “Pruning filters for efficient ConvNets,” *Proceedings of International Conference on Learning Representations, ICLR*, 2017.

[7] Z. Liu, J. Li, Z. Shen, G. Huang, S. Yan, and C. Zhang, “Learning efficient convolutional networks through network slimming,” *Proceedings of the IEEE International Conference on Computer Vision*, pp. 2736–2744, 2017.

[8] Y. He, X. Zhang, and J. Sun, “Channel pruning for accelerating very deep neural networks,” *Proceedings of the IEEE International Conference on Computer Vision*, pp. 1389–1397, 2017.

[9] Y. He, P. Liu, Z. Wang, Z. Hu, and Y. Yang, “Filter pruning via geometric median for deep convolutional neural networks acceleration,” *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition*, pp. 4340–4349, 2019.

[10] M. Lin, R. Ji, Y. Wang, Y. Zhang, B. Zhang, Y. Tian, and L. Shao, “HRank: Filter pruning using high-rank feature map,” *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 1529–1538, 2020.

[11] B. Kim, S. Yang, J. Kim, and S. Chang, “QTI submission to DCASE 2021: Residual normalization for device imbalanced acoustic scene classification with efficient design,” *DCASE2021 Challenge, Tech. Rep.*, 2021.

[12] I. Martín-Morató, F. Paissan, A. Ancilotto, T. Heittola, A. Mesaros, E. Farella, A. Brutti, and T. Virtanen, “Low-complexity acoustic scene classification in DCASE 2022 Challenge,” *arXiv preprint arXiv:2206.03835*, 2022.

[13] T. Heittola, A. Mesaros, and T. Virtanen, “Acoustic scene classification in DCASE 2020 Challenge: Generalization across devices and low complexity solutions,” *Proceedings of the Detection and Classification of Acoustic Scenes and Events (DCASE) Workshop*, 2020.

[14] A. Singh and M. D. Plumbley, “A passive similarity based CNN filter pruning for efficient acoustic scene classification,” *Interspeech (arXiv preprint arXiv:2203.15751)*, 2022.

[15] R. David, J. Duke, A. Jain, V. Janapa Reddi, N. Jeffries, J. Li, N. Kreeger, I. Nappier, M. Natraj, T. Wang, et al., “Tensor-Flow Lite Micro: Embedded machine learning for TinyML systems,” *Proceedings of Machine Learning and Systems*, vol. 3, pp. 800–811, 2021.

[16] I. Ramirez, P. Sprechmann, and G. Sapiro, “Classification and clustering via dictionary learning with structured incoherence and shared features,” *Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition*, pp. 3501–3508, 2010.

[17] J. Cramer, H.-H. Wu, J. Salamon, and J. P. Bello, “Look, listen, and learn more: Design choices for deep audio embeddings,” *Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, pp. 3852–3856, 2019.

[18] S. Hershey, S. Chaudhuri, D. P. W. Ellis, J. F. Gemmeke, A. Jansen, C. Moore, M. Plakal, D. Platt, R. A. Saurous, B. Seybold, M. Slaney, R. Weiss, and K. Wilson, “CNN architectures for large-scale audio classification,” *Proceedings of International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, pp. 131–135, 2017.
# IMPROVED DOMAIN GENERALIZATION VIA DISENTANGLED MULTI-TASK LEARNING IN UNSUPERVISED ANOMALOUS SOUND DETECTION

Satvik Venkatesh¹,², Gordon Wichern¹, Aswin Subramanian¹, Jonathan Le Roux¹

¹Mitsubishi Electric Research Laboratories (MERL), Cambridge, MA, USA

²Interdisciplinary Centre for Computer Music Research, University of Plymouth, UK

satvik.venkatesh@plymouth.ac.uk, {wichern, subramanian, leroux} @merl.com

## ABSTRACT

We investigate a novel multi-task learning framework that disentangles domain-shared features and domain-specific features for domain generalization in anomalous sound detection. Disentanglement leads to better latent features and also increases flexibility in post-processing due to the availability of multiple embedding spaces. The framework was at the core of our submissions to the DCASE2022 Challenge Task 2. We ranked 5th out of 32 teams in the competition, obtaining an overall harmonic mean of 67.57% on the blind evaluation set, surpassing the baseline by 13.5% and trailing the top rank by 3.4%. We also explored machine-specific loss functions and domain generalization methods, which showed improvements on the development set, but were less effective on the evaluation set.

**Index Terms—** Anomaly detection, disentanglement, multi-task learning, domain generalization, representation learning

## 1. INTRODUCTION

Machine condition monitoring using acoustic sensors is an important topic for industry with applications such as factory automation and predictive maintenance. Automatic detection of anomalous sounds is a particularly important application, however, all possible types of anomalous sounds may not be known in advance, and purposefully damaging machinery to collect anomalous sound recordings is undesirable. Thus, there has been much recent research interest in the field of unsupervised anomalous sound detection, where only data collected under normal operating conditions is available for training machine learning models.

Much of the recent progress in unsupervised anomalous sound detection has been driven by DCASE challenges on the topic [1-3]. Typical approaches include those based on autoencoder-like architectures [4-10], where a model trained only on normal data to reconstruct its input should exhibit large reconstruction error when presented with an anomalous example at inference time. Another class of approaches, which we refer to as surrogate task models, use an alternative supervised training task to learn a model of normality, and then measure deviations from normal to predict anomalies. Example surrogate tasks include outlier exposure [6, 11], predicting metadata (e.g., machine instance) or attributes (e.g., operating load) [12-14], and learning to predict what augmentations (e.g., time-stretching or pitch-shifting) were applied to an audio clip [15].

As in many areas where deep learning-based models have become the predominant approach, unsupervised anomalous sound

detection suffers from issues related to robustness. To better tackle such issues, the anomalous sound detection tasks of both the 2021 and 2022 DCASE challenges focused on performance under domain shift, where acoustic conditions differ based on environmental background noise or other machine operating conditions. The goal is to develop methods that should perform equally well in a source domain, where most of the (normal) training data comes from, and in a target domain, where only a few normal examples are available. The 2021 challenge task [2] assumed the domain (source or target) of the audio sample was known at inference time (a configuration referred to as domain adaptation), while the 2022 task [3] assumes the domain is unavailable at inference time (a configuration referred to as domain generalization).

While many well-known techniques exist for domain generalization (see [16] for an overview), we focus our efforts on disentangled representation learning [17], where subsets of learned feature dimensions correspond to specific factors in the dataset. Disentanglement has been successfully applied for music information retrieval in the audio domain [18] and in approaches to domain adaptation for image classification [19]. Specifically, we consider learning feature representations for each normal sound example in the training set, where subsets of features are learned using different surrogate tasks. In the case of the DCASE 2022 Task 2 dataset, we learn a subset of domain-shared features, whose surrogate task is to predict the section index regardless of domain (each section is dedicated to a specific type of domain shift, with other conditions being shared across domains), and subsets of domain-specific features each associated with a surrogate task consisting of predicting a particular machine attribute (e.g., specific states or environmental conditions of the machine), which are typically different across domains and sections. We demonstrate experimentally that our disentangled model performs better than a multi-task learning model where features are not disentangled, and further show that by weighting individual anomaly scores computed over different disentangled dimensions, we obtain an ensemble-like system using a single model. Furthermore, by examining the anomaly score in specific disentangled dimensions, we can better understand which attribute may have caused the anomaly, improving the explainability of deep learning models.

As discussed in our challenge report [20], we also explored machine-specific variations to the loss function. While they led to significant improvements on the dev set, they ultimately performed worse on the eval set. This paper thus focuses on our best performing disentangled models.

<footnote>This work was performed while S. Venkatesh was an intern at MERL.</footnote>
## 2. DISENTANGLED ANOMALY DETECTOR

In this paper, we investigate an approach that disentangles a learned latent representation into domain-shared and domain-specific features for domain generalization in anomalous sound detection, as illustrated in Fig. 1. In particular, we refer to sections as domain-shared features and to attributes as domain-specific features. For example, in Fan's section 00, machine noises occurring in the source domain are of type W and X, while those occurring in the target domain are of type Y and Z. Therefore, section 00 is common to both domains but the machine noises are different across domains.

### 2.1. Surrogate Task Training

During training, we have a dataset of $N$ normal training examples for a given machine type, $\mathcal{D} = \{(X^{(n)}, y^{(n)})\}_{n=1}^N$, where $X \in \mathbb{R}^{F \times T}$ is a magnitude spectrogram with $F$ frequencies and $T$ time frames, and $y = [y_s, y_{a_1}, \dots, y_{a_M}] \in \mathbb{N}^{M+1}$ is a vector of categorical surrogate task labels, where $y_s$ represents machine section and $y_{a_m}$ represents the categorical label of the $m$-th attribute among the $M$ different attributes available for the given machine type. We obtain a domain-shared (section) embedding $z_S$ and a domain-specific (attribute) embedding $z_A$ as:

$$z_S = \Phi^{\text{Sec}}[\text{CNN}(X)] \in \mathbb{R}^{D_S}, \quad z_A = \Phi^{\text{Att}}[\text{CNN}(X)] \in \mathbb{R}^{D_A} \quad (1)$$

where $\text{CNN}(\cdot)$ is a shared convolutional neural network, while $\Phi^{\text{Sec}}$ and $\Phi^{\text{Att}}$ represent section and attribute specific linear embedding layers, respectively (implemented as $1 \times 1$ convolutions). All parameters are trained by minimizing $\mathcal{L} = \mathcal{L}^{\text{Sec}} + \mathcal{L}^{\text{Att}}$, where

$$\mathcal{L}^{\text{Sec}} = \log \frac{\exp(w_{0,y_s} \cdot z_S + b_{0,y_s})}{\sum_{c=1}^{C} \exp(w_{0,c} \cdot z_S + b_{0,c})}, \quad (2)$$

$$\mathcal{L}^{\text{Att}} = \sum_{m=1}^{M} \log \frac{\exp(w_{m,y_m} \cdot z_A + b_{m,y_m})}{\sum_{c_m=1}^{C_m} \exp(w_{m,c_m} \cdot z_A + b_{m,c_m})} \quad (3)$$

are the cross-entropy losses for section and attributes, respectively, $w_{i,j}$ and $b_{i,j}$ are learned weight vectors and biases of the associated classifiers, $c$ indexes the $C = 6$ sections and $c_m$ indexes the $C_m$ values of the $m$-th attribute. Because not all attributes are present among all audio examples of a given machine type in the DCASE 2022 Task 2 dataset, the attribute loss in (3) is combined over all attributes in a multi-task learning fashion from the same embedding $z_A$, rather than learning disentangled feature dimensions for each attribute. If an attribute is unknown for an audio example, the corresponding term in the sum of (3) is ignored.

We note that our formulation of attribute learning in (3) as a multi-task learning problem with a different objective for each attribute differs from [21] where every possible combination of section and attribute corresponded to a different class.

### 2.2. Inference Approaches

The nearest neighbor (NN) algorithm is a simple and effective approach for anomaly detection [22, 23] given feature vectors of normal samples. As illustrated in Fig. 1, during inference we use the NN distance between a test embedding $z_q$ and all corresponding training set embeddings $z_q^{(j)}$ for computing an anomaly score, i.e.,

$$D_{\text{NN}}(z_q, \mathcal{D}) = \min_{j \in \mathcal{D}} D_{\cos}(z_q, z_q^{(j)}), \quad (4)$$

where $D_{\cos}(\cdot, \cdot)$ is the cosine distance between two embedding vectors. The disentangled model allows us to explore multiple infer-

<figure><img src="image_115.png" /><figcaption>Figure 1: Block diagram of disentangled anomaly detector. In the figure, NN stands for Nearest Neighbor. In the training phase, exclusive latent spaces were assigned to sections and attributes.</figcaption></figure>

ence approaches depending on which embedding dimensions we use for $z_q$ in (4) as discussed below.

**Disentangled Concatenated:** Use the concatenated embedding $z_C = [z_S^T, z_A^T]^T$ in (4) as shown in the bottom-left of Fig. 1.

**Disentangled Weighted:** As illustrated in the bottom-right of Fig. 1, we take a weighted average of NN distances separately computed for section embedding $z_S$ and attribute embedding $z_A$, i.e.,

$$D_{\text{NN}}^{\text{wt}}(z_S, z_A, \mathcal{D}) = w_S D_{\text{NN}}(z_S, \mathcal{D}) + w_A D_{\text{NN}}(z_A, \mathcal{D}) \quad (5)$$

where $w_S$ and $w_A$ are scalar weights, which are optimized after training is complete based on dev set performance. The best weights for each machine are shown in Table 4.

**Disentangled Sections:** Use only section embedding $z_S$ in (4).

**Disentangled Attributes:** Use only attribute embedding $z_A$ in (4).

At test time, the section label of the test sample is known, therefore, we limit the training set samples from $\mathcal{D}$ when computing the NN distance to be only those samples belonging to the appropriate section. Furthermore, our CNN architecture, detailed in Section 3.3, operates on spectrogram chunks of $T = 32$ time frames ($\sim 1$ s), while each test sample is 10 s long. Using a chunk hop size of one frame, we obtain 282 embedding vectors per 10 s audio file. Following [22], we merged the embedding vectors for each sample by calculating their mean, except for valve where merging based on standard deviation provided significant gains. We then use the merged embedding vectors for computing the anomaly score.

## 3. EXPERIMENTAL SETUP

### 3.1. Dataset

There are seven different machine types in the DCASE 2022 Task 2 dataset [3] — ToyCar, ToyTrain, Bearing, Fan, Gearbox, Slider, and Valve. ToyCar and ToyTrain are from the ToyADMOS2 dataset [24], and the five other machines are from the MIMII DG dataset [25]. The data under each machine type is divided into sections, each of which corresponds to a specific type of domain shift. For example, in Fan, section 00 refers to different machine noise between source and target domains, while section 01 refers to different factory noise.

For each audio file, information about its section as well as one or more attributes is given. For machines belonging to the MIMII
DG dataset [3], only information on the domain shifting attribute, such as the type of machine noise in Fan's section 00 and the type of factory noise in Fan's section 01, was present. For ToyCar and ToyTrain, which belong to the ToyADMOS2 dataset [24], information on all attributes was present in the filenames, even for those attributes that are not the domain shifting one. For the multi-task attribute learning (3), we make use of all present attributes, and represent them as categorical variables using all possible values found in the training set.

## 3.2. Audio Features and Training Strategy

The dataset contained 10 s audio files at a sampling rate of 16 kHz. We adopted short-time Fourier transform magnitude spectrograms as features for the neural network. The hop size was set to 32 ms and the window size was 128 ms (2048 samples). While training the neural network, the number of time steps for each audio example was 32 frames. Therefore, the input shape for the network was $1025 \times 32$. One epoch is defined as training the network on all 6000 audio files (six sections with 1000 examples in each section). For each audio file, a random chunk of 32 frames is selected for training. The advantages of this technique were reduced RAM usage, less chance of overfitting within epochs, and improved generalization compared with the baseline.

We adopted the Adam optimizer using a batch size of 32. In most cases, the learning rate was set to $10^{-4}$. For ToyCar, we found a minor improvement by setting it to $10^{-5}$. We trained the models for a maximum of 300 epochs, saved the model's weights every 5 epochs, tested the anomaly detector's performance on the development set, and selected the best performing model for each machine. We were unable to observe a clear relationship between the performances on the surrogate task and detection of anomalies. For instance, an improvement in the classification accuracy of sections (the surrogate task) was not necessarily accompanied by an improvement in anomaly detection. A similar observation has been made by previous studies using autoencoder-based models [10].

## 3.3. Neural Network Architecture

Morita et al. [22] found that the MobileFaceNet architecture [26] performed better than MobileNetV2 [27] as a feature extractor. We observed a similar improvement in initial experiments, and hence adopted MobileFaceNet. The parameter settings for MobileFaceNet can be found in Table 1. The output of the global depth-wise convolution (GDC) layer is a 512-D embedding vector. This is connected to the linear embedding layers (i.e., $1 \times 1$ convolutions) $L^{Sec}$ and $L^{Att}$ defined in Section 2, and associated softmax classification layers. Additionally, we explore minor modifications to the embedding and softmax layers as explained in Section 3.5.

## 3.4. Evaluation Metrics

We evaluate our models independently for each section and machine type using the three official metrics [3]: area under the ROC curve in the source (AUC (S)) and target (AUC (T)) domains, where the normal test samples are compared against anomalies from both domains, along with the domain agnostic partial AUC (pAUC) computed under low false-alarm-rate conditions.

For threshold-dependent metrics, we followed a similar approach to the baseline [3] and assumed the scores follow a gamma distribution. The parameters of the gamma distribution are estimated from the NN anomaly scores computed on the training set

<table><caption>Table 1: MobileFaceNet [26] architecture, where all convolutions are 2D and dw-Conv refers to depth-wise convolution. In the network, Linear Conv $1 \times 1$ (sec) is connected to Softmax (sec), and Linear Conv $1 \times 1$ (att) is connected to the other softmax layers for attributes. For each layer, we show the expansion factor (t), number of channels (c), number of repeats (n), and stride (s). All convolutions excluding the final linear layers use PReLU activations.</caption><thead><tr><th>Input</th><th>Operator</th><th>t</th><th>c</th><th>n</th><th>s</th></tr></thead><tbody><tr><td>1x32x1025</td><td>Conv 3x3</td><td>-</td><td>64</td><td>1</td><td>2</td></tr><tr><td>64x16x513</td><td>dw-Conv 3x3</td><td>-</td><td>64</td><td>1</td><td>1</td></tr><tr><td>64x16x513</td><td>Bottleneck</td><td>2</td><td>64</td><td>5</td><td>2</td></tr><tr><td>64x8x257</td><td>Bottleneck</td><td>4</td><td>128</td><td>1</td><td>2</td></tr><tr><td>128x4x129</td><td>Bottleneck</td><td>2</td><td>128</td><td>6</td><td>2</td></tr><tr><td>128x2x65</td><td>Bottleneck</td><td>4</td><td>128</td><td>1</td><td>2</td></tr><tr><td>128x1x33</td><td>Bottleneck</td><td>2</td><td>128</td><td>2</td><td>1</td></tr><tr><td>128x1x33</td><td>Conv 1x1</td><td>-</td><td>512</td><td>1</td><td>1</td></tr><tr><td>512x1x33</td><td>Linear GDC 1x33</td><td>-</td><td>512</td><td>1</td><td>1</td></tr><tr><td>512x1x1</td><td>Linear Conv 1x1 (sec)</td><td>-</td><td>128</td><td>1</td><td>1</td></tr><tr><td>512x1x1</td><td>Linear Conv 1x1 (att)</td><td>-</td><td>128</td><td>1</td><td>1</td></tr><tr><td>128x1x1</td><td>Softmax (sec)</td><td>-</td><td>6</td><td>-</td><td>-</td></tr><tr><td>128x1x1</td><td>Softmax (att<sub>1</sub>)</td><td>-</td><td>C<sub>1</sub></td><td>-</td><td>-</td></tr><tr><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td></tr><tr><td>128x1x1</td><td>Softmax (att<sub>M</sub>)</td><td>-</td><td>C<sub>M</sub></td><td>-</td><td>-</td></tr></tbody></table>

samples independently for each section (excluding self neighbors). For five machines, we set the anomaly detection threshold as the 90th percentile of the gamma distribution. For Fan and Bearing, we observed low sensitivity and hence adopted 60th percentile.

## 3.5. Other Models Considered

ArcFace [29] was shown to improve class separability by adding angular margin to the loss. We investigated this technique's advantage by training on section indices. The feature scale and margin parameters were set to 32 and 0.5 respectively. We found ArcFace did not work well in a multi-task learning setting, probably because all attributes were not present in every example.

**Conventional Multi-task Learning (MTL):** In this framework, the GDC layer from Table 1 is connected to a single 2D convolutional $1 \times 1$ layer with 256 channels. In other words, the features are in an entangled latent space.

**Machine-Specific Loss (MSL):** Although not the focus of this paper, we also compare performance against the other two systems we submitted to the challenge: (S1) MSL as described in [20] and (S2) an ensemble of MSL and the attentive neural process (ANP) approach in [10] as detailed in Table 4.

# 4. RESULTS

**Development Set** results are shown in Table 2. Training using only section labels obtains an overall harmonic mean of 72.82%, which is significantly higher than both the baselines. This improvement is attributed to adopting the Nearest Neighbor algorithm during post-processing [22] and to our new training strategy explained in Section 3.2. Adopting ArcFace, which is essentially training on section indices with additive angular margin, improved the overall performance to 73.62%, while the AUC(T) improved from 67.34% to 71.79%. MTL, which trains on sections and attributes, obtained a lower overall performance of 72.47%, but improved the AUC(T) to 70.72%. Note that the MTL model does not use ArcFace. The Disentangled Sections (Disent_Sec) model only considers the section
<table><caption>Table 2: Results of different models on the development test set. We merge the three metrics and all sections to obtain a single number per machine using the harmonic mean. We also report the harmonic mean across machines and sections for each of the three metrics.</caption><thead><tr><th>System</th><th>ToyCar</th><th>ToyTrain</th><th>Bearing</th><th>Fan</th><th>Gearbox</th><th>Slider</th><th>Valve</th><th>AUC (S)</th><th>AUC (T)</th><th>pAUC</th><th>Overall</th></tr></thead><tbody><tr><td>MSL+ANP (S2)</td><td>76.43</td><td><strong>59.96</strong></td><td><strong>73.93</strong></td><td><strong>68.89</strong></td><td><strong>85.37</strong></td><td><strong>85.93</strong></td><td><strong>95.83</strong></td><td><strong>87.55</strong></td><td><strong>73.43</strong></td><td><strong>70.36</strong></td><td><strong>76.43</strong></td></tr><tr><td>MSL (S1)</td><td>76.43</td><td>59.37</td><td>73.93</td><td>68.85</td><td>83.03</td><td>85.37</td><td>95.63</td><td>86.78</td><td>73.34</td><td>69.68</td><td>75.93</td></tr><tr><td>Disent_Wt (S4)</td><td><strong>76.95</strong></td><td>59.74</td><td>72.07</td><td>63.91</td><td>81.38</td><td>85.14</td><td>94.50</td><td>86.09</td><td>71.65</td><td>68.21</td><td>74.57</td></tr><tr><td>Disent_Cat (S3)</td><td>76.43</td><td>58.67</td><td>67.09</td><td>63.18</td><td>80.99</td><td>85.37</td><td>95.01</td><td>84.61</td><td>70.59</td><td>67.02</td><td>73.34</td></tr><tr><td>Disent_Sec</td><td>76.84</td><td>56.64</td><td>72.07</td><td>62.35</td><td>81.04</td><td>84.84</td><td>94.42</td><td>86.43</td><td>68.64</td><td>68.01</td><td>73.45</td></tr><tr><td>Disent_Att</td><td>75.26</td><td>59.74</td><td>60.82</td><td>63.02</td><td>78.86</td><td>78.88</td><td>92.72</td><td>82.12</td><td>69.31</td><td>64.09</td><td>71.08</td></tr><tr><td>MTL</td><td>75.61</td><td>59.37</td><td>68.24</td><td>59.14</td><td>80.63</td><td>83.51</td><td>94.49</td><td>81.35</td><td>70.72</td><td>66.83</td><td>72.47</td></tr><tr><td>Sec_ArcFace</td><td>72.31</td><td>58.09</td><td>71.30</td><td>68.85</td><td>79.37</td><td>82.50</td><td>92.87</td><td>86.50</td><td>71.19</td><td>66.04</td><td>73.62</td></tr><tr><td>Sec_Softmax</td><td>76.20</td><td>52.85</td><td><strong>73.93</strong></td><td>64.39</td><td>81.43</td><td>85.89</td><td>90.11</td><td>86.05</td><td>67.34</td><td>67.92</td><td>72.82</td></tr><tr><td>ANP-Boot</td><td>59.84</td><td>50.87</td><td>55.54</td><td>55.31</td><td>64.38</td><td>64.11</td><td>52.63</td><td>69.26</td><td>50.87</td><td>54.24</td><td>57.10</td></tr><tr><td>AE Baseline</td><td>51.06</td><td>39.61</td><td>54.80</td><td>58.54</td><td>63.07</td><td>57.99</td><td>50.59</td><td>68.74</td><td>41.91</td><td>53.76</td><td>52.62</td></tr><tr><td>MN Baseline</td><td>54.23</td><td>51.18</td><td>59.16</td><td>57.21</td><td>59.91</td><td>50.26</td><td>62.42</td><td>63.87</td><td>50.14</td><td>55.69</td><td>56.01</td></tr></tbody></table>

<table><caption>Table 3: Official results of different models on the evaluation test set. We were unable to present the results in the same format as Table 2 because we do not have access to all the scores.</caption><thead><tr><th rowspan="2">System</th><th colspan="2">ToyCar</th><th colspan="2">ToyTrain</th><th colspan="2">Fan</th><th colspan="2">Gearbox</th><th colspan="2">Bearing</th><th colspan="2">Slider</th><th colspan="2">Valve</th><th rowspan="2">Overall</th></tr><tr><th>AUC</th><th>pAUC</th><th>AUC</th><th>pAUC</th><th>AUC</th><th>pAUC</th><th>AUC</th><th>pAUC</th><th>AUC</th><th>pAUC</th><th>AUC</th><th>pAUC</th><th>AUC</th><th>pAUC</th></tr></thead><tbody><tr><td>Top rank [28]</td><td>88.45</td><td><strong>81.83</strong></td><td><strong>70.46</strong></td><td><strong>61.14</strong></td><td><strong>57.34</strong></td><td><strong>57.33</strong></td><td>86.04</td><td>64.22</td><td>68.85</td><td>54.45</td><td><strong>78.26</strong></td><td><strong>66.39</strong></td><td><strong>83.87</strong></td><td><strong>75.22</strong></td><td><strong>70.97</strong></td></tr><tr><td>Disent_Cat (S3)</td><td><strong>93.88</strong></td><td>78.67</td><td>58.23</td><td>54.73</td><td>48.17</td><td>50.34</td><td><strong>86.76</strong></td><td><strong>79.43</strong></td><td><strong>72.54</strong></td><td>61.86</td><td>73.64</td><td>60.70</td><td>83.72</td><td>62.93</td><td>67.57</td></tr><tr><td>Disent_Wt (S4)</td><td>93.30</td><td>75.47</td><td>57.30</td><td>54.93</td><td>46.93</td><td>50.33</td><td>86.34</td><td>78.47</td><td>71.96</td><td><strong>64.26</strong></td><td>75.94</td><td>64.29</td><td>83.05</td><td>64.01</td><td>67.49</td></tr><tr><td>MSL (S1)</td><td><strong>93.88</strong></td><td>78.67</td><td>55.53</td><td>54.33</td><td>44.50</td><td>50.84</td><td>86.47</td><td>68.54</td><td>69.94</td><td>61.64</td><td>73.64</td><td>60.70</td><td>78.51</td><td>66.08</td><td>65.66</td></tr><tr><td>MSL+ANP (S2)</td><td><strong>93.88</strong></td><td>78.67</td><td>54.92</td><td>54.22</td><td>44.29</td><td>50.97</td><td>82.37</td><td>70.76</td><td>69.94</td><td>61.64</td><td>75.96</td><td>62.40</td><td>77.69</td><td>65.39</td><td>65.57</td></tr><tr><td>MN Baseline</td><td>42.79</td><td>53.44</td><td>51.22</td><td>50.98</td><td>50.34</td><td>55.22</td><td>51.34</td><td>48.49</td><td>58.23</td><td>52.16</td><td>62.42</td><td>53.07</td><td>72.77</td><td>65.16</td><td>54.02</td></tr><tr><td>AE Baseline</td><td>61.18</td><td>60.21</td><td>43.14</td><td>49.36</td><td>41.16</td><td>50.12</td><td>61.92</td><td>51.95</td><td>59.93</td><td>53.95</td><td>58.95</td><td>54.16</td><td>54.26</td><td>51.30</td><td>52.94</td></tr></tbody></table>

<table><caption>Table 4: Detailed model setups. We indicate the best MSL for S1. The ensemble weights (Ens. wt.) of S2 and the disentanglement weights (Disent. wt.) of S4 were calculated via a grid search.</caption><thead><tr><th rowspan="2">Machine</th><th rowspan="2">MSL (S1)</th><th colspan="2">Ens. wt. (S2)</th><th colspan="2">Disent. wt. (S4)</th></tr><tr><th>MSL</th><th>ANP</th><th>w<sub>S</sub></th><th>w<sub>A</sub></th></tr></thead><tbody><tr><td>ToyCar</td><td>Disent_Cat</td><td>0.60</td><td>0.40</td><td>0.90</td><td>0.10</td></tr><tr><td>ToyTrain</td><td>MTL</td><td>0.70</td><td>0.30</td><td>0.00</td><td>1.00</td></tr><tr><td>Bearing</td><td>Sections only</td><td>1.00</td><td>0.00</td><td>1.00</td><td>0.00</td></tr><tr><td>Fan</td><td>ArcFace</td><td>0.95</td><td>0.05</td><td>0.15</td><td>0.85</td></tr><tr><td>Gearbox</td><td>Adversarial</td><td>0.65</td><td>0.35</td><td>0.80</td><td>0.20</td></tr><tr><td>Slider</td><td>Disent_Cat</td><td>0.70</td><td>0.30</td><td>0.90</td><td>0.10</td></tr><tr><td>Valve</td><td>Disent_Split</td><td>0.80</td><td>0.20</td><td>0.90</td><td>0.10</td></tr></tbody></table>

embeddings during inference and obtains an overall performance of 73.45%. Although the overall performance is lower than that of ArcFace, it showed improvements for all machines except Bearing and Fan. The Disentangled Weighted (Disent_Wt) model obtains the highest overall performance for a single model without ensembling and machine-specific losses. The MSL and MSL+ANP systems showed improvements on the development set, but, as discussed next, worsened performance on the evaluation set.

**Evaluation Set** results on the DCASE challenge blind test set are shown in Table 3. Interestingly, both of our disentanglement models performed better than MSL and MSL+ANP. As indicated in Table 4, the MSL system used multi-task learning for ToyTrain, only sections for Bearing, ArcFace for Fan, adversarial training for Gearbox, and a slightly modified version of disentanglement for valve, where every attribute was assigned a separate embedding space. In

all cases, our disentangled models outperformed the MSL systems. These observations convey that disentanglement is an effective technique for domain generalization.

However, optimizing disentanglement weights (Disent_Wt) on the dev set did not lead to improved performance over the simple concatenation approach (Disent_Cat). One hypothesis is that the optimized weights turned out to be slightly dataset-specific, which hurt generalization performance.

We ranked 5th out of 32 teams in the competition, obtaining an overall harmonic mean of 67.57%. We surpassed the baseline by 13.5% and trailed the top rank [28] by 3.4%. We surpassed the top rank for Bearing, Gearbox, and ToyCar(AUC). We believe that [28] adopted a better pipeline to train the feature extractor — (1) they average model weights from multiple epochs (2) they pre-train on all 7 machines and fine-tune for each machine. We hypothesize that our disentangled model can also be improved by incorporating these optimizations in the training pipeline.

## 5. CONCLUSION

In this study, we presented a disentangled multi-task learning framework for improved domain generalization in anomalous sound detection. We demonstrated that the disentangled model performs better than simple multi-task learning, or only learning based on domain-shared features (e.g., section indices). We also showed that there is increased flexibility in post-processing due to the multiple disentangled embedding spaces. In addition to the NN algorithm, we plan to explore other anomaly detection backends in the future, and to thoroughly evaluate the explainability of our approach.
## 6. REFERENCES

[1] Y. Koizumi, Y. Kawaguchi, K. Imoto, T. Nakamura, Y. Nikaido, R. Tanabe, H. Purohit, K. Suefusa, T. Endo, M. Yasuda, and N. Harada, "Description and discussion on DCASE2020 challenge task2: Unsupervised anomalous sound detection for machine condition monitoring," in *Proc. DCASE*, Nov. 2020, pp. 81–85.

[2] Y. Kawaguchi, K. Imoto, Y. Koizumi, N. Harada, D. Niizumi, K. Dohi, R. Tanabe, H. Purohit, and T. Endo, "Description and discussion on dcase 2021 challenge task 2: Unsupervised anomalous detection for machine condition monitoring under domain shifted conditions," in *Proc. DCASE*, Nov. 2021, pp. 186–190.

[3] K. Dohi, K. Imoto, N. Harada, D. Niizumi, Y. Koizumi, T. Nishida, H. Purohit, T. Endo, M. Yamamoto, and Y. Kawaguchi, "Description and discussion on DCASE 2022 challenge task 2: Unsupervised anomalous sound detection for machine condition monitoring applying domain generalization techniques," *In arXiv e-prints: 2206.05876*, 2022.

[4] E. Marchi, F. Vesperini, F. Eyben, S. Squartini, and B. Schuller, "A novel approach for automatic acoustic novelty detection using a denoising autoencoder with bidirectional LSTM neural networks," in *Proc. ICASSP*, Apr. 2015.

[5] E. Cakir and T. Virtanen, "Convolutional recurrent neural networks for rare sound event detection," in *Proc. DCASE*, Nov. 2017.

[6] Y. Koizumi, S. Saito, H. Uematsu, Y. Kawachi, and N. Harada, "Unsupervised detection of anomalous sound based on deep learning and the neyman–pearson lemma," *IEEE/ACM Trans. Audio, Speech, Lang. Process.*, vol. 27, no. 1, pp. 212–224, 2018.

[7] T. Hayashi, T. Komatsu, R. Kondo, T. Toda, and K. Takeda, "Anomalous sound event detection based on wavenet," in *Proc. EUSIPCO*, Sept. 2018, pp. 2494–2498.

[8] K. Suefusa, T. Nishida, H. Purohit, R. Tanabe, T. Endo, and Y. Kawaguchi, "Anomalous sound detection based on interpolation deep neural network," in *Proc. ICASSP*, May 2020, pp. 271–275.

[9] R. Giri, F. Cheng, K. Helwani, S. V. Tenneti, U. Isik, and A. Krishnaswamy, "Group masked autoencoder based density estimator for audio anomaly detection," in *Proc. DCASE*, Nov. 2020, pp. 51–55.

[10] G. Wichern, A. Chakrabarty, Z.-Q. Wang, and J. Le Roux, "Anomalous sound detection using attentive neural processes," in *Proc. WASPAA*, 2021, pp. 186–190.

[11] P. Primus, V. Haunschmid, P. Praher, and G. Widmer, "Anomalous sound detection as a simple binary classification problem with careful selection of proxy outlier examples," in *Proc. DCASE*, Nov. 2020, pp. 170–174.

[12] R. Giri, S. V. Tenneti, F. Cheng, K. Helwani, U. Isik, and A. Krishnaswamy, "Self-supervised classification for detecting anomalous sounds," in *Proc. DCASE*, Nov. 2020, pp. 46–50.

[13] J. A. Lopez, H. Lu, P. Lopez-Meyer, L. Nachman, G. Stemmer, and J. Huang, "A speaker recognition approach to anomaly detection," in *Proc. DCASE*, Nov. 2020, pp. 96–99.

[14] K. Wilkinghoff, "Sub-cluster adacos: Learning representations for anomalous sound detection," in *Proc. IJCNN*, 2021, pp. 1–8.

[15] T. Inoue, P. Vinayavekhin, S. Morikuni, S. Wang, T. H. Trong, D. Wood, M. Tatsubori, and R. Tachibana, "Detection of anomalous sounds for machine condition monitoring using classification confidence," in *Proc. DCASE*, Nov. 2020, pp. 66–70.

[16] J. Wang, C. Lan, C. Liu, Y. Ouyang, T. Qin, W. Lu, Y. Chen, W. Zeng, and P. Yu, "Generalizing to unseen domains: A survey on domain generalization," *IEEE Transactions on Knowledge and Data Engineering*, 2022.

[17] A. Veit, S. Belongie, and T. Karaletsos, "Conditional similarity networks," in *Proceedings of the IEEE conference on computer vision and pattern recognition*, 2017, pp. 830–838.

[18] J. Lee, N. J. Bryan, J. Salamon, Z. Jin, and J. Nam, "Metric learning vs classification for disentangled music representation learning," in *Proc. ISMIR*, 2020.

[19] Z. Ding and Y. Fu, "Deep domain generalization with structured low-rank constraint," *IEEE Transactions on Image Processing*, vol. 27, no. 1, pp. 304–313, 2017.

[20] S. Venkatesh, G. Wichern, A. Subramanian, and J. Le Roux, "Disentangled surrogate task learning for improved domain generalization in unsupervised anomalous sound detection," *DCASE2022 Challenge, Tech. Rep.*, July 2022.

[21] K. Wilkinghoff, "Utilizing sub-cluster adacos for anomalous sound detection under domain shifted conditions," *DCASE2021 Challenge, Tech. Rep.*, July 2021.

[22] K. Morita, T. Yano, and K. Tran, "Anomalous sound detection using cnn-based features by self supervised learning," *DCASE2021 Challenge, Tech. Rep.*, July 2021.

[23] M. Jones, D. Nikovski, M. Imamura, and T. Hirata, "Exemplar learning for extremely efficient anomaly detection in real-valued time series," *Data mining and knowledge discovery*, vol. 30, no. 6, pp. 1427–1454, 2016.

[24] N. Harada, D. Niizumi, D. Takeuchi, Y. Ohishi, M. Yasuda, and S. Saito, "ToyADMOS2: another dataset of miniature-machine operating sounds for anomalous sound detection under domain shift conditions," *arXiv preprint arXiv:2106.02369*, 2021.

[25] K. Dohi, T. Nishida, H. Purohit, R. Tanabe, T. Endo, M. Yamamoto, Y. Nikaido, and Y. Kawaguchi, "Mimii dg: Sound dataset for malfunctioning industrial machine investigation and inspection for domain generalization task," *arXiv preprint arXiv:2205.13879*, 2022.

[26] S. Chen, Y. Liu, X. Gao, and Z. Han, "Mobilefacenets: Efficient cnns for accurate real-time face verification on mobile devices," in *Chinese Conference on Biometric Recognition*. Springer, 2018, pp. 428–438.

[27] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, "Mobilenetv2: Inverted residuals and linear bottlenecks," in *Proceedings of the IEEE conference on computer vision and pattern recognition*, 2018, pp. 4510–4520.

[28] Y. Zeng, H. Liu, L. Xu, Y. Zhou, and L. Gan, "Robust anomaly sound detection framework for machine condition monitoring," *DCASE2022 Challenge, Tech. Rep.*, July 2022.

[29] J. Deng, J. Guo, N. Xue, and S. Zafeiriou, "Arcface: Additive angular margin loss for deep face recognition," in *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, 2019, pp. 4690–4699.
# DETECT WHAT YOU WANT: TARGET SOUND DETECTION

Helin Wang¹†, Dongchao Yang¹†, Yuexian Zou¹*, Fan Cui², Yujun Wang²

¹ADSPLAB, School of ECE, Peking University, Shenzhen, China
²Xiaomi Corporation, Beijing, China

## ABSTRACT

Human beings can perceive a target sound type from a multi-source mixture signal by the selective auditory attention, however, such functionality was hardly ever explored in machine hearing. This paper addresses the target sound detection (TSD) task, which aims to detect the target sound signal from a mixture audio when a target sound's reference audio is given. We present a novel target sound detection network (TSDNet) which consists of two main parts: A conditional network which aims at generating a sound-discriminative conditional embedding vector representing the target sound, and a detection network which takes both the mixture audio and the conditional embedding vector as inputs and produces the detection result of the target sound. These two networks can be jointly optimized with a multi-task learning approach to further improve the performance. In addition, we study both strong-supervised and weakly-supervised strategies to train TSDNet and propose a data augmentation method by mixing two samples. To facilitate this research, we build a target sound detection dataset (i.e. URBAN-TSD) based on URBAN-SED and UrbanSound8K datasets, and experimental results indicate our method could get the segment-based F scores of 76.3% and 56.8% on the strongly-labelled and weakly-labelled data respectively.

**Index Terms—** target sound detection, conditional embedding, weakly supervised, data augmentation

## 1. INTRODUCTION

Human beings has the ability to focus auditory attention on a particular sound in a multi-source environment, however, there were few studies in this area for machine hearing. In this paper, we initially define a target sound detection (TSD) task, which aims to recognize and localize target sound source within a mixture audio given a reference audio or/and a sound label. For example, the violin sound can be detected within a concert recording and the talking sound can be detected in a noisy cafe environment. TSD has a lots of potential applications, such as noise monitoring for smart cities [1], bioacoustic species and migration monitoring [2] and large-scale multimedia indexing [3]. To the best of our knowledge, this paper is the first attempt that explicitly tackles this problem.

There is one similar task with TSD, i.e. sound event detection (SED). SED aims to classify and localize all pre-defined sound events (e.g., train horn, car alarm) within an audio clip, which has

been widely studied [4, 5, 6, 7]. Compared to SED, TSD only focuses on detecting the event that we care about and ignores other events. Furthermore, TSD does not require to pre-define categories set, so it can be easy to extend to open domain detection. Other related tasks about extracting the target signal are speaker extraction [8, 9] which extracts the target speech from a mixture speech given a reference utterance of the target speaker, and acoustic events sound selection problems [10]. Different from them, our work focuses on the detection task, which is more suitable for many multimedia retrieval applications and the training data is easier to obtain.

To solve the TSD task, we propose a target sound detection network (i.e. TSDNet), and treat TSD as a binary classification problem for each frame of the audio, where the positive class is the sound event of interest, and the negative class is formed by the combination of all foreground and background interfering events and noises. By using reference audio, TSDNet can focus on the target sound and ignore other interference. More specifically, TSD-Net is composed of a conditional network which is used to generate a sound-discriminative conditional embedding vector from the reference audio, and a detection network which is applied to obtain binary-classification results at each frame with the conditional embedding vector and mixture audio as inputs. In order to get more sound-discriminative conditional embedding vector for TSD-Net, the conditional network is jointly optimized with both the TSD task and a sound event classification task. We further explore a data augmentation method for TSD, which randomly mixes two training samples to form a new training sample. In addition, we explore to train the TSDNet on both strong- and weakly-supervised TSD tasks. Here, weakly-supervised TSD task means the dataset only provides the presence or absence of target sound within the mixture sound, but not any timestamp information, which is more challenging.

Our contributions can be summarized as follows: (1) We propose a novel network for TSD that can be trained with both strong-labelled and weakly-labelled data. (2) We explore the jointly training method to get the robust conditional embedding vector, and propose a data augmentation method for the TSD task. (3) We establish a dataset for TSD, and our method achieves the segment-based F measures of 76.3% and 56.8% on the strongly-labelled and weakly-labelled data respectively.

## 2. PROPOSED METHOD

The architecture of our proposed network (TSDNet) is shown in Figure 1. TSDNet consists of two components: a conditional network which inputs the reference audio and outputs a conditional embedding vector, and a detection network which uses the conditional embedding vector and the mixture audio to get the detection results. In this section, we will describe the details of the whole network.

<footnote>† Indicates equal contribution.</footnote>

<footnote>* Corresponding Author: zouyx@pku.edu.cn</footnote>

<footnote>This paper was partially supported by Shenzhen Science & Technology Research Program (No: GXWD20201231165807007-20200814115301001; No: JSGG20191129105421211) and NSFC (No: 62176008).</footnote>
<figure><img src="image_116.png" /><figcaption>Figure 1: The architecture of our proposed TSDNet. Here, FC denotes the fully-connected layer. Reference signal indicates that the signal from the target class.</figcaption></figure>

<figure><img src="image_117.png" /><figcaption>Figure 2: Two fusion modules.</figcaption></figure>

## 2.1. Conditional Network

The purpose of the conditional network is to produce a global conditional embedding vector to represent the reference information. The input to the conditional network can be either a reference audio or a specific label, or both. In this paper, we focus on a reference audio as the input, for the reason that it is more challenging but easier to be transferred to new classes. Inspired by the powerful ability of extracting time-frequency robust features from audio with convolutional neural networks (CNNs) [11, 12, 13, 14], we apply a VGG-like CNN network for the conditional network, which uses the spectrogram feature as input and consists of 4 convolutional blocks with 64, 128, 256 and 512 output channels, respectively. Each convolutional block contains 2 convolutional layers with kernel size of 3 × 3, followed by downsampling with average pooling size of 2 × 2. Batch normalization [15] and ReLU function [16] are applied to all the convolutional layers. Global pooling layer [11] containing a global max-pooling function and a global average-pooling function is applied to get the global feature vector, which is then fed to a fully-connected layer to get the global conditional embedding vector with a fixed dimension of 128.

## 2.2. Detection network

We have built two types of detection network: strong- and weakly-supervised network. The details are given as follows.

**Strong-supervised network** The strong-supervised detection network is based on the state-of-the-art work for weakly supervised sound event detection by Dinkel et al. [4]. As shown in Figure 1, the network has two inputs: the conditional embedding vector and the mixture audio, which is trained to minimize the difference between the frame-level prediction results and the ground-truth labels. To be more specific, the network is composed of 4 convolutional layers, 1 Bi-GRU layer, and 2 fully-connected layers, each with a

LeakyReLU activation except the last layer, which has a sigmoid activation. Given the input feature of the mixture audio $x \in \mathbb{R}^{t \times f}$, where t and f denote the number of frames and the dimension of each frame respectively, the network aims to predict frame-level probabilities $\hat{p}_i = \mathbb{P}(Y = k | X = x_i, e; \phi)$ where $\phi$ denotes the trainable parameters of detection network, e denotes the embedding from the conditional network and $x_i$ denotes the i-th frame of the mixture audio $x$. Here, the value of k is 0 or 1. Given the ground-truth label $p_i \in \{0, 1\}$ for each frame, the strong-supervised network can be optimized by minimizing the binary cross-entropy (BCE) loss function:

$$ \mathcal{L}_{sed} = \sum_{i=1}^{t} (-p_i \log \hat{p}_i - (1-p_i) \log(1-\hat{p}_i)) \quad (1) $$

where t indicates the number of frames.

**Weakly-supervised network** The difference between strong-supervised network and weakly-supervised network is that the latter needs a pooling layer to get the clip-level prediction. We add a linear softmax (LinSoft) pooling layer [17] after the last layer of the strong-supervised detection network. It aims to predict a clip-level probability $\hat{P} = f_{LSF}(\hat{p}_1, \hat{p}_2, ..., \hat{p}_t)$ where $f_{LSF}(\cdot)$ denotes the linear softmax pooling function and T denotes the number of frames. Given the clip-level ground-truth label $P \in \{0, 1\}$, BCE loss is also applied as the loss function:

$$ \mathcal{L}'_{sed} = -P \log \hat{P} - (1-P) \log(1-\hat{P}) \quad (2) $$

## 2.3. Fusion Module

As Figure 2 shows, we employ two fusion strategies to combine the conditional embedding and the feature of mixture audio. One is repeatedly concatenating the conditional embedding to the feature of mixture audio in each time frame. The other is projecting the conditional embedding and the feature of mixture audio to the same dimension by a 1-D convolutional layer, and then using multiplication operation to fuse them.

## 2.4. Mixup-TSD

Following the advanced data augmentation methods [18, 19, 20] for audio classification, we propose a data augmentation method based on the widely-used mixup [18] for TSD task. The core idea is to create a new training sample by mixing a pair of two training samples. More specifically, we can generate a new training sample $(M_{new}, R_{new}, y_{new})$ from the data and label pair $(M_1, R_1, y_1)$ and $(M_2, R_2, y_2)$ by the following equation.

$$ X_{new} = \lambda X_1 + (1 - \lambda X_2) \quad (3) $$

$$ R_{new} = \lambda R_1 + (1 - \lambda R_2) \quad (4) $$

$$ y_{new} = \lambda y_1 + (1 - \lambda y_2) \quad (5) $$

where $M_1$ and $M_2$ are mixture audios, $R_1$ and $R_2$ are reference audios, $y_1$ and $y_2$ are the corresponding labels. Such Mixup-TSD works by generating lots of new training samples, and particularly, the target sound turns to be multi-label event instead of single-label event. We argue that this method can perform well at the early training stage, but needs a fine-tuning stage to fit for the single-label test. In practice, we set a ratio $\alpha$ to control whether to use Mixup-TSD while training, which means Mixup-TSD is applied with a probability of $\alpha$. We set a linear decay for $\alpha$ from 0.3 to 0 during the whole training iterations.
<figure><img src="image_118.png" /><figcaption>Figure 3: The process of data generation.</figcaption></figure>

## 2.5. Jointly Training

To get a more sound-discriminative conditional embedding vector for TSDNet, we propose a jointly optimizing method for the conditional network via a multi-task manner. As shown in Figure 1, the feature vector after the global pooling layer is fed to a fully-connected layer and a softmax nonlinearity to get the classification results. The conditional network can be optimized by minimizing the cross entropy loss between the predicted label and the ground truth label of the reference audio, along with the original detection task. The whole loss function is defined by formula (6).

$$ \mathcal{L}_{total} = \mathcal{L}_{sed} + \mathcal{L}_{cls} \quad (6) $$

where $\mathcal{L}_{sed}$ denotes the loss of the detection task, and $\mathcal{L}_{cls}$ denotes the loss of the classification task, which is the cross-entropy loss between the ground-truth class labels and the predicted class labels.

## 3. URBAN-TSD DATASET

As there was no dataset for TSD task, we build a dataset called URBAN-TSD based on URBAN-SED [21] and UrbanSound8K datasets [22]. URBAN-SED is a sound event detection dataset with an urban setting, containing 10 event labels. This dataset’s source material is the UrbanSound8K dataset containing 27.8 hours of data split into about 4-second clip segments. The URBAN-SED dataset encompasses 10,000 soundscapes generated using the Scaper soundscape synthesis library [21], which have been split into 6000 training, 2000 validation and 2000 test clips. The dataset contains mostly 10-second excerpts with strong labels, whereas each clip contains between one and nine events. We establish two types of dataset: strong-labelled dataset (URBAN-TSD-strong) and weakly-labelled dataset (URBAN-TSD-weak). The details of the number of samples are shown in Table 1.

### 3.1. URBAN-TSD-strong

We have built two strong datasets: URBAN-TSD-strong and URBAN-TSD-strong+. The details are given as follows.

**URBAN-TSD-strong dataset:** In this dataset, a sample includes three parts: a mixture audio, a reference audio and the strong label (the timestamp information of target sound). Mixture audios come from URBAN-SED dataset, and reference audio comes from UrbanSound8K. As shown in Figure 3, if there are *N* sound events in the mixture audio, we can generate *N* positive samples. For each positive sample, we randomly choose another sample that is in the same class from the UrbanSound8K as the reference audio.

**URBAN-TSD-strong+ dataset:** To further verify the ability of the model facing all negative frames, we generate samples that do not

<table><caption>Table 1: The number of strong- and weakly-labelled data.</caption><thead><tr><th>Type</th><th>Strong</th><th>Strong+</th><th>Weak</th></tr></thead><tbody><tr><td>Training</td><td>23106</td><td>29106</td><td>41059</td></tr><tr><td>Validation</td><td>7681</td><td>9681</td><td>13661</td></tr><tr><td>Test</td><td>7702</td><td>9702</td><td>13682</td></tr></tbody></table>

<table><caption>Table 2: The performance comparison between TSDNet and other SED methods. Note that multiplication fusion is applied.</caption><thead><tr><th>Method</th><th>Segment-based F1</th></tr></thead><tbody><tr><td>CRNN [23]</td><td>64.7</td></tr><tr><td>CDur [4]</td><td>64.8</td></tr><tr><td>CTrans [24]</td><td>64.51</td></tr><tr><td>SEDT-AQ-FT-P3 [25]</td><td>65.77</td></tr><tr><td>Ours</td><td><strong>73.1</strong></td></tr></tbody></table>

contain the target sound, which are called as negative samples. We add those samples because the mixture audio may not contain our target sound in the real world. The process of generating negative samples can summarize as: For any mixture audio from URBAN-SED, we randomly choose a reference audio whose sound events do not happen in the mixture audio. The timestamp information (label) of negative samples are all set as 0.

### 3.2. URBAN-TSD-weak

We also build a weakly-labelled dataset which is easier to obtain but more challenging. Comparing with strong-labelled dataset, there is no onset or offset time provided. The label is set as 1 (which indicates target sound happens in the mixture audio) or 0 (which indicates target sound does not happen in the mixture audio) for the whole audio clip. Similarly, for any mixture audio from URBAN-SED, if *N* events happen, we can generate *N* positive samples. For weakly-labelled dataset, negative samples are very important for training, and we generate the same number of negative samples as positive samples.

## 4. EXPERIMENTS

### 4.1. Experimental setups

**Training the conditional network:** We pre-train the conditional network with the classification task on the UrbanSound8K dataset. All the raw audios are down-sampled to 44.1kHz and applied a Short Time Fourier Transform (STFT) with a window size of 400 samples, followed by a Mel-scaled filter bank on perceptually weighted spectrogram. After that, we get log-mel spectrogram and MFCC feature, and concatenate them as the input, which is the state-of-the-art solution<fnref n="1" />. The Adam optimizer [26] is used for a total of 50 epochs, with an initial learning rate of $1 \times 10^{-3}$.

**Training the detection network:** All the raw audios are down-sampled to 22.05kHz and applied a Short Time Fourier Transform (STFT) with a window size of 2048 samples, followed by a Mel-scaled filter bank on perceptually weighted spectrogram. This results in 64 Mel frequency bins and around 50 frames per second. The Adam optimizer [26] is used for 100 epochs, with an initial learning rate of $1 \times 10^{-3}$. Note that we only update the detection

<footnote n="1">¹https://www.kaggle.com/adinishad/urbansound-classification-with-pytorch-and-fun/notebook</footnote>
<table><caption>Table 3: The segment-based F-measure (%) of TSDNet with different fusion strategies. We carry out the experiments for three times and report the mean and standard deviation values.</caption><thead><tr><th>Method</th><th>strong</th><th>strong+</th><th>weak</th></tr></thead><tbody><tr><td>Concatenation</td><td>73.1±0.61</td><td>67.2±0.41</td><td>52.8±0.53</td></tr><tr><td>Multiplication</td><td><strong>73.1±0.19</strong></td><td><strong>69.3±0.23</strong></td><td><strong>53.0±0.32</strong></td></tr></tbody></table>

<table><caption>Table 4: The segment-based F-measure (%) of different settings for Mixup-TSD. Here, multiplication fusion is applied.</caption><thead><tr><th>Mixup-TSD setting</th><th>strong</th><th>strong+</th><th>weak</th></tr></thead><tbody><tr><td>fixed ratio 0</td><td>73.1</td><td>69.3</td><td>53.0</td></tr><tr><td>fixed ratio 0.2</td><td>74.9</td><td>71.1</td><td>55.2</td></tr><tr><td>fixed ratio 0.5</td><td>74.2</td><td>70.8</td><td>54.7</td></tr><tr><td>fixed ratio 0.8</td><td>74.0</td><td>70.4</td><td>53.6</td></tr><tr><td>fixed ratio 1.0</td><td>73.8</td><td>70.2</td><td>53.2</td></tr><tr><td>linear decay ratio</td><td><strong>75.1</strong></td><td><strong>71.6</strong></td><td><strong>55.8</strong></td></tr></tbody></table>

network with the detection loss.

**Mixup-TSD:** For all the experiments, we use the Mixup-TSD method on the spectrogram level as we find spectrogram-level method works better than waveform-level method. Under otherwise stated, the linear decay is set for $\alpha$ from 0.3 to 0.

**Metrics:** We use the segment-based F-measure [27] as the evaluation metric, which is the most commonly used metrics for sound event detection. All the F-scores are macro-averaged.

## 4.2. The performance comparison between TSDNet and other SED methods

In this part, we conduct experiments on URBAN-TSD-strong dataset to validate the effectiveness of our proposed method (TSDNet). We choose four previous state-of-the-art SED methods on URBAN-SED dataset. As for the baselines, we follow the model architecture in [24] to build the Transformer-based model, which is referred to as CTrans in this paper. CDur [4] is the backbone of our TSDNet's detection network. Table 2 reports the results. We can see that our TSDNet significantly improve the performance due to introducing the reference audio.

## 4.3. Experimental results of different fusion strategies

We evaluate our baseline method with different fusion strategies. Table 3 reports the segment-based F-measure on the three datasets. The multiplication gets larger mean values and smaller standard deviation values in all situations, which is a more effective fusion method than the concatenation. In addition, TSDNet performs the worst on the URBAN-TSD-weak dataset, for the reason that no timestamp information is available during training.

## 4.4. Experimental results of Mixup-TSD

We further evaluate the Mixup-TSD method on three datasets, and report the performance in Table 4. If no augmentation is used, TSDNet can obtain the segment-based F-measures of 73.1%, 69.3% and 53.0% respectively. We can see that the Mixup-TSD method can significantly improve the performance over the baseline under all settings. More specifically, for a fixed ratio $\alpha$, the performance tends to decrease with the ratio increasing, and the best performance

<table><caption>Table 5: The performance of jointly training. Note that multiplication fusion and Mixup-TSD are applied.</caption><thead><tr><th>Jointly training</th><th>Strong</th><th>Strong+</th><th>Weak</th></tr></thead><tbody><tr><td>×</td><td>73.1</td><td>69.3</td><td>53.0</td></tr><tr><td>✓</td><td><strong>76.3</strong></td><td><strong>72.2</strong></td><td><strong>56.8</strong></td></tr></tbody></table>

<table><caption>Table 6: Experimental results of open domain. Note that multiplication fusion is applied.</caption><thead><tr><th>Jackhammer</th><th>Siren</th><th>Street_music</th><th>Average</th></tr></thead><tbody><tr><td>40.9</td><td>42.2</td><td>48.0</td><td>43.7</td></tr></tbody></table>

can be achieved when the ratio is 0.2. We argue that the Mixup-TSD method can perform well at the early training stage, but it will change the detection target from single-label event to multi-label event so we need a fine-tuning stage to fit for the single-label test. With a linear decay ratio from 0.3 to 0, we can get the segment-based F-measures of 75.1%, 71.6% and 55.8% respectively.

## 4.5. Experimental results of jointly training

In addition, we explore whether jointly training can further improve the performance of target sound detection. Firstly, we initialize the TSDNet with a pre-trained model which is only trained by the detection task. Secondly, we fine-tune TSDNet by a multi-task manner introduced in Section 2.5. We set a learning rate of $1 \times 10^{-4}$ for the whole model, and the Adam optimizer [26] is used for a total of 30 epochs. Experimental results are shown in Table 5. We can see that TSDNet with jointly training obtains 76.3%, 72.2% and 56.8% respectively, which brings 4.3%, 4.2% and 7.1% improvement over the baseline respectively. We find that the weak-supervised training still performs much worse than the strong-supervised training. We will study more effective weak-supervised methods in the future.

## 4.6. Open domain target sound detection

Table 6 shows the results of TSDNet evaluated on the open domain. We choose three events (jackhammer, siren, street_music) as new classes, which does not occur in the training set. Specifically, we first exclude all of the data include the three events in the URBAN-TSD-strong dataset, named URBAN-TSD-strong-7. Then we train TSDNet on URBAN-TSD-strong-7. We can see that TSDNet can be applied to the new class, e.g., the F-score of street_music achieved 48.0% and the average F-score of the three new classes achieved 43.7%. We believe that adding more training data can further improve the performance of new class.

## 5. CONCLUSIONS

In this paper, we proposed a target sound detection network (TSDNet) which can be trained with both strong-supervised learning and weakly-supervised learning manners. In addition, our proposed Mixup-TSD data augmentation method and jointly training strategy further improve the performance of TSDNet. TSDNet is more applicable to real scenarios because it does not require prior knowledge about the number of sound events and easy to extend to the new events. In the future, we will explore more robust system. The source code and datasets of this work are released.<fnref n="2" />

<footnote n="2">²https://github.com/yangdongchao/Target-sound-event-detection</footnote>
## 6. REFERENCES

[1] J. P. Bello, C. Silva, O. Nov, R. DuBois, A. Arora, J. Salamon, C. Mydlarz, and H. Doraiswamy, “Sonyc: A system for the monitoring, analysis and mitigation of urban noise pollution,” *arXiv preprint arXiv:1805.00889*, 2018.

[2] D. Stowell and D. Clayton, “Acoustic event detection for multiple overlapping similar sources,” in *2015 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA)*. IEEE, 2015, pp. 1–5.

[3] S. Hershey, S. Chaudhuri, D. Ellis, J. Gemmeke, A. Jansen, R. C. Moore, M. Plakal, D. Platt, R. A. Saurous, B. Seybold, et al., “CNN architectures for large-scale audio classification,” in *IEEE international conference on acoustics, speech and signal processing (ICASSP)*. IEEE, 2017, pp. 131–135.

[4] H. Dinkel, M. Wu, and K. Yu, “Towards duration robust weakly supervised sound event detection,” *IEEE/ACM Transactions on Audio, Speech, and Language Processing*, vol. 29, pp. 887–900, 2021.

[5] L. Lin, X. Wang, H. Liu, and Y. Qian, “Specialized decision surface and disentangled feature for weakly-supervised polyphonic sound event detection,” *IEEE/ACM Transactions on Audio, Speech, and Language Processing*, vol. 28, pp. 1466–1478, 2020.

[6] Q. Kong, Y. Xu, W. Wang, and M. D. Plumbley, “Sound event detection of weakly labelled data with cnn-transformer and automatic threshold optimization,” *IEEE/ACM Transactions on Audio, Speech, and Language Processing*, vol. 28, pp. 2450–2460, 2020.

[7] Q. Kong, C. Yu, Y. Xu, T. Iqbal, W. Wang, and M. D. Plumbley, “Weakly labelled audioset tagging with attention neural networks,” *IEEE/ACM Transactions on Audio, Speech, and Language Processing*, vol. 27, no. 11, pp. 1791–1802, 2019.

[8] Q. Wang, H. Muckenhirn, K. Wilson, P. Sridhar, Z. Wu, J. R. Hershey, R. A. Saurous, R. J. Weiss, Y. Jia, and I. L. Moreno, “Voicefilter: Targeted voice separation by speaker-conditioned spectrogram masking,” *Proc. Interspeech*, pp. 2728–2732, 2019.

[9] M. Ge, C. Xu, L. Wang, E. S. Chng, J. Dang, and H. Li, “Multi-stage speaker extraction with utterance and frame-level reference signals,” in *IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. IEEE, 2021, pp. 6109–6113.

[10] T. Ochiai, M. Delcroix, Y. Koizumi, H. Ito, K. Kinoshita, and S. Araki, “Listen to what you want: Neural network-based universal sound selector,” *Proc. Interspeech*, pp. 1441–1445, 2020.

[11] Q. Kong, Y. Cao, T. Iqbal, Y. Wang, W. Wang, and M. Plumbley, “Panns: Large-scale pretrained audio neural networks for audio pattern recognition,” *IEEE/ACM Transactions on Audio, Speech, and Language Processing*, vol. 28, pp. 2880–2894, 2020.

[12] H. Wang, Y. Zou, D. Chong, and W. Wang, “Environmental sound classification with parallel temporal-spectral attention,” in *Proc. Interspeech*, 2020, pp. 821–825.

[13] H. Wang, D. Chong, D. Huang, and Y. Zou, “What affects the performance of convolutional neural networks for audio

event classification,” in *International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIW)*. IEEE, 2019, pp. 140–146.

[14] H. Wang, Y. Zou, and D. Chong, “Acoustic scene classification with spectrogram processing strategies,” in *Proceedings of 5th the Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE)*, 2020, pp. 210–214.

[15] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in *International conference on machine learning*. PMLR, 2015, pp. 448–456.

[16] V. Nair and G. Hinton, “Rectified linear units improve restricted boltzmann machines,” in *ICML*, 2010.

[17] Y. Wang, J. Li, and F. Metze, “A comparison of five multiple instance learning pooling functions for sound event detection with weak labeling,” in *IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. IEEE, 2019, pp. 31–35.

[18] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, “mixup: Beyond empirical risk minimization,” *arXiv preprint arXiv:1710.09412*, 2017.

[19] D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le, “SpecAugment: A simple data augmentation method for automatic speech recognition,” *Proc. Interspeech*, pp. 2613–2617, 2019.

[20] Y. Tokozume, Y. Ushiku, and T. Harada, “Learning from between-class examples for deep sound recognition,” in *International Conference on Learning Representations*, 2018.

[21] J. Salamon, D. MacConnell, M. Cartwright, P. Li, and J. P. Bello, “Scaper: A library for soundscape synthesis and augmentation,” in *IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA)*. IEEE, 2017, pp. 344–348.

[22] J. Salamon, C. Jacoby, and J. P. Bello, “A dataset and taxonomy for urban sound research,” in *Proceedings of the 22nd ACM International Conference on Multimedia*, 2014, pp. 1041–1044.

[23] I. Martín-Morató, A. Mesaros, T. Heittola, T. Virtanen, M. Cobos, and F. Ferri, “Sound event envelope estimation in polyphonic mixtures,” in *IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. IEEE, 2019, pp. 935–939.

[24] K. Miyazaki, T. Komatsu, T. Hayashi, S. Watanabe, T. Toda, and K. Takeda, “Weakly-supervised sound event detection with self-attention,” in *ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. IEEE, 2020, pp. 66–70.

[25] Z. Ye, X. Wang, H. Liu, Y. Qian, R. Tao, L. Yan, and K. Ouchi, “Sound event detection transformer: An event-based end-to-end model for sound event detection,” *arXiv preprint arXiv:2110.02011*, 2021.

[26] D. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in *ICLR*, 2015.

[27] A. Mesaros, T. Heittola, and T. Virtanen, “Metrics for polyphonic sound event detection,” *Applied Sciences*, vol. 6, no. 6, p. 162, 2016.
# MATCHING TEXT AND AUDIO EMBEDDINGS: EXPLORING TRANSFER-LEARNING STRATEGIES FOR LANGUAGE-BASED AUDIO RETRIEVAL

Benno Weck¹,², Miguel Pérez Fernández¹,², Holger Kirchhoff¹, Xavier Serra²

¹ Huawei Technologies, Munich Research Center, Germany
{firstname.lastname}@huawei.com

² Universitat Pompeu Fabra, Music Technology Group, Spain
{firstname.lastname}01@estudiant.upf.edu, xavier.serra@upf.edu

## ABSTRACT

We present an analysis of large-scale pretrained deep learning models used for cross-modal (text-to-audio) retrieval. We use embeddings extracted by these models in a metric learning framework to connect matching pairs of audio and text. Shallow neural networks map the embeddings to a common dimensionality. Our system, which is an extension of our submission to the Language-based Audio Retrieval Task of the DCASE Challenge 2022, employs the RoBERTa foundation model as the text embedding extractor. A pretrained PANNs model extracts the audio embeddings. To improve the generalisation of our model, we investigate how pretraining with audio and associated noisy text collected from the online platform Freesound improves the performance of our method. Furthermore, our ablation study reveals that the proper choice of the loss function and fine-tuning the pretrained models are essential in training a competitive retrieval system.

## 1. INTRODUCTION

The DCASE2022 challenge subtask 6b provides a platform to stimulate research in the underexplored problem domain of language-based audio retrieval [1]. The goal of this task is to find the closest matching audio recordings for a given text query. A possible application for this task is a search engine for audio files in which a user can enter a free-form textual description to retrieve matching recordings. Such systems need to draw a connection between the two modalities: audio and text.

Given the complex nature of both audio and text, we expect that a system can only perform well in this task if it can capitalise on a large amount of training data. Due to the novelty of the task, not many previous studies and systems exist for language-based audio retrieval and training data is still limited. We instead turn to the fields of machine listening, specifically audio tagging, and natural language processing to draw inspiration from related problems and make use of existing resources such as pretrained models. It has become a popular approach to use large-scale pretrained models in a transfer learning setup for tasks where only limited training data is available.

The goal of this work is to study a simple, generic cross-modal alignment system. Our approach should be able to process audio and text independently to be used in a cross-modal retrieval context. Therefore, we leverage the power of pretrained models and a metric learning framework to semantically link the two modalities. We limit the complexity of our approach by employing the pretrained models with fixed weights and only train shallow network architectures to perform the alignment. Additionally, this paper presents

<figure><img src="image_119.png" /><figcaption>Figure 1: Overview of the architecture of our system. An audio tower and a text tower process the respective input data separately and produce a single embedding.</figcaption></figure>

an analysis of our submission [2] to the Language-based Audio Retrieval Task of the DCASE2022 Challenge. With an ablation study, we investigate the impact of different training strategies on the performance of our system. This helps us to understand the differences in performance between our system and other submissions to the challenge.

The remainder of this paper is structured as follows. In the next section, we introduce the methodological framework of our system. Section 3 explains the experiments that lead to our challenge submission and Section 4 presents the results of the submitted systems. The results of additional experiments performed as an ablation study are discussed in Section 5. We summarise our findings in Section 6.

## 2. METHOD

We adopt a metric learning [3] framework in our approach, which differs from a classification scenario used in related tasks such as audio tagging. In a classification scenario, the outputs of a network are the predictions for the different classes and the features that characterise each of those classes remain in the intermediate layers of the network. However, in metric learning, the goal is to obtain those features directly, so that the output of the network can be used to measure the similarity between two different inputs. The
features learned by the system can be referred to as an 'embedding space'. For each input, a network trained with metric learning will return an embedding $Z \in \mathbb{R}^F$, where $F$ is the size of the embedding, which is a hyper-parameter.

Metric learning usually relies on 'positive' and 'negative' examples to teach the networks. Positive examples are pairs of inputs that share some similarities, e.g., two sounds of birds singing. Negative examples, on the other hand, contain dissimilar content, e.g., a recording of a bird singing and a car's ignition system. The positive examples should be 'closer' in the embedded space, while the negative ones should lie in different regions. In our case, positive examples are audios and their corresponding descriptions.

Our system consists of two components – an audio tower and a text tower – to separately process the audio and text input. Each tower is further divided into an encoder, $E(\cdot)$, and an embeddings' adapter, $A(\cdot)$. As the audio encoder $E_a$ and the text encoder $E_t$, we employ pretrained models. An overview of our method is presented in Figure 1.

More specifically, an audio input $X_a$ or a text input $X_t$ are processed by $E_a$ and $E_t$, respectively, as

$$ 
\begin{aligned}
Z_a &= E_a(X_a), \\
Z_t &= E_t(X_t),
\end{aligned}
\qquad (1) 
$$

where $Z_i \in \mathbb{R}^{T_i \times F_i}$, $i \in \{a, t\}$ is a sequence of $T_i$ intermediate representations with $F_i$ features provided by the pretrained model (i.e., an embedding sequence). Then, the adapters $A_a$ and $A_t$ will process $Z_a$ and $Z_t$ as

$$ 
\begin{aligned}
Z'_a &= A_a(Z_a), \\
Z'_t &= A_t(Z_t),
\end{aligned}
\qquad (2) 
$$

where $Z'_a, Z'_t \in \mathbb{R}^{F'}$ are single embeddings and $F'$ denotes their dimensionality. The intermediate embedding sequences $Z_a$ and $Z_t$ produced by the audio and text encoder respectively will differ in dimensionality. The main purpose of the adapters is to match the dimensionality of text and audio embeddings in order to enable comparisons. We use the metric learning techniques described above to align the embedded spaces $Z_a$ and $Z_t$, so during training the adapters will learn to bring both into a common embedding space.

We experimented with two different losses. The first is the contrastive loss [4], which we used for our submission to the DCASE2022 challenge. Given the cosine similarity $s$ between a pair of embeddings with labels $l_1$ and $l_2$, the contrastive loss is defined by:

$$ L_{\text{contrastive}} = \begin{cases} 1-s & \text{if } l_1 = l_2 \\ \max(0, s) & \text{otherwise.} \end{cases} \qquad (3) $$

The second loss that we use in our experiments is the Normalized Temperature-scaled Cross Entropy (NT-Xent) loss [5], which is used by the leading submissions in the DCASE2022 challenge. For a more concise explanation of this loss, we refer the reader to the technical reports of the top-ranked teams [6, 7].

For the final application as a text-to-audio retrieval system, we compute the embedding of the text query $Z'_t$ and compares it to all pre-computed embeddings $Z'_a$ of the audio items in the dataset by means of the cosine similarity. Ranking the audio items by their similarity score in descending order provides the retrieval results.

<table><caption>Table 1: Hand-picked examples of descriptions and text labels from the metadata of the FSD50k dataset.</caption><thead><tr><th>Description</th><th>Tags</th></tr></thead><tbody><tr><td>“Typing on a mechanical keyboard”</td><td>“click”, “keyboard”, “mechanical”, “computer”, “typing”, “button”</td></tr><tr><td>“Pouring liquid in a shot glass, picking it up, drinking & slamming it down (not too hard) on the table.”</td><td>“slam”, “glass”, “pour”, “drink”, “liquid”, “alcohol”, “shot”</td></tr><tr><td>“opening of shower curtain, turning shower on, water running, turning shower off, getting out”</td><td>“shower”, “water”, “bathroom”, “bathtub”, “human”</td></tr></tbody></table>

# 3. EXPERIMENTS

## 3.1. Datasets

As the main dataset in our work, we employ the development dataset provided for this challenge, Clotho v2 [8], and use its official splits for training, validation, and final evaluation (testing). We posit that the Clotho dataset is relatively small for the training of deep-learning-based retrieval systems and any system might benefit from additional training data. Datasets combining audio and text are scarce, however, and the few that exist besides Clotho are either specific to a certain domain (e.g., urban soundscapes only [9]) or their audio content is not freely accessible [10]. This is why we decided to use weakly aligned text and audio pairs collected from the online platform Freesound [11], which also served as the data source for Clotho. Freesound allows users to upload an audio recording along with a textual description and a set of tags. This type of metadata was used before to extend the training data of Clotho but in the context of an automated audio captioning task [12]. For simplicity and reproducibility, we limit ourselves to the dev subset of the FSD50k dataset [13]. We assume that the audios in this dataset closely resemble the challenge audio data as the dataset mainly comprises recordings of sound events. Moreover, similarly to Clotho, audio clips are not longer than 30 seconds. The descriptions and tags in the dataset contain rich information about the content of the audio clip as can be seen from the examples given in Table 1. Nevertheless, the text data is noisy and also contains some undesired text.<fnref n="1" /> To clean the descriptions we remove all HTML mark-up and limit each text to 500 characters in a pre-processing step. To form a 'sentence' out of the tags, we join them with a single white space in the order given in the dataset. The dev split of the FSDK50 dataset contains almost 44100 audio files and we use half of them. By using descriptions and tag sequences, we can extend the training data by 40966 text-audio pairs (more than twice the amount of caption-audio pairs in the training subset of Clotho). We refer to Clotho's data as 'clean' and FSD50k's data as 'noisy'.

## 3.2. Evaluation & Metrics

We evaluate the ranked retrieval results generated by our systems with the same four metrics as the challenge organisers. Specifically, we report three 'recall at k' metrics (Recall@1, Recall@5, Recall@10) and one 'mean average precision at k' (mAP@10), where a score for a given query is computed for the top-k retrieved results and all scores are averaged over the entire set of queries. We direct the reader to [14] for an in-depth explanation of the metrics.

<footnote n="1">¹For example: “CAUTION: THIS PACK IS A CHEAP HOME RECORD. (But this one sounds a bit better)”</footnote>
<table><caption>Table 2: Retrieval metrics for the four submitted systems, the two leading teams, and the challenge baseline. The 95% confidence intervals computed by jackknife resampling are given in parentheses. Results marked with * were reported by the challenge organisers.</caption><thead><tr><th rowspan="2"></th><th colspan="4">Development test set</th><th>Challenge test set</th></tr><tr><th>Recall@1</th><th>Recall@5</th><th>Recall@10</th><th>mAP@10</th><th>mAP@10</th></tr></thead><tbody><tr><td>Challenge baseline*</td><td>0.03</td><td>0.11</td><td>0.19</td><td>0.07</td><td>0.061</td></tr><tr><td>ensmbl_5* [6]</td><td>0.188</td><td>0.447</td><td>0.587</td><td>0.299</td><td>0.276</td></tr><tr><td>Mei_Surrey_1* [7]</td><td>0.150</td><td>0.400</td><td>0.530</td><td>0.260</td><td>0.251</td></tr><tr><td>ATAE</td><td>0.071 (0.064 - 0.078)</td><td>0.217 (0.206 - 0.228)</td><td>0.325 (0.312 - 0.337)</td><td>0.136 (0.128 - 0.143)</td><td>0.114</td></tr><tr><td>ATAE-ET</td><td>0.064 (0.057 - 0.070)</td><td>0.194 (0.184 - 0.205)</td><td>0.288 (0.275 - 0.300)</td><td>0.121 (0.114 - 0.128)</td><td>0.113</td></tr><tr><td>ATAE-EP-F</td><td>0.067 (0.061 - 0.074)</td><td>0.200 (0.189 - 0.210)</td><td>0.299 (0.286 - 0.311)</td><td>0.127 (0.120 - 0.134)</td><td>0.121</td></tr><tr><td>ATAE-NP-F</td><td>0.072 (0.065 - 0.079)</td><td>0.225 (0.214 - 0.236)</td><td>0.325 (0.313 - 0.338)</td><td>0.139 (0.131 - 0.146)</td><td>0.128</td></tr></tbody></table>

## 3.3. Implementation details

Our system is implemented by relying on the PyTorch [15] framework in connection with the pytorch-metric-learning package [16]. For the text processing, we employ the Transformers library [17] and use the pretrained distilroberta-base model as the text encoder. This model is a compressed version of the original RoBERTa model [18] created by a knowledge distillation procedure [19]. It is smaller and faster than the original variant while retaining high performance on downstream tasks. Similar to our previous work on audio captioning [20], we decided to use the penultimate layer as the intermediate embeddings $Z_t$. The extracted text embeddings have a dimensionality $F_t$ of 768.

For the audio processing, we use a pretrained PANNs model [21] as the audio encoder. We follow the authors' suggestion and compute embeddings by taking the post-activation output of the penultimate layer of their CNN14 model.<fnref n="2" /> All audio clips are re-sampled to a sampling rate of 32 kHz in a preprocessing step. The extracted intermediate audio embeddings $Z_a$ have a dimensionality $F_a$ of 2048.

We use simple feed-forward neural networks to adapt each embedding sequence to the common dimensionality. Both adapters consist of a two-layer perceptron with a layer size of 512 and a recified linear unit (ReLU) as activation function after the first layer. We use the average of all embeddings in a sequence as the final representation.

The system is optimised by minimising the contrastive loss with the Adam algorithm [22] ($\alpha = 0.001, \beta_1 = 0.9, \beta_2 = 0.999$, and $\epsilon = 10^{-8}$). We do not fine-tune the encoder models in our approach and only optimise the adapters. To form a minibatch we randomly select 32 audio-text pairs from the training set. We compute the loss for every possible combination of similar and dissimilar samples (including text-to-text and audio-to-audio pairs) and take the mean across all non-zero loss values. Every epoch the mAP@10 metric is computed on the validation dataset. We start training with a learning rate of 0.0001 and reduce it by a factor of 10 if no improvement was found for five epochs. Finally, the training is stopped after ten epochs with no improvement and the model weights are reverted to the checkpoint of the epoch with the highest score.

## 3.4. Submitted systems

We submit four different configurations of our system. All share the same model hyperparameter configurations but differ in the way

the available training data was used to train them. Specifically, we experiment with: 1. adding no external dataset in our training, 2. extending the training data with noisy data from the FSD50k dataset, 3. pretraining with noisy and clean data and later fine-tuning with clean data only, and 4. pretraining exclusively with noisy data and fine-tuning with clean data only.

In every training (also if we refer to it as pretraining or fine-tuning), we follow the optimisation procedure described above.

**ATAE: Aligned Text and Audio Embeddings** In its standard configuration, our system is trained solely with the challenge development dataset Clotho. We refer to it as 'Aligned Text and Audio Embeddings' or ATAE for short.

**ATAE-ET: Aligned Text and Audio Embeddings – Extended dataset for Training** Next, we want to investigate if adding extra training data helps to improve retrieval performance. To achieve this we combine the noisy FSD50k and the clean Clotho data into a single training dataset.

**ATAE-EP-F: Aligned Text and Audio Embeddings – Extended dataset for Pretraining – Fine-tuning** To balance out the potential negative effects of the noise in the training data, we fine-tune the trained ATAE-ET model by again training with the clean Clotho dataset.

**ATAE-NP-F: Aligned Text and Audio Embeddings – Noisy dataset for Pretraining – Fine-tuning** Finally, to be able to better judge the effect of the noisy data for pretraining, we use the datasets in two separate training stages. We first train a model on the noisy data and then fine-tune it on the clean dataset.

## 4. RESULTS

Table 2 compares the metrics achieved for our four systems with the challenge baseline and two of the leading submissions on the challenge development test set and the challenge test set. We follow the lead of the challenge organisers and report a jackknife approximated 95% confidence interval for each metric [23]. Based on the results on the development test set, we make the following observations. First, our approach produces good quality results even in the standard training setup (ATAE: mAP@10 = 0.136 for the development test set). Second, extending the challenge dataset with additional (noisy) training data significantly degrades retrieval performance (ATAE-ET: mAP@10 = 0.121). Third, even fine-tuning the

<footnote n="2">²Pretrained weights can be found at: https://doi.org/10.5281/zenodo.3987831</footnote>