# Multi-Scale Features for Transformer Model to Improve the Performance of Sound Event Detection
## 1 Introduction
They have performed well in recognizing objects in images of varying scales by offsetting the scale variation using multi-level feature pyramids. Instead of explicitly extracting multi-scale features, they combine low-resolution layer feature maps with high-resolution feature maps of CNNs to produce multi-scale features for the object detection problem. Motivated by the success of the feature-pyramid network, a CRNN architecture with a feature-pyramid component was proposed to improve the performance of SED in [22]. In a recent study [23], the Transformer showed state-of-the-art performance in language modeling, and it was thought to outperform a conventional RNN in sequential data modeling. Owing to this successful result, the Transformer is now widely used in speech and audio processing [24; 25; 26]. The self-attention mechanism of the Transformer enables the learning of the long-range temporal dependencies of the data very efficiently, with less computational complexity and a larger number of computations that can be parallelized than the RNN. In [25], the Transformer was applied to SED and achieved better performance than the baseline CRNN in the DCASE challenge 2019 task 4, showing that the self-attention mechanism is also very promising for SED. As mentioned in the above, various approaches in deep learning have addressed the problem of modeling time correlation in sequential data. In particular, the problem is crucial in SED because the varied sound duration of competing classes is one of the major issues that degrade the performance of SED. This has been addressed in SED from two different viewpoints. One is to use multi-scale features in the DNN by either directly extracting multiple features of different scales from the spectral analysis of the sound signal or combining the multi-level feature maps of the CNN layers in a feature-pyramid framework. Another approach is to use an efficient DNN architecture to better model the time correlation. Representatively, the Transformer is very promising for improving the performance of SED compared to the conventional RNN. Since there has been little research that combines the two approaches, herein, we developed a DNN architecture that can take advantage of multi-scale features in the Transformer to further improve the performance of SED. Owing to their availability, unlabeled data are important for improving SED performance. A mean-teacher model has been developed and popularly used in recent studies [27; 28] as an efficient method for exploiting unlabeled data to reduce overfitting in the semi-supervised learning of deep learning models. In [28], a mean-teacher model based on CRNN was proposed for SED to allow a large amount of unlabeled data for training. It showed the best performance in the DCASE challenge 2018. Since unlabeled data are available for training, herein, we developed a mean-teacher model based on the Transformer encoder to demonstrate the effectiveness of the multi-scale features. Herein, we propose a method for applying multi-level features of the CNN in a DNN based on the Transformer encoder for SED. By using the Transformer model instead of the conventional CRNN, we expect that we can leverage the superior performance of the Transformer encoder over RNNs in modeling local and global context information. The multi-scale features will contribute to further improving the capability of the Transformer encoder in processing the time correlation information. To the best of our knowledge, this is the first time that multi-scale features have been used in the Transformer encoder-based DNN for SED. The remainder of this paper is organized as follows: In Section 2, we introduce the Transformer encoder used in this study. In Section 3, the architecture of the proposed neural network is explained. In Section 4, we introduce the databases and present the experimental results. Finally, Section 5 concludes the paper.