# Global Structure-Aware Drum Transcription Based on Self-Attention Mechanisms
## 2 Related Work
### Automatic Drum Transcription (ADT)  
Some studies have attempted to use knowledge learned from an extensive collection of unpaired drum scores to improve ADT. A language model can be trained from such data in an unsupervised manner and used to encourage a transcription model to estimate a musically natural drum pattern. [PERSON] et al. [27] used a template-based language model for classifying audio signals into a limited number of drum patterns with a support vector machine (SVM). [PERSON] et al. [28] proposed a framework of knowledge distillation [29], one way to achieve transfer learning [30], in which an NMF-based teacher model was applied to a DNN-based student model.  
Language models have been actively used in the field of ASR. In a classical ASR system consisting of independently trained acoustic and language models, the language model is used in combination with the acoustic model in the decoding stage to generate syntactically and semantically natural word sequences. The implementation of the decoder, however, is highly complicated. In an end-to-end ASR system having no clear distinction of acoustic and language models, only paired data can be used for training an integrated model. Transfer learning is a promising way of making effective use of a language model trained from huge unpaired text data [23]. For example, a pretrained language model is used for softening the target word distribution of paired speech data such that not only ground-truth transcriptions but also their semantically coherent variations are taken into account as target data in the supervised training [25].  
A frame-level language model has been used in AMT. [PERSON] et al. [31] used a deep belief network for modeling transitions of chord symbols and improved the chord recognition method based on NMF. [PERSON] et al. [32] used a language model for estimating the most likely chord sequence from the chord posterior probabilities estimated by an RNN-based chord recognition system. As pointed out in [33; 34], however, language models can more effectively be formulated at the tatum level for learning musically meaningful structures. [PERSON] et al. [35] used N-gram as a symbolic language model and  
Figure 1: The supervised training of a neural transcription model (encoder–decoder model) with global structure-aware regularization based on a pretrained language model.  
improved a DNN-based chord recognition system. [PERSON] et al. [36] used an RNN-based symbolic language model with a duration model based on the idea that a frame-level language model can only smooth the onset probabilities of chord symbols. [PERSON] et al. [37] investigated the predictive power of LSTM networks and demonstrated that a long short-term memory (LSTM) working at the level of 16 th note timesteps could express musical structures such as note transitions.  
A tatum-level language model has also been used in ADT. [PERSON] et al. [10] proposed a Bayesian approach using a DNN-based language model as a prior of drum scores. [PERSON] et al. [19] proposed a regularized training method with an RNN-based pretrained language model to output musically natural drum patterns. However, these tatum-level language models cannot learn global structures, although the drum parts exhibit repetitive structure in music signals.