MT3: Multi-Task Multitrack Music Transcription

Automatic Music Transcription ( AMT ) , inferring musical notes from raw audio , is a challenging task at the core of music understanding . Unlike Automatic Speech Recognition ( ASR ) , which typically focuses on the words of a single speaker , AMT often requires transcribing multiple instruments simultaneously , all while preserving fine-scale pitch and timing information . Further , many AMT datasets are “ low-resource ” , as even expert musicians find music transcription difficult and time-consuming . Thus , prior work has focused on task-specific architectures , tailored to the individual instruments of each task . In this work , motivated by the promising results of sequence-to-sequence transfer learning for low-resource Natural Language Processing ( NLP ) , we demonstrate that a general-purpose Transformer model can perform multi-task AMT , jointly transcribing arbitrary combinations of musical instruments across several transcription datasets . We show this unified training framework achieves high-quality transcription results across a range of datasets , dramatically improving performance for low-resource instruments ( such as guitar ) , while preserving strong performance for abundant instruments ( such as piano ) . Finally , by expanding the scope of AMT , we expose the need for more consistent evaluation metrics and better dataset alignment , and provide a strong baseline for this new direction of multi-task AMT.1 1 INTRODUCTION . Recorded music often contains multiple instruments playing together ; these multiple “ tracks ” of a song make music transcription challenging for both algorithms and human experts . A transcriber must pick each note out of the audio mixture , estimate its pitch and timing , and identify the instrument on which the note was performed . An AMT system should be capable of transcribing multiple instruments at once ( Multitrack ) for a diverse range of styles and combinations of musical instruments ( Multi-Task ) . Despite the importance of Multi-Task Multitrack Music Transcription ( MT3 ) , several barriers have prevented researchers from addressing it . First , no model has yet proven capable of transcribing arbitrary combinations of instruments across a variety of datasets . Second , even if such models existed , no unified collection of AMT datasets has been gathered that spans a variety of AMT tasks . Finally , even within current AMT datasets , evaluation is inconsistent , with different research efforts using different metrics and test splits for each dataset . Compounding this challenge , many music datasets are relatively small in comparison to the datasets used to train large-scale sequence models in other domains such as NLP or ASR . Existing opensource music transcription datasets contain between one and a few hundred hours of audio ( see Table 1 ) , while standard ASR datasets LibriSpeech ( Panayotov et al. , 2015 ) and CommonVoice ( Ardila et al. , 2020 ) contain 1k and 9k+ hours of audio , respectively . LibriSpeech alone contains more hours of audio than all of the AMT datasets we use in this paper , combined . Taken as a whole , AMT fits the general description of a “ low-resource ” task , where data is scarce . In this work , we provide a strong empirical contribution to the field by overcoming each of these barriers and enabling Multi-Task Multitrack Music Transcription ( MT3 ) . Our contributions include : 1We encourage reviewers to view more extensive transcription results , including audio examples , at https : //storage.googleapis.com/mt3/index.html . Unified framework for training : We define a tokenization scheme with a compact and flexible vocabulary to convert between model output tokens and multitrack MIDI files , enabling a sequence-tosequence approach inspired by Raffel et al . ( 2019 ) and Xue et al . ( 2020 ) that supports datasets with different combinations of instruments . This allows us to simultaneously leverage several datasets which were previously only used in isolation due to differences in instrumentation . Benchmark collection of diverse datasets : We assemble six multitrack AMT datasets , spanning a variety of dataset sizes , styles , and instrumentations . Together they form the largest known collection publicly available for multi-task AMT training . Consistent evaluation : We define standard test set splits and apply a consistent set of note-based metrics across all six datasets . We also introduce a new instrument-sensitive transcription metric to jointly evaluate note and instrument accuracy . SOTA Baseline : Training an off-the-shelf T5 architecture with our framework , we realize strong baseline models that achieve SOTA transcription performance on each individual multitrack dataset , outperforming prior dataset-specific transcription models as well as professional-quality DSP-based transcription software . Our model , which we refer to as MT3 , demonstrates very high instrument labeling accuracy across all six datasets , even when many instruments are simultaneously present , and is robust to the grouping of instruments . Improving low-resource AMT : By training a single model across a mixture of all six datasets , we find that MT3 performance dramatically improves for low-resource datasets over the baseline models ( up to 260 % relative gain ) , while preserving strong performance on high-resource datasets . 2 RELATED WORK . 2.1 TRANSFORMERS FOR SEQUENCE MODELING . The Transformer architecture , originally proposed in Vaswani et al . ( 2017 ) , has recently demonstrated strong performance across many sequence modeling tasks in several domains . For example , T5 ( Raffel et al. , 2019 ) demonstrated that many language tasks previously addressed with separate models could be addressed using a single text-to-text encoder-decoder Transformer model . Extending this approach , mT5 ( Xue et al. , 2020 ) used a single Transformer to model multiple languages , demonstrating that a unified architecture could also serve as a general multilingual model , leveraging high-resource language datasets to improve model performance on lower-resource datasets . Other prominent examples of Transformer-based architectures for sequence modeling include BERT ( Devlin et al. , 2018 ) and the GPT family of models , most prominently GPT-3 ( Brown et al. , 2020 ) . Transformers have also been applied to some audio modeling tasks . For example , Transformerbased models have been used for audio classification ( Gong et al. , 2021 ; Verma & Berger , 2021 ) , captioning ( Mei et al. , 2021 ) , compression ( Dieleman et al. , 2021 ) , speech recognition ( Gulati et al. , 2020 ) , speaker separation ( Subakan et al. , 2021 ) , and enhancement ( Koizumi et al. , 2021 ) . Transformers have also been used for generative audio models ( Dhariwal et al. , 2020 ; Verma & Chafe , 2021 ) , which in turn have enabled further tasks in music understanding ( Castellon et al. , 2021 ) . 2.2 MUSIC TRANSCRIPTION . Historically , music transcription research has focused on transcribing recordings of solo piano ( Poliner & Ellis , 2006 ; Böck & Schedl , 2012 ; Kelz et al. , 2016 ) . As a result , there are a large number of transcription models whose success relies on hand-designed representations for piano transcription . For instance , the Onsets & Frames model ( Hawthorne et al. , 2017 ) uses dedicated outputs for detecting piano onsets and the note being played ; Kelz et al . ( 2019 ) represents the entire amplitude envelope of a piano note ; and Kong et al . ( 2020 ) additionally models piano foot pedal events ( a piano-specific way of controlling a note ’ s sustain ) . Single-instrument transcription models have also been developed for other instruments such as guitar ( Xi et al. , 2018 ) and drums ( Cartwright & Bello , 2018 ; Callender et al. , 2020 ) , though these instruments have received less attention than piano . Although not as widespread , some multi-instrument transcription systems have been developed . For example , Manilow et al . ( 2020 ) presents the Cerberus model , which simultaneously performs source separation and transcription for a fixed and predefined set of instruments . Lin et al . ( 2021 ) also perform both separation and transcription , albeit based on an audio query input and with the addition of synthesis . ReconVAT ( Cheuk et al. , 2021 ) uses an approach based on U-Net and unsupervised learning techniques to perform transcription on low-resource datasets ; however , the model does not predict instrument labels , instead outputting a single pianoroll that combines all instruments into a single “ track ” . A similar limitation applies to the early transcription system introduced alongside the MusicNet dataset by Thickstun et al . ( 2016 ) . Tanaka et al . ( 2020 ) uses a clustering approach to separate transcribed instruments , but the model output does not include explicit instrument labels . In contrast , our model outputs a stream of events representing notes from an arbitrary number of instruments with each note explicitly assigned to an instrument ; it learns to detect the presence ( or absence ) of instruments directly from audio spectrograms ( see Figure 1 ) . The work most closely related to ours is Hawthorne et al . ( 2021 ) , which uses an encoder-decoder Transformer architecture to transcribe solo piano recordings . Here , we extend their approach to transcribe polyphonic music with an arbitrary number of instruments . We adhere to their philosophy of using components as close to “ off-the-shelf ” as possible : spectrogram inputs , a standard Transformer configuration from T5 , and MIDI-like output events . 3 TRANSCRIPTION MODEL . At its core , music transcription can be posed as a sequence-to-sequence task , where the input is a sequence of audio frames , and the output is a sequence of symbolic tokens representing the notes being played . A key contribution of this work is to frame multi-instrument transcription , where different source instruments are present in a single input audio stream , within this paradigm and allow the model to learn which instruments are present in the source audio using Transformer model paired with a novel vocabulary designed to support this general task . 3.1 TRANSFORMER ARCHITECTURE . A key contribution of our work is the use of a single generic architecture — Transformers via T5 ( Raffel et al. , 2019 ) — to address a variety of tasks previously tackled using complex , handcrafted , dataset-specific architectures . The T5 architecture is an encoder-decoder Transformer model which closely follows the original form in Vaswani et al . ( 2017 ) . In the T5 architecture , a sequence of inputs is mapped to a sequence of learned embeddings plus fixed positional embeddings ; we use absolute positional embeddings instead of the bucketed “ relative ” embeddings used in Raffel et al . ( 2019 ) to ensure that all positions can be attended to with equal resolution . The model uses a series of standard Transformer self-attention “ blocks ” in both the encoder and decoder . In order to produce a sequence of output tokens , the model uses greedy autoregressive decoding : an input sequence is fed in , the output token with the highest predicted probability of occurring next is appended to the sequence , and the process is repeated until an endof-sequence ( EOS ) token is produced . We use the T5 “ small ” model , which contains approximately 60 million parameters . While much larger models are commonly used in language modeling tasks , we found that increasing model size tended to exacerbate overfitting . Full details on the Transformer architecture used in this work are given in Appendix A. Additionally , we will make our code available along with the release of this paper at https : //anonymized/for/review . 3.2 MODEL INPUTS AND OUTPUTS . As shown in Figure 1 , MT3 uses log Mel spectrograms as inputs . For the outputs , we construct a token vocabulary inspired by the MIDI specification , which we refer to as “ MIDI-like ” because it contains a subset of the original MIDI2 specification ( 1996 ) ( e.g . our vocabulary does not represent “ control change ” MIDI events ) . This output is a modification of the vocabulary of Hawthorne et al . ( 2021 ) with the following differences : ( 1 ) addition of instrument change tokens , which allow for multiple instruments to be represented in a single event stream ; ( 2 ) removal of velocity , as most of our training datasets do not contain velocity annotations ( and there is no standardized method for coding velocity across datasets ) ; ( 3 ) “ ties ” to better handle notes that span multiple segments . The vocabulary , illustrated in Figure 2 , consists of the following token types : Instrument ( 128 values ) : Indicates which instrument the following messages should be directed to . The specified instrument will be used for all subsequent events until the next Instrument event . The choice of 128 distinct values is selected to match the original General MIDI specification , which contains 128 “ programs ” used to designate specific instruments . We further discuss the challenges of representing instruments using program numbers below . Note ( 128 values ) : Represents a note-on or note-off event for one of the 128 MIDI pitches . On/Off ( 2 values ) : Changes whether subsequent Note events are interpreted as note-on or note-off . Time ( 205 values ) : Indicates the absolute time location of one or more events within a segment , quantized to 10 ms intervals . This time applies to all subsequent events until the next Time event , which allows for an arbitrary number of notes to occur at a given time point . Time events must occur 2MIDI ( Musical Instrument Digital Interface ) is an industry standard music technology protocol used to represent musical data and allow communication between musical devices . ( https : //www.midi.org/ ) in chronological order . ( The number of possible Time values depends on the Transformer ’ s input sequence length , which is 2.048 seconds for all of our experiments . ) Drum ( 128 values ) : Represents a drum onset from one of 128 drum types in the General MIDI standard . Drums are not a focus of this work , but we include them for completeness . End Tie Section ( 1 value ) : Ends the “ tie ” section at the beginning of a segment ( see below ) . EOS ( 1 value ) : Used to indicate the end of a sequence . A key contribution of this work is the demonstration that this highly general and flexible output vocabulary can be used to learn a single transcription model that works well across different instruments , datasets , and orchestrations , without manual tuning of the model or vocabulary . In contrast , prior works have been limited to models which transcribe only a single instrument ( Hawthorne et al. , 2021 ) , contain separate transcription “ heads ” for a fixed set of instruments ( Manilow et al. , 2020 ) , or ignore the instrument dimension entirely and only transcribe the notes ( Cheuk et al. , 2021 ) . One limitation of sequence models applied to audio is that most audio sequences are too large to fit in memory when modeling using a Transformer architecture , which requires O ( n2 ) memory with respect to sequence length for the self-attention blocks . In order to address these constraints , we use the procedure described by Hawthorne et al . ( 2021 ) : audio is split into smaller , non-overlapping segments , with input spectrograms and event tokens extracted from each segment . The model processes each segment independently . The same procedure is used for both training and inference ; however , at inference time we take the additional step of concatenating the decoded events from all segments into a single sequence to reconstruct the full example . One issue with transcribing cropped audio segments independently is that a note may span multiple segments . While our basic approach is often able to handle such cases , we find that occasionally the model will forget to “ turn off ” a note . To ameliorate this problem , we introduce a “ tie ” section at the beginning of each segment where the model must declare which notes are already active ; that is , the model is trained to emit Program and Pitch tokens for already-active notes , followed by the End Tie Section token , followed by the events of the segment . When concatenating segments to reconstruct an entire transcribed example , we end any notes not explicitly declared in the tie section . This allows the model to fail gracefully when it detects a note-on in one segment but not the corresponding note-off in a subsequent segment .

Summary: This paper presents a universal automatic music transcription (AMT) model agnostic to the number of instrument tracks. The model is trained across various AMT datasets, each of which was used as a different AMT task, and less biased to the volume of individual datasets. This is enabled by the novel MIDI-like output data representation designed to include "instrument" tokens and also supplementary tokens such as "tie" or "EOS" tokens. The authors conducted comprehensive experiments with 6 datasets, calculated not only regular accuracy metrics (Frame F1, Onset F1, Onset-Offset F1) but also multi-instrument F1 which is a new metric for the universal AMT task. They significantly improved the accuracy over all datasets and also showed that the model trained with mixed datasets are generally more effective than those trained with a single dataset.