MT3: Multi-Task Multitrack Music Transcription

Automatic Music Transcription ( AMT ) , inferring musical notes from raw audio , is a challenging task at the core of music understanding . Unlike Automatic Speech Recognition ( ASR ) , which typically focuses on the words of a single speaker , AMT often requires transcribing multiple instruments simultaneously , all while preserving fine-scale pitch and timing information . Further , many AMT datasets are “ low-resource ” , as even expert musicians find music transcription difficult and time-consuming . Thus , prior work has focused on task-specific architectures , tailored to the individual instruments of each task . In this work , motivated by the promising results of sequence-to-sequence transfer learning for low-resource Natural Language Processing ( NLP ) , we demonstrate that a general-purpose Transformer model can perform multi-task AMT , jointly transcribing arbitrary combinations of musical instruments across several transcription datasets . We show this unified training framework achieves high-quality transcription results across a range of datasets , dramatically improving performance for low-resource instruments ( such as guitar ) , while preserving strong performance for abundant instruments ( such as piano ) . Finally , by expanding the scope of AMT , we expose the need for more consistent evaluation metrics and better dataset alignment , and provide a strong baseline for this new direction of multi-task AMT.1 1 INTRODUCTION . Recorded music often contains multiple instruments playing together ; these multiple “ tracks ” of a song make music transcription challenging for both algorithms and human experts . A transcriber must pick each note out of the audio mixture , estimate its pitch and timing , and identify the instrument on which the note was performed . An AMT system should be capable of transcribing multiple instruments at once ( Multitrack ) for a diverse range of styles and combinations of musical instruments ( Multi-Task ) . Despite the importance of Multi-Task Multitrack Music Transcription ( MT3 ) , several barriers have prevented researchers from addressing it . First , no model has yet proven capable of transcribing arbitrary combinations of instruments across a variety of datasets . Second , even if such models existed , no unified collection of AMT datasets has been gathered that spans a variety of AMT tasks . Finally , even within current AMT datasets , evaluation is inconsistent , with different research efforts using different metrics and test splits for each dataset . Compounding this challenge , many music datasets are relatively small in comparison to the datasets used to train large-scale sequence models in other domains such as NLP or ASR . Existing opensource music transcription datasets contain between one and a few hundred hours of audio ( see Table 1 ) , while standard ASR datasets LibriSpeech ( Panayotov et al. , 2015 ) and CommonVoice ( Ardila et al. , 2020 ) contain 1k and 9k+ hours of audio , respectively . LibriSpeech alone contains more hours of audio than all of the AMT datasets we use in this paper , combined . Taken as a whole , AMT fits the general description of a “ low-resource ” task , where data is scarce . In this work , we provide a strong empirical contribution to the field by overcoming each of these barriers and enabling Multi-Task Multitrack Music Transcription ( MT3 ) . Our contributions include : 1We encourage reviewers to view more extensive transcription results , including audio examples , at https : //storage.googleapis.com/mt3/index.html . Unified framework for training : We define a tokenization scheme with a compact and flexible vocabulary to convert between model output tokens and multitrack MIDI files , enabling a sequence-tosequence approach inspired by Raffel et al . ( 2019 ) and Xue et al . ( 2020 ) that supports datasets with different combinations of instruments . This allows us to simultaneously leverage several datasets which were previously only used in isolation due to differences in instrumentation . Benchmark collection of diverse datasets : We assemble six multitrack AMT datasets , spanning a variety of dataset sizes , styles , and instrumentations . Together they form the largest known collection publicly available for multi-task AMT training . Consistent evaluation : We define standard test set splits and apply a consistent set of note-based metrics across all six datasets . We also introduce a new instrument-sensitive transcription metric to jointly evaluate note and instrument accuracy . SOTA Baseline : Training an off-the-shelf T5 architecture with our framework , we realize strong baseline models that achieve SOTA transcription performance on each individual multitrack dataset , outperforming prior dataset-specific transcription models as well as professional-quality DSP-based transcription software . Our model , which we refer to as MT3 , demonstrates very high instrument labeling accuracy across all six datasets , even when many instruments are simultaneously present , and is robust to the grouping of instruments . Improving low-resource AMT : By training a single model across a mixture of all six datasets , we find that MT3 performance dramatically improves for low-resource datasets over the baseline models ( up to 260 % relative gain ) , while preserving strong performance on high-resource datasets . 2 RELATED WORK . 2.1 TRANSFORMERS FOR SEQUENCE MODELING . The Transformer architecture , originally proposed in Vaswani et al . ( 2017 ) , has recently demonstrated strong performance across many sequence modeling tasks in several domains . For example , T5 ( Raffel et al. , 2019 ) demonstrated that many language tasks previously addressed with separate models could be addressed using a single text-to-text encoder-decoder Transformer model . Extending this approach , mT5 ( Xue et al. , 2020 ) used a single Transformer to model multiple languages , demonstrating that a unified architecture could also serve as a general multilingual model , leveraging high-resource language datasets to improve model performance on lower-resource datasets . Other prominent examples of Transformer-based architectures for sequence modeling include BERT ( Devlin et al. , 2018 ) and the GPT family of models , most prominently GPT-3 ( Brown et al. , 2020 ) . Transformers have also been applied to some audio modeling tasks . For example , Transformerbased models have been used for audio classification ( Gong et al. , 2021 ; Verma & Berger , 2021 ) , captioning ( Mei et al. , 2021 ) , compression ( Dieleman et al. , 2021 ) , speech recognition ( Gulati et al. , 2020 ) , speaker separation ( Subakan et al. , 2021 ) , and enhancement ( Koizumi et al. , 2021 ) . Transformers have also been used for generative audio models ( Dhariwal et al. , 2020 ; Verma & Chafe , 2021 ) , which in turn have enabled further tasks in music understanding ( Castellon et al. , 2021 ) . 2.2 MUSIC TRANSCRIPTION . Historically , music transcription research has focused on transcribing recordings of solo piano ( Poliner & Ellis , 2006 ; Böck & Schedl , 2012 ; Kelz et al. , 2016 ) . As a result , there are a large number of transcription models whose success relies on hand-designed representations for piano transcription . For instance , the Onsets & Frames model ( Hawthorne et al. , 2017 ) uses dedicated outputs for detecting piano onsets and the note being played ; Kelz et al . ( 2019 ) represents the entire amplitude envelope of a piano note ; and Kong et al . ( 2020 ) additionally models piano foot pedal events ( a piano-specific way of controlling a note ’ s sustain ) . Single-instrument transcription models have also been developed for other instruments such as guitar ( Xi et al. , 2018 ) and drums ( Cartwright & Bello , 2018 ; Callender et al. , 2020 ) , though these instruments have received less attention than piano . Although not as widespread , some multi-instrument transcription systems have been developed . For example , Manilow et al . ( 2020 ) presents the Cerberus model , which simultaneously performs source separation and transcription for a fixed and predefined set of instruments . Lin et al . ( 2021 ) also perform both separation and transcription , albeit based on an audio query input and with the addition of synthesis . ReconVAT ( Cheuk et al. , 2021 ) uses an approach based on U-Net and unsupervised learning techniques to perform transcription on low-resource datasets ; however , the model does not predict instrument labels , instead outputting a single pianoroll that combines all instruments into a single “ track ” . A similar limitation applies to the early transcription system introduced alongside the MusicNet dataset by Thickstun et al . ( 2016 ) . Tanaka et al . ( 2020 ) uses a clustering approach to separate transcribed instruments , but the model output does not include explicit instrument labels . In contrast , our model outputs a stream of events representing notes from an arbitrary number of instruments with each note explicitly assigned to an instrument ; it learns to detect the presence ( or absence ) of instruments directly from audio spectrograms ( see Figure 1 ) . The work most closely related to ours is Hawthorne et al . ( 2021 ) , which uses an encoder-decoder Transformer architecture to transcribe solo piano recordings . Here , we extend their approach to transcribe polyphonic music with an arbitrary number of instruments . We adhere to their philosophy of using components as close to “ off-the-shelf ” as possible : spectrogram inputs , a standard Transformer configuration from T5 , and MIDI-like output events . 3 TRANSCRIPTION MODEL . At its core , music transcription can be posed as a sequence-to-sequence task , where the input is a sequence of audio frames , and the output is a sequence of symbolic tokens representing the notes being played . A key contribution of this work is to frame multi-instrument transcription , where different source instruments are present in a single input audio stream , within this paradigm and allow the model to learn which instruments are present in the source audio using Transformer model paired with a novel vocabulary designed to support this general task . 3.1 TRANSFORMER ARCHITECTURE . A key contribution of our work is the use of a single generic architecture — Transformers via T5 ( Raffel et al. , 2019 ) — to address a variety of tasks previously tackled using complex , handcrafted , dataset-specific architectures . The T5 architecture is an encoder-decoder Transformer model which closely follows the original form in Vaswani et al . ( 2017 ) . In the T5 architecture , a sequence of inputs is mapped to a sequence of learned embeddings plus fixed positional embeddings ; we use absolute positional embeddings instead of the bucketed “ relative ” embeddings used in Raffel et al . ( 2019 ) to ensure that all positions can be attended to with equal resolution . The model uses a series of standard Transformer self-attention “ blocks ” in both the encoder and decoder . In order to produce a sequence of output tokens , the model uses greedy autoregressive decoding : an input sequence is fed in , the output token with the highest predicted probability of occurring next is appended to the sequence , and the process is repeated until an endof-sequence ( EOS ) token is produced . We use the T5 “ small ” model , which contains approximately 60 million parameters . While much larger models are commonly used in language modeling tasks , we found that increasing model size tended to exacerbate overfitting . Full details on the Transformer architecture used in this work are given in Appendix A. Additionally , we will make our code available along with the release of this paper at https : //anonymized/for/review . 3.2 MODEL INPUTS AND OUTPUTS . As shown in Figure 1 , MT3 uses log Mel spectrograms as inputs . For the outputs , we construct a token vocabulary inspired by the MIDI specification , which we refer to as “ MIDI-like ” because it contains a subset of the original MIDI2 specification ( 1996 ) ( e.g . our vocabulary does not represent “ control change ” MIDI events ) . This output is a modification of the vocabulary of Hawthorne et al . ( 2021 ) with the following differences : ( 1 ) addition of instrument change tokens , which allow for multiple instruments to be represented in a single event stream ; ( 2 ) removal of velocity , as most of our training datasets do not contain velocity annotations ( and there is no standardized method for coding velocity across datasets ) ; ( 3 ) “ ties ” to better handle notes that span multiple segments . The vocabulary , illustrated in Figure 2 , consists of the following token types : Instrument ( 128 values ) : Indicates which instrument the following messages should be directed to . The specified instrument will be used for all subsequent events until the next Instrument event . The choice of 128 distinct values is selected to match the original General MIDI specification , which contains 128 “ programs ” used to designate specific instruments . We further discuss the challenges of representing instruments using program numbers below . Note ( 128 values ) : Represents a note-on or note-off event for one of the 128 MIDI pitches . On/Off ( 2 values ) : Changes whether subsequent Note events are interpreted as note-on or note-off . Time ( 205 values ) : Indicates the absolute time location of one or more events within a segment , quantized to 10 ms intervals . This time applies to all subsequent events until the next Time event , which allows for an arbitrary number of notes to occur at a given time point . Time events must occur 2MIDI ( Musical Instrument Digital Interface ) is an industry standard music technology protocol used to represent musical data and allow communication between musical devices . ( https : //www.midi.org/ ) in chronological order . ( The number of possible Time values depends on the Transformer ’ s input sequence length , which is 2.048 seconds for all of our experiments . ) Drum ( 128 values ) : Represents a drum onset from one of 128 drum types in the General MIDI standard . Drums are not a focus of this work , but we include them for completeness . End Tie Section ( 1 value ) : Ends the “ tie ” section at the beginning of a segment ( see below ) . EOS ( 1 value ) : Used to indicate the end of a sequence . A key contribution of this work is the demonstration that this highly general and flexible output vocabulary can be used to learn a single transcription model that works well across different instruments , datasets , and orchestrations , without manual tuning of the model or vocabulary . In contrast , prior works have been limited to models which transcribe only a single instrument ( Hawthorne et al. , 2021 ) , contain separate transcription “ heads ” for a fixed set of instruments ( Manilow et al. , 2020 ) , or ignore the instrument dimension entirely and only transcribe the notes ( Cheuk et al. , 2021 ) . One limitation of sequence models applied to audio is that most audio sequences are too large to fit in memory when modeling using a Transformer architecture , which requires O ( n2 ) memory with respect to sequence length for the self-attention blocks . In order to address these constraints , we use the procedure described by Hawthorne et al . ( 2021 ) : audio is split into smaller , non-overlapping segments , with input spectrograms and event tokens extracted from each segment . The model processes each segment independently . The same procedure is used for both training and inference ; however , at inference time we take the additional step of concatenating the decoded events from all segments into a single sequence to reconstruct the full example . One issue with transcribing cropped audio segments independently is that a note may span multiple segments . While our basic approach is often able to handle such cases , we find that occasionally the model will forget to “ turn off ” a note . To ameliorate this problem , we introduce a “ tie ” section at the beginning of each segment where the model must declare which notes are already active ; that is , the model is trained to emit Program and Pitch tokens for already-active notes , followed by the End Tie Section token , followed by the events of the segment . When concatenating segments to reconstruct an entire transcribed example , we end any notes not explicitly declared in the tie section . This allows the model to fail gracefully when it detects a note-on in one segment but not the corresponding note-off in a subsequent segment .

Summary: The authors combine numerous automatic music transcription datasets to devise a training framework. They train an off-the-shelf T5 architecture on the combined data and outperform the current state-of-the-art reported on individual datasets. They report numerous experiments to demonstrate the robustness of the model on low resource datasets & out-of-dataset transcription against task-informed variations of F1-scores, including a novel multi-instrument-based F1-score introduced in the paper.