MT3: Multi-Task Multitrack Music Transcription
Authors: Joshua P Gardner
Ian Simon
Ethan Manilow
Curtis Hawthorne
Jesse Engel
Venue: ICLR 2022 Spotlight (2022)
Keywords: music transcription
transformer
multi-task learning
low resource learning
music understanding
music information retrieval

Abstract: Automatic Music Transcription (AMT), inferring musical notes from raw audio, is a challenging task at the core of music understanding. Unlike Automatic Speech Recognition (ASR), which typically focuses on the words of a single speaker, AMT often requires transcribing multiple instruments simultaneously, all while preserving fine-scale pitch and timing information. Further, many AMT datasets are ``low-resource'', as even expert musicians find music transcription difficult and time-consuming. Thus, prior work has focused on task-specific architectures, tailored to the individual instruments of each task. In this work, motivated by the promising results of sequence-to-sequence transfer learning for low-resource Natural Language Processing (NLP), we demonstrate that a general-purpose Transformer model can perform multi-task AMT, jointly transcribing arbitrary combinations of musical instruments across several transcription datasets. We show this unified training framework achieves high-quality transcription results across a range of datasets, dramatically improving performance for low-resource instruments (such as guitar), while preserving strong performance for abundant instruments (such as piano). Finally, by expanding the scope of AMT, we expose the need for more consistent evaluation metrics and better dataset alignment, and provide a strong baseline for this new direction of multi-task AMT.

Published as a conference paper at ICLR 2022
MT3: MULTI-TASK MULTITRACK MUSIC TRANSCRIPTION
Josh Gardner, Ian Simon, Ethan Manilow; Curtis Hawthorne, Jesse Engel Google Research, Brain Team
ABSTRACT
Automatic Music Transcription (AMT), inferring musical notes from raw audio, is a challenging task at the core of music understanding. Unlike Automatic Speech Recognition (ASR), which typically focuses on the words of single speaker, AMT often requires transcribing multiple instruments simultaneously. all while preserving fine-scale pitch and timing information. Further, many AMT datasets are "low-resource". as even expert musicians find music transcription difficult and time-consuming. Thus, prior work has focused on task-specific architectures, tai- lored to the individual instruments of each task. In this work, motivated by the promising results of sequence-to-sequee
transfer learning for low-resource Nat- ural Language Processing (NLP), we demonstrate that a general-purpose Trans- former model can perform multi-task AMT, jointly transcribing arbitrary com- binations of musical instruments across several transcription datasets. We show this unified training framework achieves high high-quality transcription results across a range of datasets, dramatically improving performance for low-resource instru- ments (such as guitar), while preserving strong performance for abundant instru- ments (such as piano). Finally, by expanding the scope of AMT, we expose the need for more consistent evaluation metrics and better dataset alignment, and pro- vide a strong baseline for this new direction multi-task AMT
INTRODUCTION
Recorded music often contains multiple instruments playing together; these multiple "tracks" of song make music transcription challenging for both algorithms and human experts. A transcriber must pick each note out the audio mixture, estimate its pitch and timing, and identify the instru- ment which the note was performed. An AMT system should be capable of transcribing multiple instruments at once (Multitrack) for a diverse range of styles and combinations of musical instru- ments (Multi-Task, Despite the importance of Multi-Task Multitrack Music Transcription (MT3), several barriers have prevented researchers from addressing it. First, no model has yet proven capa- ble of transcribing arbitrary combinations of instruments across variety of datasets. Second, such models existed, no unified collection of AMT datasets has been gathered that spans a variety of AMT tasks. Finally, even within current AMT datasets, evaluation is inconsistent, with different research efforts using different metrics and test splits for each dataset.
Compounding this challenge, many music datasets are relatively small in comparison the datasets used to train large-scale sequence models in other domains such as NLP or ASR. Existing open- source music transcription datasets contain between one and a few hundred hours of audio (see Ta- T. while standard ASR datasets LibriSpeech (Panayotov et al. 2015) and Common Voice (Ardila al. 2020) contain 1 and 9k+ hours of audio, respectively. I LibriSpeech alone contains more hours of audio than all of the AMT datasets we use in this paper, combined. Taken as whole, AMT fits the general description of a "low-resource" task, where data scarce.
In this work, we provide strong empirical contribution the field by overcoming each of these barriers and enabling Multi-Task Multitrack Music Transcription (MT3). Our contributions include:
Paul G. Allen School of Computer Science & Engineering. Work performed as Google Research intern. Interactive Audio Lab, Northwestem University. Work performed as a Google Student Researcher. We encourage reviewers to view more extensive transcription results, including audio examples, https://storage googl wttmmtt/mmmmtccccccmccccco idex. htm]

---

Published as a conference paper at ICLR 2022
MAESTRO
Cerberus4
GuiarSet
MT3
MusicNet
Slakh2100
URMP
Figure 1: MT3 is capable of transcribing an arbitrary number of instruments from raw audio spec- trograms. Shown here are real 4-second audio clips, pianorolls reconstructed from the model's tokenized output, and the corresponding instrument labels (additional Slakh2100 instruments omit- ted due to space). Note that in some cases, multiple notes predicted from a monophonic instrument (such clarinet French horn) reflects an ensemble containing multiple players of that instrument.
Unified framework for training: We define tokenization scheme with a compact and flexible vo- cabulary to convert between model output tokens and multitrack MIDI files, enabling sequence-to- sequence approach inspired by Raffel et al. (2019) and Xue et al. (2020) that supports datasets with different combinations of instruments. This allows us to simultaneously leverage several datasets which were previously only used in isolation due to differences in instrumentation.
Benchmark collection of diverse datasets: We assemble six multitrack AMT datasets, spanning variety of dataset sizes, styles, and instrumentations. Together they form the largest known collection publicly available for multi- task AMT training.
Consistent evaluation: We define standard test set splits and apply a consistent set of note-based metrics across all six datasets. We also introduce new instrument-senaen transcription metric to jointly evaluate note and instrument accuracy.
Training off-the-shelf T5 architecture with framework, realize strong baseline models that achieve SOTA transcription performance on each individual multitrack dataset, outperforming prior dataset-specific transcription models well professional-qualiyy DSP-based transcription software. Our model, which we refer to as MT3, demonstrates very high instrument labeling accuracy across all six datasets, even when many instruments are simultaneously present, and is robust to the grouping of instruments.
SOTA Baseline:
Improving low-resource AMT:
By training single model across a mixture of all six datasets, we find that MT3 performance dramatically improves for low-resource datasets over the baseline models (up to 260% relative gain), while preserving strong performance on high-resource datasets.
RELATED WORK
TRANSFORMERS FOR SEQUENCE MODELING
The Transformer architecture, originally proposed in Vaswani al. (2017), has recently demon- strated strong performance across many sequence modeling tasks in several domains. For example, TS (Raffel et al. 2019) demonstrated that many language tasks previously addressed with separate
2

---

Published as a conference paper at ICLR 2022
models could be addressed using single text-to-text encoder-decoder Transformer model. Extend- ing this approach, mT5 (Xue et al.) 2020) used a single Transformer to model multiple languages, demonstrating that unified architecture could also serve general multilingual model, lever- aging high-resource language datasets to improve model performance on lower-resource datasets. Other prominent examples of transformer-basee architectures for sequence modeling include BERT Devlin et al. 2018) and the GPT family of models, most prominently GPT-3 (Brown et al. 2020
Transformers have also been applied some audio modeling tasks. For example, Transformer- based models have been used for audio classification (Gong et al. [02T] Verma & Berger 2021) captioning (Mei et al., (22II) compression (Dieleman et 20.1), speech recognition (Gulati al, 2020), speaker separation (Subakan et al.) (2021), and enhancement (Koizumi et al.) (2021). Trans- formers have also been used for generative audio models (Dhariwal et al. 20200 Verma Chafe 2021. which in turn have enabled further tasks in music understanding (Castellon et al. 2021).
2.2 MUSIC TRANSCRIPTION
Historically, music transcription research has focused transcribing recordings of solo piano (Po- liner & Ellis 2006 Böck & Schedl, 2012 Kelz et al. 2016). As a result, there are large number of transcription models whose success relies on hand-designed representations for piano transcrip- tion. For instance, the Onsets & Frames model (Hawthorne et al. 2017) uses dedicated outputs for detecting piano onsets and the note being played: Kelz et al. (2019) represents the entire ampli- tude envelope of a piano note; and Kong et al. (2020) additionally ly models piano foot pedal events (a transcription models have also been developed for other instruments such guitar (Xi et al. (018)) and drums (Cartwright & Bello, 2018 Callender et al., 2020). though these instruments have received less attention than piano.
piano-specific way of controlling a note's sustain). Single-instrumer
Although not widespread, some multi-instrument transcription systems have been developed. For xxamll,,Manilow et al. (2020) presents the Cerberus model, which simultaneously performs source Manilow separation and transcription for fixed and predefined set of instruments. Lin et al. (221)) also perform both separation and transcription, albeit based on an audio query input and with the addition of synthesis. Recon (Cheuk et al., (02T) uses approach based on U-Net and unsupervised learning techniques to perform transcription on low-resource datasets; however, the model does not predict instrument labels, instead outputting single pianoroll that combines all instruments into single "track". A similar limitation applies to the early transcription system introduced alongside the MusicNet dataset by Thickstun al. (2016). Tanaaa al. (2020) uses a clustering approach to separate transcribed instruments, but the model output does not include explicit instrument labels. In contrast, our model outputs a stream of events representing notes from an arbitrary number of instruments with each note explicitly assigned to an instrument; it learns to detect the presence (or absence) of instruments directly from audio spectrograms (see Figure D.
The work most closely related to ours is Hawthorne et al. (2021), which uses encoder-decoder Transformer architecture to transcribe solo piano recordings. Here, we extend their approach to transcribe polyphonic music with an arbitrary number of instruments. We adhere to their philos- ophy of using components as close to "off-the-shelf" as possible: spectrogram inputs, a standard Transformer configuration from T5, and MIDI-like output events.
TRANSCRIPTION MODEL
At its core, music transcription can be posed as sequence-to-sequeee task, where the input is a sequence of audio frames, and the output being played. A key contribution of this work is to frame multi-instrrmmnt transcription, where different source instruments are present in a single input audio stream, within this paradigm and allow the model to learn which instruments are present in the source audio using Transformer model paired with novel vocabulary designed support this general task.
sequence of symbolic tokens representing the notes
3.1 TRANSFORMER ARCHITECTURE
A key contribution of our work is the use of a single generic architecture - Transformers via T5 (Raffel al. 2019 to address variety of tasks previously tackled using complex, handcrafted,
3

---

Published as a conference paper at ICLR 2022
Piano Roll
MIDI-Like Target/Output Tokens
<time aprog 26>
<ate
<note 24>
<time 0.63>
<ott> <nate 22>
Tolenizatioo
sprog 0>
Son?
shote 28>
<prog 20>
<note 27>
<time 1.55>
<prog
<nate
<time 1.60>
<prog
Detokenization
<on> <hote 33>
<time 1.75> <prug
<off> <hote 33>
<time 2.03> <prog 32>
<hoee 31>
Figure 2: Tokenizzatin//deokenniittionn as described in Section 3.2 MIDI data (left, represented here as multitrack "pianoroll") can be tokenized into MIDI-like target tokens for training (right). Output tokens using the same vocabulary can be deterministically decoded back into MIDI data.
dataset-specific architectures. The T5 architecture is an encoder-decoder Transformer model which closely follows the original form in Vaswani al. (2017).
In the T5 architecture, a sequence of inputs is mapped to a sequence of learned embeddings plus fixed positional embeddings; we use absolute positional embeddings instead of the bucketed "relative" embeddings used in Raffel et al. (2019) to ensure that all positions can be attended to with equal resolution. The model uses series of standard Transformer self-attention "blocks" in both the encoder and decoder. In order to produce sequence of output tokens, the model uses greedy autoregressive decoding: an input sequence is fed in, the output token with the highest predicted probability of occurring next is appended to the sequence, and the process is repeated until an end- oe-sequence (EOS) token is produced. We use the T5 "small" model, which contains approximately 60 million parameters. While much larger models are commonly used in language modeling tasks, we found that increasing model size tended to exacerbate overfitting. Full details on the Transformer architecture used in this work are given in Appendix A Additionally, we make our code available along with the release of this paper at httpps://g/ thub.
3.2 MODEL INPUTS AND OUTPUTS
As shown in Figure I MT3 uses log Mel spectrograms as inputs. For the outputs, we construct token vocabulary inspired by the MIDI specification, which refer as "MIDI-like" because it contains a subset of the original MIDI specification (1996) (c.g. our vocabulary does not represent "control change" MIDI events). This output is modification of the vocabulary Hawwhooree al. (2021) with the following differences: (1) addition of instrument change tokens, which allow for multiple instruments to be represented in single event stream; (2) removal of velocity, as most of our training datasets do not contain velocity annotations (and there is no standardized method for coding velocity across datasets); (3) "ties" to better handle notes that span multiple segments. The vocabulary, illustrated in Figure 2 consists of the following token types:
Instrument (128 values): Indicates which instrument the following messages should be directed to. The specified instrument will be used for all subsequent events until the next Instrument event. The choice of 128 distinct values is selected to match the original General MIDI specification, which contains 128 "programs" used to designate specific instruments. We further discuss the challenges of representing instruments using program numbers below.
Represents note-on note-off event for one of the 128 MIDI pitches. On/Off (2 values):
Note (128 values):
Changes whether subsequent Note events are interpreted as note-on or note-off.
Time (205 values): Indicates the absolute time location of one or more events within segment, quantized to 10 ms intervals. This time applies to all subsequent events until the next Time event, which allows for an arbitrary number of notes to occur given time point. Time events must oceur
MIDI (Musical Instrument Digital Interface) is industry standard music technology protocol used to represent musical data and allow communication between musical devices. (https: http://wwwww/ www.iiiiddr.......oo.oo.....
4

---

Published as a conference paper at ICLR 2022
in chronological order. (The number of possible Time values depends on the Transformer's input sequence length, which 2.048 seconds for all of our experiments.)
Drum (128 values): Represents a drum onset from of 128 drum types in the General MIDI standard. Drums are not a focus of this work, but we include them for completeness.
End Tie Section (1 value): Ends the "tie" section at the beginning of segment (see below).
EOS (1 value): Used to indicate the end of a sequence.
A key contribution of this work is the demonstration that this highly general and flexible output vocabulary can be used to learn a single transcription model that works well across different instru- ments, datasets, and orchestrations, without manual tuning of the model vocabulary. In contrast, prior works have been limited to models which transcribe only a single instrument (Hawthorne et al. 2021), contain separate transcription "heads" for fixed set of instruments (Manilow et al. 2020 or ignore the instrument dimension entirely and only transcribe the notes (Cheuk et al. 2021)
One limitation of sequence models applied to audio is that most audio sequences too large fit in memory when modeling using a Transformer architecture, which requires ((n²) memory with respect to sequence length for the self-attention blocks. In order to address these constraints, we use the procedure described by Hawthorne et al. (2021): audio is split into smaller, non-overiapping segments, with input spectrograms and event tokens extracted from each segment. The model pro- cesses each segment independently. The same procedure is used for both training and inference; however, at inference time we take the additional step of concatenating the decoded events from all segments into single sequence to reconstruct the full example.
One issue with transcribing cropped audio segments independently that a note may span multiple segments. While our basic approach is often able to handle such cases, we find that occasionally the model will forget to "turn off"" note. To ameliorate this problem, we introduce a "tic" section at the beginning of each segment where the model must declare which notes are already active; that is, the model is trained to emit Program and Pitch tokens for already-active notes, followed by the End Tie Section token, followed by the events of the segment. When concatenating segments to reconstruct an entire transcribed example, we end any notes not explicitly declared in the tie section. This allows the model to fail gracefully when it detects a note-on in one segment but the corresponding note-off in subsequent segment.
3.3 MULTI-TASK MIXTURE
In addition to removing the cumbersome task of constructing specialized architectures and loss functions for different instrumentations and datasets, our general output vocabulary also allows our model to be trained on a mixture of several datasets simultaneously, similar to how multilingual translation models such as mT5 are trained on several languages (Xue al. 2020). This approach not only simplifies model design and training, but also increases the amount and diversity of training data available to the model. As noted previously, scarcity of training data has been major challenge for prior AMT modeling efforts. This mixture approach has not previously been demonstrated in the music transcription literature; instead, prior works have often focused on training separate models for individual datasets (i.e. Cheuk et al. (202I)) We note that "mixing" here refers to including data from multiple datasets within a single training batch.
In order to balance model performance
low- and high-resource datasets, sampling strategy for the mixing as follows: if dataset has n, examples, we sample an example from m2,)0.3, )0.3, similar to mT5 (Xue et al. 2020). This has the effect
temperature
that dataset with probability (n: Σ, m2y) of increasing the frequency with which the model observes examples from low-resource datasets during training, while observing examples from high-resource datasets with lower frequency.
EXPERIMENTS
We conduct series of experiments to test our approach. In particular, evaluate the overall transcription quality of our model across six datasets including both high- and low-resource datasets, evaluate the effect of instrument groupings, and results identify labeling issues with certain
5

---

Published as a conference paper at ICLR 2022
Dataset
Hrs. Audio
Num. Songs
Num. Instr.
Instr. Per Song
Align Low-Resource
Synthetic
Drums
Slakh2100
969
1405
35
4-48
Good
Cerberus4
5413
1327
1
1
Good
MAESTROv3
199
1276
Good
MusicNet
34
330
1-8
Poor
GuitarSet
3
360
1
Good
URMP
44
14
2-5
Fair
Table Datasets used in this paper.
datasets. We also assess our models' generalization to out-of-domain data with a series of leave-one- dataset-out experiments and results to identify label quality issues in Section
4.1 DATASETS
Our experiments use six datasets of varying size, recording process, instrumentation, and genre. In addition to demonstrating the flexibility of our approach, this also allows us to compare number of different baseline models, each which can only be fairly applied to specific datasets, and to offer single SOTA baseline across all datasets using MT3. The six datasets described briefly below; we provide further information about these datasets in Table and Appendix B
MAESTROv3: MAESTROv3 (Hawthorne al. 2018) is collected from virtual classical pi- ano competition, where audio and detailed MIDI data are collected from performers playing on Disklavier pianos that electronically capture the performance of each note in real time.
Slakh2100: Slakh2100 consists audio generated by rendering MIDI files using professional- grade, sample-based synthesis software. Its construction is detailed in Manilow al. (2019). During training, we use a form of data augmentation to combine together different subsets of the individual- instrument mixes, which we describe in Appendix
Cerberus4 is derived from the 100 dataset, obtained by mixing all combinations of the four instruments (guitar, bass, drums, piano) in tracks where those instruments are active. These are also the instruments used by the 4-instrument Cerberus model of Manilow et al. (202)).
Cerberus4:
GuitarSet (Xi et al., 2018) is composed of live guitar performances of varied genre, tempo, and style, recorded using high-precision hexaphonic pickup that individually captures the sound of each guitar string. MIDI labels for each track are derived from these recordings.
GuitarSet:
MusicNet (Thickstun et al. 2016 consists of freely-licensed classical music record- ings from variety instruments and ensemble types paired with human-gent genaaattd transcriptions crowdsourced from expert human transcribers. Since labels are primarily aligned with dynamic time warping, they are less accurate than other datasets.
MusicNet:
The University of Rochester Multi-Modal Music Performance (URMP) Dataset (Li et al. 2018 is composed of multi-instrument classical pieces with diverse instrumentation. The individual instruments recorded separately and mixed, and the aligned MIDI labels come from human annotators who corrected fo curves derived from the p PINN (Mauch & Dixon 2014) algorithm.
URMP:
We discuss the label alignment quality of MusicNet and URMP in Appendix D.2 In the phrasing of the NLP literature, we will refer to GuitarSet, MusicNet, and URMP as "low-resource" datasets, as they contain only 3, 34, and 1.3 hours of total audio, respectively. This makes these datasets chal- lenging to learn from particularly MusicNet and URMP, which contain many distinct instruments with several instruments per track, shown in Table 1
4.2 EVALUATION
The metrics used to evaluate multi ieltiiinsttumentt transcription models in the literature are inconsistent, when ignoring the
single-instrumet distinction. For example, several variants of "FI score" can be computed using the standard library for music transcription evaluation mi eval (Raffel al. 2014), which differ based whether note offsets (the time at which

---

Published as a conference paper at ICLR 2022
note ends) should be considered in addition to pitch values and note onsets (the time at which note begins) in evaluating whether a prediction is correct.
To provide the fairest and most complete comparison to existing work, we evaluate models on each dataset using three standard measures of transcription performance: Frame FI, Onset FI, and Onset- Offset F1. We use the standard implementation of these metrics from the mi r eval 1 Python toolkit. For each metric, mi _eval uses bipartite graph matching to find the optimal pairing of reference and estimated notes, then computes precision, recall, and Fl score using the following criteria:
Frame F1 score uses a binary measure of whether a pianoroll-like representation of the predictions and targets match. Each second is divided into a fixed number of "frames" (we use 62.5 frames per second), and a sequence of notes is represented as binary matrix of size [frames x 128) indicating the presence or absence of an active note at given pitch and time.
Onset F1 score considers a prediction to be correct if it has the same pitch and is within +50 ms of a reference onset. This metric ignores note offsets.
Onset-Offset F1 score is the strictest metric commonly used in the transcription literature. In ad- dition to matching onsets and pitch as above, notes must also have matching offsets. The criterion for matching offsets is that offsets must be within 0.2 reference duration or 50 ms from each other, whichever greater: [offset_diff] S max (0.2 reference_duuaatin 50 ms) (Raffel et al., 2014).
While these three metrics are standard for music transcription models, they offer only a limited view multi-instrument model, as none consider which instrument is predicted to
of the performance play which notes in sequence. This is due to the facts that (1) most prior music transcription models were limited to single-instrument transcription, and (2) even most multi-instrument transcription models did not assign specific instruments to predicted notes; also believe the lack of true multi-instrument metric has led to this gap in prior work. Thus, we also propose and evaluate our models' performance using novel metric which we call multi-innttummett Fl.
Multi-instrument F1 adds to the Onset-Offset F1 score the additional requirement that the instru- ment predicted to play a note must match the instrument of the reference note. This is stricter metric than the MV2H metric proposed in McLeod & Steedman (2018), as MV2H ignores off- sets and also eliminates notes from ground-truth during evaluation of the instrument labels when the pitch is not correctly detected: instiminstrument F1 is also more directly related to the existing transcription metrics widely used in prior works.
Due to the limitations of previous models, it is often not possible to compute a mulli-insttummen FI score for previous works; as a result, we only provide this metric for our model (shown in Table 3
We also evaluate our models' ability to transfer to unseen datasets by conducting "leaonggllvveeeee (leave-one-datasee out" (LODO) training experiments. These results demonstrate the generality of our approach in transferring to entirely new datasets, and are also useful in evaluating the impact of the various datasets used on the final model performance.
Two of our datasets (Slakh2100, Cerberus4) contain drums When evaluating our model, we match reference and estimated drum hits using onset time and General MIDI drum type (as the concept of a "drum offset" is not meaningful); a more rigorous evaluation methodology focused specifically on drums can be found in Callender al. (2020).
4.2.1 BASELINES
For each dataset, we compare to one or more baseline models. In addition to comparing to previous works which developed machine learning models for multi-instrument transcription one more of our datasets, we also compare our results to a professional-quaal DSP software for polyphonic pitch transcription, Melodyne ! Details on our usage of Melodyne are provided in Section C
As a consequence of the -ataset-specific training and architectures mentioned above, not all models appropriate for all datasets. As result, we only provide results for baseline models on datasets containing the instruments for which the original model was designed: for example, we evaluate the Cerberus model (Manilow al., (2020) only on the Cerberus4 dataset, which contains the four instruments for which the model contains specific transcription heads, and GuitarSet, where we
https:/ /ww. www.eeeeeeeeeemmmo. eemny..com.

---

Published as a conference paper at ICLR 2022
Model
MAESTRO
Cerberus4
GuitarSet
MusicNet
Slakh2100
URMP
Frame FI
Hawthorne et al. 2021
0.66
Manilow et al. 2020
0.63
0.54
Cheuk et al. 2021)
0.48
Melodyne
0.41
0.39
0.62
0.13
0.47
0.30
MT3 (single dataset) MT3 (mixture)
0.88
0.85
0.82
0.60
0.78
0.49
0.86
0.87
0.89
0.68
0.79
0.83
Onset F1
Hawthomee et al. (2021)
0.96
Manilow et
0.67
0.16
Cheuk et al. 2021
0.29
Melodyne
0.52
0.24
0.28
0.04
0.30
0.09
MT3 (single dataset) MT3 (mixture)
0.96
0.89
0.83
0.39
0.76
0.40
0.95
0.92
0.90
0.50
0.76
0.77
Onset+Offsee Fl
Hawthorne et al. 2222
0.84
Manilww et al.
2020
0.37
0.08
Cheuk et 2021
0.11
Melodyne
0.06
0.07
0.13
0.01
0.10
0.04
MT3 (single dataset)
0.84
0.76
0.65
0.21
0.57
0.16
MT3 (mixture)
0.80
0.80
0.78
0.33
0.57
0.58
Mixture (1%)
-5.3
+5.2
+19.5
+54.0
+0.1
+263
Table 2: Transcription F1 scores for Frame, Onset, and Ofset+Offiset metrics defined in Section 4.2
Across all metries and all datasets, MT3 consistently outperforms the baseline systems compare against. Dataset mixing during training ("mixture")). specifically, shows large performance increase over single dataset training, especially for "low-resource" datasets like GuitarSet, MusicNet, and URMP. Percent increase over single-dataset training for Onset+Offset F1 is shown in the last row.
only use the output the model's "guitar" head. (While Cerberus was not originally trained on GuitarSet, Manilow al. (2020) uses GuitarSet evaluation dataset; compare Cerberus due to the lack of an alternative baseline for GuitarSet.) On MAESTRO and MusicNet, we compare to Hawthhomee et al. (02TT) and Cheuk et al. (202T) which trained those respective datasets.
Wherever possible, we provide results from baseline models computed on the same test split used to evaluate MT3. This may overestimate the performance of some baselines if tracks or track segments in our validation/test sets may have been included in the training sets of the baseline models (due to the lack of consistent train/est//alidation split for some of these datasets). In an effort address this issue for future work, we provide exact details to reproduce our train/test/validatio splits in Appendix B and give further details on the baselines used, including information on reproducing our results, in Section For each model, we compute the reported metrics using pretrained models provided by the original authors. The ability to provide evaluation results for single model across several datasets is additional benefit of our approach and contribution of the current work.
4.3 RESULTS
Our main results are shown in Table which compares our models' performance on the six datasets described above. Our model achieves transcription performance exceeding the current state of the art for each of the six datasets evaluated across all three standard transcription metrics (Frame, Onset, and Onset + Offset Fl), as shown in Table 2 This is particularly notable due to the fact, mentioned above, that each baseline model was specifically designed (in terms of architecture and loss function), trained, and tuned on the individual datasets listed. Additionally, to significantly advance the state the art on the three resource-limited datasets discussed above, SuitarSet, MusicNet, and URMP. Table also demonstrates large gain in performance the
model is able
8

---

Published as a conference paper at ICLR 2022
MIDI Grouping
MAESTRO
Cerberus4
GuitarSet
MusicNet
Slakh2100
URMP
Flat
0,81
0.74
0.78
0.33
0.48
0.62
MIDI Class
0.80
0.81
0.78
0.31
0.62
0.59
Full
0.82
0.76
0.78
0.34
0.55
0.50
Table 3: dulti-instrument Fl score for MT3 (mixture) trained and evaluated at different levels of instrument granularity. The Flat grouping treats all non-drum instruments as single instrument. This resembles the setup of many prior notes played by all instruments, without respect to their source. The MIDI Class grouping maps instruments to their MIDI class (Table (). This creates groupings of eight program numbers each, with general classes for piano, guitar, bass, strings, brass, etc. The Full grouping retains instruments' program numbers as annotated in the source dataset. This requires the model to distinguish notes played by e.g. violin, viola, cello, all of which grouped in the "strings" MIDI Class.
transcription works which transcribe
resource-limited datasets when using the mixture formulation of our task, particularly for the multi- instrument datasets MusicNet and URMP: the mixture performance leads to an Onset-Offset F1 gain of 54% on MusicNet and 263% on URMP. Our model outperforms other baselines specifically opti- mized for low-resource datasets, such as Cheuk et al. (2021). while also remaining competitive with or outperforming models tuned for large single-instrument datasets, i.e. Hawthorne et al. (2021).
Instruments "in the wild" come in many different forms, and labeling exactly which sound sources contain the same instrument is a necessary but nontrivial task. While the original General MIDI 1.0 specification (199)) provides 1:1 mapping of program numbers 128 instruments" and coarser grouping of these instruments into "classes" of eight program numbers (Table these map- pings necessarily reductive: not all instruments are represented in the original 128 instruments mapped to program numbers (e.g. ukulele), and other instruments (piano, organ, guitar) include multiple program numbers which may be desirable to treat as single instrument for the purposes of transcription (e.g. transcribing program numbers 32-39 as a single "bass" class). To explore the effect of instrument label granularity, we train and evaluate models with three levels of instrument groupings: Flat, MIDI Class, and Full, shown in Table 3 We evaluate models at these three grouping levels according to the multi-instrument transcription metric defined in Section 4.2
Table 3 demonstrates that our model makes few instrument label errors when it predicts onsets and offsets correctly, even at the highest level of granularity ("Full"), as the multi-instrument Fl scores close to the onset-offset Fl scores in Table 2 We also provide an example transcription in Figure 3. which shows the distinct instrument tracks for an input from the Slakh2100 dataset. In addition to transcription results, we provide further experimental results in Appendix [D] There, we assess the ability of our approach to generalize to unseen datasets by conducting a set of leave-one-datasee (LODO) experiments; we also provide evidence regarding the label quality of our datasets by varying the onset and offset tolerance threshold used to compute F1 scores.
5 CONCLUSION AND FUTURE WORK
In this work we have shown that posing multi-instrument music transcription as a sequence-to- sequence task and training a generic Transformer architecture simultancously on a variety of datasets advances the state of the art in mulli-instrument transcription, most notably in the low- resource scenario. We also introduced and applied a consistent evaluation method using note on- settoffiet+innstrumett FI scores, using a standard instrument taxonomy.
Our work suggests several future research directions. As labeled data for multi-instrumeet tran- scription with realistic audio is very expensive, transcription models may benefit from training on unlabeled data, in a self- or semi-supervised fashion. There may also be value in a variety of data augmentation strategies, e.g. mixing together unrelated examples to generate new training data. Finally, high-quality AMT models such as ours present new frontiers for other musical modeling tasks, such generative music modeling (i.e. Dhariwal et al. (2020); Huang al. (2018)); the tran- scriptions from our model could be used as training data for a symbolic music generation model.
https://en. wik htttps:/ee..iikieeda...om
Generaa http:////whh..../oooooooooo///an

---

Published as a conference paper at ICLR 2022
6 REPRODUCIBILITY STATEMENT
In conjunction with the release of this work, we will make our model code, along with the code we used to replicate prior baseline models, available at httpp:://gittuub..mmm///ggggg
7 ETHICAL CONSIDERATIONS
One limitation of our system (and the baseline systems against which compare) is that it is trained on and applicable only to music from the "Western tradition". The characteristic of Western music most relevant to this work is that it is typically composed of discrete notes belonging one of 12 pitch classes i.e. "C", "C#", "D", etc. As such, music that does not have a well-defined mapping onto these 12 pitch classes outside the scope of this work. This excludes many non- westee musical traditions such as Indian ragas and Arabic maqams, and Western genres like the blues that rely on microtonality. These types of music would be better suited to alternate representations e.g- single or multiple non-discretized pitch tracks. We note that such data should also be considered "low-resource" given the low availability of transcription datasets representing such traditions, and represent important area for future work. See Holzapfel et al. (2019) for user study on existing AMT systems this context), and Viraraghavan et al. (2020) for an approach to transcription in one non- non-Wester Western domain.
REFERENCES
Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gregor Weber. Common Voice: A measiicly-multliingen speech corpus. In Proceedings of the 12th Language Resources and Evaluation Conference, pp. 4218-4222, Marseille, France, May 2020. European Language Re- sources Association. ISBN 979-10-95546-3444
URL https://aclantthol org
2020.1rec- 1.520
MIDI Manufacturers Association. The complete MIDI 1.0 detailed specification. Los Angeles, CA, The MIDI Manufacturers Association, 1996.
Sebastian Böck and Markus Schedl. Polyphonic piano note transcription with recurrent neural networks. In 2012 IEEE international conference on acousties, speech and signal processing (ICASSP), pp. 121-124. IEEE, 2012.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, and Skye Wanderman-Milne. JAX: composable transformations of Python + NumPy programs, 2018 http://gittubb.oomgooogggea 4:16, 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.
Lee Callender, Curtis Hawthorne, and Jesse Engel. Improving perceptual quality of drum transcrip- tion with the expanded Groove MIDI Dataset. arXiv preprint arXiv:2004. 20t.20004400.8.. 2020.
Mark Cartwright and Juan Pablo Bello. Increasing drum transcription vocabulary using data synthe- sis. In Proc. International Conference on Digital Audio Effects (DAFx), pp. 72-79, 2018.
Rodrigo Castellon, Chris Donahue, and Percy Liang. Codified audio language modeling learns useful representations for music information retrieval. arXiv preprint arXiv:2 2021.
Kin Wai Cheuk, Dorien Herremans, and Li Su. Recon A semi-supervised automatic autooma music tran- scription framework for low-resource real-world data. arXiv preprint arXiv:2107.04954 2021.
Jacob Devlin, Ming- Mei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional Transformers for language understanding. arXiv preprint /8X.v:/810.04588 2018.
Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. Jukebox: A generative model for music. arXiv preprint arXiv:2005. 00344, 2020.
10

---

Published as a conference paper at ICLR 2022
Sander Dieleman, Charlie Nash, Jesse Engel, and Karen Simonyan. Variable-rate discrete represen- tation learning. arXiv preprint vrXiv:2103.06089 2021.
Yuan Gong, Yu-An Chung, and James Glass. AST: Audio Spectrogram Transformer. arXiv preprint 004::2004.01778,, 2021.
Anmol Gulati, James Qin, Chung-Chene Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al. Conformer: Convolution-augmenned Transformer for speech recognition. arXiv preprint wrXiv:2005.08100, 2020.
Curtis Hawthorne, Erich Elsen, Jialin Song, Adam Roberts, Ian Simon, Colin Raffel, Jesse Engel, Sageev Oore, and Douglas Eck. Onsets and Frames: Dual-objective piano transcription. arXiv preprint arXiv: 1/100 2017.
Curtis Hawthorne, Andriy Stasyuk, Adam Roberts, Ian Simon, Cheng-Zhi Anna Huang, Sander Dieleman, Erich Elsen, Jesse Engel, and Douglas Eck. Enabling factorized piano music modeling and generation with the MAESTRO dataset. arXiv preprint arXiv: 1810. 111..811..2227..1 12247, 2018.
Curtis Hawthorne, Ian Simon, Rigel Swavely, Ethan Manilow, and Jesse Engel. Sequence-to- sequence piano transcription with Transformers. arXiv preprint arXiv:2107.09142: 2021.
Jonathan Heek, Anselm Levskaya, Avital Oliver, Marvin Ritter, Bertrand Rondepierre, Andreas Steiner, and Mare van Zee. Flax: A neural network library and ecosystem for JAX. Version 0.3, 3, 2020.
Andre Holzapfel, Emmanouil Benetos, et al. Automatic music transcription and ethnomusicology: A user study. In In Proceedings of the 20th International Society for Music Information Retrieval Conference, ISMIR, 2019.
Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Noam Shazeer, Ian Simon, Curtis Hawthorne, Andrew M Dai, Matthew D Hoffman, Monica Dinculescu, and Douglas Eck. Music Transformer. arXiv preprint arXiv: ww....09..00.400 04000, 2018
Eric J Humphrey. Justin Salamon, Oriol Nieto, Jon Forsyth, Rachel M Bittner, and Juan Pablo Bello. JAMS: A JSON annotated music specification for reproducible MIR research. In ISMIR, pp. 591- 596, 2014.
Rainer Kelz, Matthias Dorfer, Filip Korzeniowski, Sebastian Böck, Andreas Arzt, and Gerhard Wid- mer. On the potential of simple framewise approaches to piano transcription. arXiv preprint
1r1.0:161..05153. 2016.
Rainer Kelz, Sebastian Böck, and Gerhard Widmer. Deep polyphonic ADSR piano note transcrip- tion. In ICASSP 2019-2019 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 246-250. IEEE, 2019.
Yuma Koizumi, Shigeki Karita, Scott Wisdom, Hakan Erdogan, John R Hershey, Llion Jones, and Michiel Bacchiani. DF-Conformer: Integrated architecture of Conv-TasNet and Conformer using linear complexity self-attention for speech enhancement. arXiv preprint
2021.
Qiuqiang Kong, Bochen Li, Xuchen Song, Yuan Wan, and Yuxuan Wang. High-resolution piano transcription with pedals by regressing onsets and offsets times. arXiv preprint 31X1:0001... 2020.
Bochen Li, Xinzhao Liu, Karthik Dinesh, Zhiyao Duan, and Gaurav Sharma. Creating multitrack classical music performance dataset for multimodal music analysis: Challenges, insights, and applications. IEEE Transactions on Multimedia, 21(2):522-535, 2018.
Liwei Lin, Qiuqiang Kong, Junyan Jiang, and Gus Xia. A unified model for zero-shot music source separation, transcription and synthesis. arXiv preprint arXiv:2108.03445. 2021.
Ethan Manilow, Gordon Wichern, Prem Seetharaman, and Jonathan Le Roux. Cutting music source separation some Slakh: A dataset study the impact of training data quality and quantity. In 2019 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), pp. 45-49 IEFE, 2019.
11

---

Published as a conference paper at ICLR 2022
Ethan Manilow, Prem Seetharaman, and Bryan Pardo. Simultaneous separation and transcription of mixtures with multiple polyphonic and percussive instruments. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 771-775. IEEE, 2020.
Matthias Mauch and Simon Dixon. PУIN: fundamental frequency estimator using probabilistic threshold distributions. In 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 659-663. IEEE, 2014.
Andrew McLeod and Mark Steedman. Evaluating automatic polyphonic music transcription. In ISMIR, pp. 42-49, 2018.
Xinhao Mei, Xubo Liu, Qiushi Huang, Mark D Plumbley, and Wenwu Wang. Audio Captioning Transformer. arXiv preprint v7Xiv2210.098817, 2021.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. LibriSpeech: an ASR corpus based public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5210- IEEE, 2015.
Graham E Poliner and Daniel PW Ellis. A discriminative model for polyphonic piano transcription. EURASIP Journal on Advances in Signal Processing, 2007:1-9, 2006.
Colin Raffel. Learning-boo baseed methods for comparing sequences, with applications to audio-to-
MIDI alignment and matching. Columbia University, 2016.
Colin Raffel, Brian McFee, Eric Humphrey, Justin Salamon, Oriol Nieto, Dawen Liang, Daniel PW Ellis,, and C Colin Raffel. mir_eval: transparent implementation of common MIR metrics. In In Proceedings of the 15th International Society for Music Information Retrieval Conference, ISMIR, 2014.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text Transformer. arXiv preprint arXiv: 2019.
Cem Subakan, Mirco Ravanelli, Samuele Cornell, Mirko Bronzi, and Jianyuan Zhong. Attention is all you need in speech separation. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 21-25. IEEE, 2021.
Keitaro Tanaka, Takayuki Nakatsuka, Ryo Nishikimi, Kazuyoshi Yoshii, and Shigeo Morishima. Multi-instrument music transcription based on deep spherical clustering of spectrograms and pitchgrams. In International Society for Music Information Retrieval Conference (ISMIR), pp- 327-334, 2020.
John Thickstun, Zaid Harchaoui, and Sham Kakade. Learning features of music from scratch. arXiv preprint arXiv:1611.09%8 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Iliia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems, pp. 5998-6008. 2017.
Prateek Verma and Jonathan Berger. Audio Transformers: Transformer architectures for large scale audio understanding. Adieu convolutions. arXiv preprint arXiv:2105.000 00335, 2021.
Prateek Verma and Chris Chafe. A generative model for raw audio using Transformer architectures. arXiv preprint arXiv:2 10. 16036, 2021.
Venkata Subramanian Viraraghavan, Arpan Pal, Hema Murthy, and Rangarajan Aravind. State- based transcription of components of carnatic music. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 811-811888 811-815. IEEE, 2020.
Qingyang Xi, Rachel M Bittner, Johan Pauwels, Xuzhou Ye, and Juan Pablo Bello. dataset for guitar transcription. In ISMIR, pp. 453-460, 2018
GuitarSet: A
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mT5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.1 /9944, 2020.
12

---

Published as a conference paper at ICLR 2022
Pred cted Flano Roll for Instrument: Pano
Pradicted Plan fall for Instrumant: Drawbar Organ
Predicted Rana Rell for Instrument Acoustik Guitar
Predicted Rane Roll for Instrument near Bass
Predictod Plano Roll for netrument Drums
Figure 3: 15-second excerpt of MT3 transcriptions from mix from the Slakh2100 dataset. Black lines indicate model input frames. Blue notes indicate "True Positive" notes with correct predicted onset, offset, pitch, and instrument. In this segment, the model achieves an Onset-Offsel F1 of 0.665. More extensive results from MT3 can be found the companion website https::/ storage. wow.lapp.......p.s...o com/ mt 3/tnm// index. html
13

---

Published as a conference paper at ICLR 2022
Model
MLP Dim
Num. Heads
Num. Layers
Embed. Dim.
LR
Params
TS Small
1024
512
le 3
93.7M
Table 4: Details of model architecture used.
A MODEL AND TRAINING DETAILS
We use the "small" model architecture described Raffel et (2019), with the modifications defined in the T5.1.1 recipe"] This is standard Transformer architecture, and we use the imple- mentation available in t which is built on FLAX (Heek et al. 2020 and JAX (Bradbury et al, 2020).
All mixture models are trained for IM steps using a fixed learning rate of 0.001. The dataset-specific models trained for 2 steps, these models tended to converge much faster, particularly on the smaller datasets. Due to computational constraints we also train the LODO models for 21 steps.
B DATASET DETAILS
This section describes the datasets used in the experiments throughout this work. Descriptive statis- ties for each dataset provided in Table [ However, because not all datasets provide of- ficial train-test split, and because we perform preprocessing and filtering to extract suitable multi- instrument transcription datasets from the unprocessed versions of some datasets, provide further details for reproducibility here.
B. MAESTROV3
The MAESTRO (MIDI and Audio Edited for Synchronous TRacks and Organization) v3 datase (Hawthorne et al. 2018) contains 198.7 hours of piano performances captured via a Disklavier piano equipped with MIDI capture device which ensures fine alignment (R:3ms) between note labels and audio waveforms. The MAESTRO dataset contains mostly classical music and only includes piano performances (no other instruments).
MAESTRO includes a standard train/validation/test split, which ensures that the same composition does not appear in multiple subsets. 962 performances are in the train set, 137 are in the validation set, and 177 are in the test set. More detailed statistics on the MAESTRO dataset are available at
B.2 SLAKH2100
The Lakh MIDI Dataset (Raffel, 2016) is collection of 176,581 unique MIDI files scraped from publicly-avaiie available sources on the Internet, spanning multiple genres. The Synthesized Lakh Dataset (Slakh, or Slakh2100 100) (Manilow et al. 2019). is a dataset constructed by creating high-quality ren- derings of 2100 files from Lakh MIDI using professional-qualige virtual instruments. The 2100 files selected all contain at least piano, bass, guitar, and drums, where each of these four instruments plays at least notes.
When training 100,h2100, choose 10 random subsets of at least 4 instruments from each of the 2100 MIDI files as a form of data augmentation, expanding the number of training examples by factor of 10 (though individual stems will in general appear in more than example).
We use the standard Slakh2100 train/validation/eest splits for all experiments.
https://goo.gle///
14

---

Published as a conference paper at ICLR 2022
B.3 CERBERUS4
We refer to as Cerberus4 another slice of the S1akh2100 dataset; in this case for each MIDI file we extract all subsets of instruments containing (exactly) one of each of piano, guitar, bass, and drums. This is intended to reflect the dataset construction in Manilow et al. (2020), but using entire tracks instead of shorter segments and with no additional criteria on instrument "activity"..
Cerberus4 contains 1327 tracks representing 542.6 hours of audio. We use the Slakh2100 train/test/validation split separate Cerberus4 tracks. The training set contains 960 tracks with 418.13 1. hours of audio, the test set contains 132 tracks with 46.1 hours of audio, and the validation set contains 235 tracks with 78.4 hours of audio.
B.4 GUITARSET
GuitarSe g a dataset consisting of high-quality guitar recordings and time-aligned annotations. GuitarSet contains 360 excerpts, which are the result of 6 guitarists each playing 30 lead sheets (songs) in two versions ("comping" and "soloing"). Those 30 lead sheets are a combination five styles (Rock, Singer-Songwriter. Bossa Nova, Jazz, and Funk), three progressions (12 Bar Blues, Autumn Leaves, and Pachelbel Canon), and two tempi (slow and fast). The original GuitarSet annotations provided in the JAMS format (Humphrey al. 2014), which convert to MIDI for use with standard evaluation libraries.
There is official train-test split for GuitarSet. We establish the following split: for every style, we use the first two progressions for train and the final for validation. For convenience, we provide the exact train/validation split for tracks as part of the open-source release for this paper. This split produces 478 tracks for training, and 238 for validation.
B.5 MUSICNET
MusicNe Thickstun et al. 2016) consists of 330 recordings of classical music with MIDI annota- tions. The annotations were aligned recordings via dynamic time warping, and were then verified by trained musicians.
The standard train/test split for MusicNet Thickstun et (2016) only contains 10 test tracks and no validation set. We perform our own random split of the dataset into rrain/alidattonntest sets, and we provide the exact track IDs in each split our open-source code release for this paper.
We discuss potential label quality issues with MusicNet in Appendix D
B.6 URMP
The University of Rochester Multi- Music Performance (URMP) dataset (Li et al. 2018 10 consists of audio, video, and MIDI annotation of multi-instrument musical pieces assembled from coordinated but separately recorded performances of individual tracks. That is, each part of each piece is recorded isolation by an individual performer in coordination with the other performers (to ensure complete isolation of audio). The resulting mix is produced by combining the individual instrument tracks.
The dataset includes 11 duets, 12 trios, 14 quartets, and 7 quintets. In total, there are 14 different instruments in the dataset, including strings (violin, viola, cello, double bass), woodwinds (flute, oboe, clarinct, bassoon, soprano saxophone, tenor saxophone), and brass (trumpet, horn, trombone, tuba).
The dataset also includes videos and sheet music, which
used in this paper.
We use the following pieces for validation: 1, 2, 12, 13, 24, 25, 31, 38, 39. The remaining pieces are used for training. This validation split reserves two duets; two trios; three quartets; and two quintets for the validation set, ensuring a diverse instrumentation in the validation split.
html
15

---

Published as a conference paper at ICLR 2022
We discuss potential label quality issues with URMP in Appendix D
C BASELINE DETAILS
Manilow et al. (2020): For this bascline, used model trained with the authors' original code for the Cerberus 4-instrument model (guitar, piano, bass, drums) on the slakh-redux dataset, which omits duplicate tracks included in the original release of Slakh2100. We use the same procedure for randomly cropping audio and filtering for active instruments described above and in the original Cerberus paper (Manilow al. 2020), using the public train/validdationfess splits for Slakh.
Cheuk et al. (2021): We use the authors' pretrained models and inference script provided at https: / / thub.....ttuuuguog thub. www.ccoo//i.i.. wttp//Kiiiaaiiiiaaa Cheuk wwt.ai/wwwwccceeeeeeeeeeeea correspondence with the authors, we divide the audio tracks from MusicNet into 20-second seg- ments for inference, and conduct evaluation on these segments directly. We discard any segments without any active notes in the ground-truth annotations, because the mir eval metrics are unde- fined without any notes in the reference track.
Due to resource limitations. following
TT| is professional-qualinn audio software tool designed to provide note-based analysis and editing of audio recordings. Melodyne contains multiple algorithms for polyphonic pitch tracking, including algorithms for "polyphonic sustain" (designed for instruments with slow decay, such as strings) and "polyphonic decay" (designed for instruments with a fast decay. For all results in this paper, we used Melodyne Studio version 5.1.1.003. The raw wav files for each dataset were imported into Melodyne, and the default warchttacking settings were used to transcribe the audio (which allows Melodyne to automatically select the algorithm most suited to a given audio file). Melodyne exports MIDI files directly, which were used for our downstream analysis.
Melodyne: Melodyne
Melodyne does not provide programmatic interface. Due to the large amount of manual effort required to perform large-scale transcription with Melodyne, for MAESTRO, Slakh10, Cerberus4, and GuitarSet, our analysis of Melodyne performed on random subset of 30 tracks from the test set for each of these datasets. For URMP and MusicNet we evaluate Melodyne on the entire test set.
Because Melodyne is proprietary third-party software tool, we are not available to provide further details on the exact algorithms used to transcribe audio.
D ADDITIONAL RESULTS
D.1 EVALUATING ZERO-SHIOT GENERALIZATION WITH LEAVE-ONE-DATASETT-OO
In order to evaluate the generalizability of our proposed model, we evaluate our model on a chal- (LODO) ex-
lenging zero-shot generalization task. The procedure in these seave-one-dataset-on periments is as follows: Let the Dr.,i represent an individual transcription dataset (e.g. MAESTRO), such that the full training set is Drr = U. Dar,; For each dataset Dj. j. we train an MT3 model using the same mixture procedure described above, but with training set Drr Drr D Then, we evaluate on each dataset D E Drr-
Since Slakh2100 and Cerberus4 are generated using the same subset of track stems and the same synthesis software, we jointly either include or exclude those datasets in our LODO experiments.
The results of this study are shown in Tables and 6 Table 5 shows that model is able to achieve nontrivial note prediction performance for most datasets, attaining multi-instrument Fl and onset-offset Fl scores which outperform the baseline models on each of the low-resource datasets (GuitarSet, URMP). For all of the datasets, our model achieves LODO onset F1 scores between 0.14 and 0.78. For the much more challenging iulti-instrument F1 score, our model's performance on the LODO task varies by dataset.
In general, these results show that our model can obtain non-trivial transcription performance even for datasets it has never seen during training, despite large differences in the sonic qualities, compo- sitional styles, and instrumentation of the zero-shot evaluation datasets. However, the LODO exper- iments also point to the sensitivity of the model to the absence of particular datasets, highlighting
https://// /ww. eelemmony..ommmm
www.mmocoo/ee//////eee www.cco//eenenee http://e/eeemeee/e//eeeeeeeee wdwne wwaatt//w/wattttooee 1a-mel m- a-meeoddye
16

---

Published as a conference paper at ICLR 2022
Evaluation Dataset GuitarSet
Left-Out Dataset
MAESTRO
Cerberus4
MusicNet
Slakh2100
URMP
Frame F1
None
0.86
0.87
0.89
0.68
0.79
0.83
MAESTRO
0.60
0.86
0.89
0.69
0.75
0.82
Cerberus4 + Slakh2100
0.87
0.55
0.87
0.67
0.55
0.78
GuitarSet
0.86
0.86
0.58
0.71
0.76
0.82
MusicNet
0.86
0.86
0.89
0.53
0.76
0.79
URMP
0.86
0.86
0.89
0.71
0.76
0.76
Onset FI
None
0.95
0.92
0.90
0.50
0.76
0.77
MAESTRO
0.28
0.76
0.78
0.35
0.52
0.57
Cerberus4 + Slakh2100
0.82
0.21
0.75
0.30
0.14
0.49
GuitarSet
0.81
0.76
0.32
0.36
0.53
0.59
MusicNet
0.80
0.76
0.78
0.18
0.53
0.54
URMP
0.81
0.76
0.79
0.36
0.53
0.23
Ofsett-Offstt-Poorrre Fl
None
0.80
0.80
0.78
0.33
0.57
0.58
MAESTRO
0.28
0.76
0.78
0.33
0.52
0.50
Cerberus4 + Slakh2100
0.82
0.07
0.75
0.29
0.02
0.42
GuitarSet
0.81
0.76
0.19
0.35
0.53
0.53
MusicNet
0.80
0.75
0.78
0.14
0.53
0.47
URMP
0.81
0.75
0.79
0.35
0.53
0.17
Table 5: saave-one-dataset-on transcription scores.
Eval Dataset GuitarSet
Training
MAESTRO
Cerberus4
MusicNet
Slakh2100
URMP
Frame FI
Full mixture
0.86
0.87
0.89
0.68
0.79
0.83
Zero-shot
0.60
0.55
0.58
0.53
0.55
0.76
Onset Fl
Full mixture Zero-shot
0.95
0.92
0.90
0.50
0.76
0.77
0.28
0.21
0.78
0.18
0.14
0.23
Onset++ffset+PPogrree Fl
Full mixture Zero-shot 0.28
0.80
0.80
0.78
0.33
0.57
0.58
0.07
0.19
0.14
0.02
0.17
Table 6: Zero-shot transcription scores.
the resource-constrained nature the available music transcription datasets even when combined. For example, the Slakh2100 100 Cerberus4 combination is the only dataset in our LODO experiments that contains bass and synthesizer; without training on those datasets, the model is unable to learn to identify these instruments.
D.2 ONSET-OFESET THRESHOLD SENSITIVITY ANALYSIS
There are many reasons that transcription dataset may be more difficult than another. In the course of our experiments, we observed potential errors in labeling for some our datasets, includ- ing incorrect onset/offset times, particularly in MusicNet and URMP. The original MusicNet paper estimated an error rate of around 4% Thickstun et al. (2016); the error rate of URMP annotations has not been investigated, our knowledge.
17

---

Published as a conference paper at ICLR 2022
0.8
Inset-Offset F1
0.6
Dataset
0.4
MusicNet
MAESTRO
GuitarSet
Cerberus4
0.2
URMP
Slakh2100
0.0
0.1
022
0.3
0.4
0.5
Threshold
Figure 4: Onset-Offset Fl performance of our model over varying thresholds for the Onset-Offset F1 metric described in Section 4.2 The default threshold of 50 ms is indicated by a dashed line.
While direct investigation of labeling errors is beyond the scope of this work, present some initial evidence regarding label timing errors in Figure 4 Here, we systematically increase the tolerance used for the Onset-Offset F1 metric (described in Section 4.2) over a grid of values for t [10ms, 500ms], and compute the Onset-Offsee F1 for MT3 using threshold t for both onset and offset. Our results are consistent with the presence of label timing errors both URMP and MusicNet: As the threshold increases, performance on datasets with high-quality timing labels tends level off to baseline value. However, performance on URMP and MusicNet continues to increase as the threshold increases, which is suggestive of large timing errors beyond the standard 50 ms threshold used to evaluate the performance of all models in the experiments in our work.
These results suggest that MusicNet and also potentially URMP may be affected by label timing is- sues which could affect the learning, quality, and generalizability of models trained on these datasets (particularly considering that the default threshold for correct prediction is only 50 ms, small tim- ing errors can significantly increase the difficulty of properly modeling a dataset with noisy timing labels). While there is visible evidence of labeling errors [2 in the MusicNet dataset upon inspection,
iee using the MusicNet inspector too [3 we are not aware of scholarly work which has formally investigated this important issue to date. We encourage further investigation into labeling issues on these datasets.
E MIDI CLASS GROUPINGS
Program numbers for the "MIDI Class" grouping results described in Table 3 match the original MIDI classes from the original specification, where MIDI group for program p corresponds to floor (p/8); we give a complete listing of the MIDI program number to MIDI class mappings in grouped by "class",
T However, for the Cerberus4 and SLAKH2100 datasets, instruments which is a categorization of groups of patches used to synthesize those instruments. We construct mapping of SLAKH "class" to MIDI program numbers, given in Table 7 These program numbers are applied to each "class" in the Cerberus4 and SLAKH2 100 datasets as a simple lookup, and the associated program numbers are used.
F OPEN-SOURCE IMAGE ATTRIBUTION
The instrument icons used in Figure I are used under the Creative Commons license via the Noun Project. We gratefully acknowledge the following creators of these images:
httto://disc.uu/p//3/33 htttss:/mmsiiiae wwspccorr. tiu... io/
18

---

Published as a conference paper at ICLR 2022
Slakh2100 Class
MT3 Instrument Token Number
Acoustic Piano
0
Electric Piano
Chromatic Percussion
Organ
Acoustic Guitar
24
Clean Electric Guitar
26
Distorted Electric Guitar
29
Acoustic Bass
32
Electric Bass
33
Violin
40
Viola
41
Cello
42
Contrabass
43
Orchestral Harp
46
Timpani
47
String Ensemble
48
Synth Strings
50
Choir and Voice
52
Orchestral Hit
55
Trumpet
56
Trombone
57
Tuba
58
French Horn
60
Brass Section
61
Soprano/Alto Sax
64
Tenor Sax
66
Baritone Sax
67
Oboe
68
English Horn
69
Bassoon
70
Clarinet
71
Pipe
73
Synth Lead
80
Synth Pad
88
Table 7: Mapping S1akh2100 "classes" MT3 Instrument Token numbers used for all experi- ments using the Slakh2100 dataset (i.e., all columns labeled Slakh2100 in experiments throughout this paper). Slakh2100 classes have slightly more granularity than the 16 MIDI Classes (see Table and our MT3 Instrument tokens are designed to roughly correspond MIDI program numbers.
19

---

Published as a conference paper at ICLR 2022
MIDI Program Numbers
Instruments
Piano
1-8
Acoustic Grand Piano, Bright Acoustic Piano, Electric Grand Piano, Honky-tonk Piano, Electric Piano 1, Electric Piano 2, Harpsichord, Clavinet
Chromatic Percussion
9-16
Celesta, Glockenspiel, Music Box, Vibraphone, Marimba, Xylophone, Tubular Bells, Dulcimer
Organ
17-24
Drawbar Organ, Percussive Organ, Rock Organ, Church Organ, Reed Organ, Accordion, Harmonica, Tango Accordion
Guitar
25-32
Acoustic Guitar (nylon), Acoustic Guitar (steel), Electric Guitar (jazz),
Electric Guitar (clean), Electric Guitar (muted), Electric Guitar (over- driven), Electric Guitar (distortion), Electric Guitar (harmonics)
Bass Acoustic Bass, Electric Bass (finger), Electric Bass (picked). Fretless
33-40
Bass, Slap Bass
Slap Bass 2, Synth Bass Synth Bass
Strings
41-48
Violin, Viola, Cello, Contrabass, Tremolo Strings, Pizzicato Strings, Or-
chestral Harp, Timpani
Ensemble
49-56
String Ensemble 1, String Ensemble 2, Synth Strings Synth Strings 2,
Choir Aahs, Voice Oohs, Synth Voice or Solo Vox, Orchestra Hit Brass
57-64
Trumpet, Trombone, Tuba, Muted Trumpet, French Horn, Brass Section, Synth Brass
Synth Brass
Reed
65-72
Soprano Sax, Alto Sax, Tenor Sax, Baritone Sax, Oboe, English Horn, Bassoon, Clarinet
Pipe
73-80
Piccolo, Flute, Recorder, Pan Flute, Blown bottle, Shakuhachi, Whistle, Ocarina
Synth Lead
81-88
Lead (square), Lead 2 (sawtooth), Lead 3 (calliope). Lead (chiff), Lead 5 (charang), Lead (space voice), Lead (fifths), Lead (bass and lead)
Synth Pad
89-96
Pad (new age or fantasia), Pad 2 (warm), Pad 3 (polysynth or poly, Pad 4 (choir), Pad (bowed glass bowed), Pad (metallic), Pad 7 (halo), Pad (sweep)
Synth Effects
97-111
FX I (rain), FX 2 (soundtrack), FX 3 (crystal), FX 4 (atmosphere), FX 5 (brightness), FX (goblins), FX (echoes or echo drops), FX 8 (sci-fi
star theme)
Other
105-112
Sitar, Banjo, Shamisen, Koto, Kalimba, Bag pipe, Fiddle, Shanai
Percussive
113-120
Tinkle Bell, Agogô, Steel Drums, Woodblock, Taiko Drum, Melodic Tom or 808 Toms, Synth Drum, Reverse Cymbal
Sound Effects
121-128
Guitar Fret Noise, Breath Noise, Seashore, Bird Tweet, Telephone Ring, Helicopter, Applause, Gunshot
Table 8: All instruments defined by the MIDI specification Association (199), grouped by MIDI Class (rows with grey background), program numbers, and their associated instrument names.
20

---

Published as a conference paper at ICLR 2022
Piano by Juan Pablo Bravo from the Noun Project.
Guitar by varvarvarvarra from the Noun Project.
Bass by Josue Calle from the Noun Project.
Drum Set by Sumyati from the Noun Project.
Oboe by Rank Sol from the Noun Project.
Clarinet by Pham Thanh Lôc from the Noun Project.
French Horn by Creative Stall from the Noun Project
Bassoon by Lars Meiertoberens from the Noun Project.
Flute by Symbolon from the Noun Project.
Violin by Benedikt Dietrich from the Noun Project.
(Electric) Piano by b farias from the Noun Project.
Viola by Vasily Gedzun from the Noun Project.
Violin by Francesco Cesqo Stefanini from the Noun Project.
Cello by Valter Bispo from the Noun Project.
(String) bass by Soremba from the Noun Project.
Acoustic guitar by farra nugraha from the Noun Project.
21


Reviews:
[{"review_id": "j52w6BsNETg", "reviewer": "Reviewer_vgm4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The contribution of the paper is mainly written in the \"Summary of the paper\" part, I will write some questions in this section.\n\nI think multi-instrument music transcription task can be regarded as two parts which are transcription (addressing note) and classification (instrument). As the performance of the transcription part can be enhanced apart from the instrument classification part when a large transcription data is used for training, I wonder the performance of a model trained without instrument supervision (without instrument parts in token). This may give some insights about the effect of the multi-instrument token.\n\nSince the main target was on dataset-wise experiment to measure zero-shot performance, the main results are all reported in dataset-wise. However, I wonder the performance difference between instrument-wise split. For example, piano, bass, guitar, ... split (no matter each instrument contains mixed dataset). I think this will give some insights on instrument-level analysis.\n", "summary_of_the_paper": "This paper proposes a multi-task multi-track music transcription framework. Music transcription task has mainly been tackled individually for each instrument type. However, in this work, the authors jointly trained the model using several datasets with different instrument types. As a result, the proposed model provides better transcriptional results compared to models trained on individual data sets. The contribution of the paper follows three parts. 1. multi-task transcription model: This is a good direction of tackling low-resourced transcription task. 2. MIDI-like representation for learning multi-instrument piano rolls: Similar ideas are explored in MIDI-based music generation work recently, and it is good to see this direction is also proposed in multi-instrument transcription task. 3. multi-instrument F1 score metric: Since multi-instrument transcription task is somewhat new, having this kind of metric will be a good for following researches. \n", "summary_of_the_review": "Overall, the paper is well-written, and I vote for accepting the paper. The authors proposed dataset split, model, and some evaluation metric for somewhat new task.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "bH6D9lijBK", "reviewer": "Reviewer_zmLf", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper seems to be a great milestone in the AMT research. It is probably the first unified AMT model that can take music audio with an arbitrary number of instruments. Most of all, leveraging the ability of being trained with all types of AMT datasets, it achieves impressive improvement in the accuracy. \n\n< Good parts >\n- Drum is always a special track in MIDI because the same contains different drum samples over different MIDI notes.  It is great to have drum tokens explicitly added.\n- The introduction of \"end of tie section\" token is a nice idea, considering the nature of music data which requires input audio to be segmented with a fixed length in the practical experiment setting. \n- The evaluation of instrument labeling is conducted as three different granularity of instrument grouping. In particularly, MIDI class, based on the instrument family, make a great sense. \n- The generalization test to out-of-domain data with a series of leave-one-dataset-out experiments is also valuable. It emphasizes the strength of the proposed model and well-aligned with the universality. \n- It is interesting to see the performance of the commercial software (Melodyne, which is known to be based on DSP) in comparison to the previous models. \n\n< Weak parts >\n- The main contribution is the MIDI-like data representation for the output. This representation was \"hand-designed\" rather than learned. I understand that it is natural to design the output form manually and can be seen as a case of knowledge integration to learning models. However, the output representation was simply applied to an existing Transformer model (T5 small) with minor change. In other words, technical novelty is weak. \n- The proposed model was not evaluated with datasets with vocal which is the most essential sound source in popular music. This might be due to the lack of datasets which include vocal tracks in mixtures and their labels. However, the lack of vocal track evaluation limits the capacity of the model.  Since vocal is one of the most expressive instruments, it would be interesting to see how the model works for vocal with background instrumental sounds. \n\n< Questions / Minor parts >\n- The demo examples show that the input audio and output MIDI are temporally aligned well. I wonder if this is because of the absolute positional encoding. What happen if the relative positional encoding?\n- page 7: \"...if it is has the same pitch...\" -->  \"if it has the same pitch\" \n", "summary_of_the_paper": "This paper presents a universal automatic music transcription (AMT) model agnostic to the number of instrument tracks. The model is trained across various AMT datasets, each of which was used as a different AMT task, and less biased to the volume of individual datasets. This is enabled by the novel MIDI-like output data representation designed to include \"instrument\" tokens and also supplementary tokens such as \"tie\" or \"EOS\" tokens. The authors conducted comprehensive experiments with 6 datasets, calculated not only regular accuracy metrics (Frame F1, Onset F1, Onset-Offset F1) but also multi-instrument F1 which is a new metric for the universal AMT task. They significantly improved the accuracy over all datasets and also showed that the model trained with mixed datasets are generally more effective than those trained with a single dataset. ", "summary_of_the_review": "This paper achieved a great milestone in AMT. The demo examples are very impressive and show great potential as a musical tool or a data generator for symbolic music modeling. The main weakness is that the main contribution is designed by hands, in other words, the technical novelty is low. Also, vocal tracks are not included in the experiment. But, this seems to be a matter of dataset availability.\n\n\n\n\n\n\n\n ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "XJU-6Oqusz", "reviewer": "Reviewer_RijS", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- Should we really call it multi-task? I'm not sold that the difference in target instruments is significant enough to call it multi-task.\n- Experiment > Frame F1 - is there any reason for the choice of 62.5 frame per second?\n- I might have missed this from the text - How exactly were the single-instrument metrics computed? Considering every note a single instrument? If it's not specified in the paper, please add it. \n- It's great that even Melodyne was included in the experiment.\n- Re: Table 3 - please add the grouping strategies in the appendix, or at least at the demo page. I expect this work will be impactful in the field, and as a consequence, the grouping strategies would need to be shared somewhere.\n- Appendix A seems to be a bit too simple to me. T1 \"small\" model - does it really define everything about the model? \n", "summary_of_the_paper": "The paper introduces a system called MT3 that performs music transcription with various types of datasets. In the experiment, the system showed state-of-the-art performance in all the relevant metrics and on the selected datasets, which are comprehensive. ", "summary_of_the_review": "Very solid work. A few more details about the experiment and the model would be great. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "M3JyQk386Oo", "reviewer": "Reviewer_4Man", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strong points & contributions:\n\n- Combining AMT datasets to provide a unified training framework\n- Outperforming the relevant state-of-the-art in AMT\n- Widening the scope into multi-instrument transcription\n- Systematic analysis of model training under diverse setups\n- Comparison of numerous architectures against numerous evaluation metrics. \n- A novel, musically relevant evaluation metric taking instruments into consideration\n- Out-of-dataset transcription experiments\n\nBelow I suggest minor improvements and future work:\n\n- An ideal AMT system should be capable of transcribing multiple instruments at once.  \n\n  I think the notion of \"ideal\" is ill-defined. Any music transcription, let it be human or machine annotated, and carried for an engineering or musicological task in mind, should be tailored for the analytic purpose. The purpose does not necessarily encompass robust handling of multiple instruments, time/pitch precision, or else. For example, the most crucial goal may be obtaining a music score like explained in: \n\n  `Carvalho R. G. C., Smaragdis P. (2017). Towards end-to-end polyphonic music transcription: transforming music audio directly to a score. In 2017 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), New Paltz, NY, USA, pp. 151–55.`\n\nand if the method's output notation is not satisfactory for human users, the results will be far from desired.\n\n- The authors clearly demonstrate the case of AMT having relatively low resources. However, they omit to discuss the (even lower resourced) studies applied to music out of 12 tone-equal-temperament (see suggestions below). The authors should include at least the recent work such as:\n\n  `Holzapfel A., Benetos E. (2019). Automatic music transcription and ethnomusicology: a user study. In Proceedings of the 20th International Society for Music Information Retrieval Conference (ISMIR), Delft, Netherlands, pp. 678–84.`\n\n  `V. S. Viraraghavan, A. Pal, H. Murthy, and R. Aravind, \"State-Based Transcription of Components of Carnatic Music,\" ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2020, pp. 811-815, DOI: 10.1109/ICASSP40776.2020.9054435.`\n\n- The authors should contrast their metrics of choice and the novel metric in the paper with the MV2H metric proposed in:\n\n  `McLeod A., Steedman M. (2018). Evaluating automatic polyphonic music transcription. In Proceedings of the 19th International Society for Music Information Retrieval Conference (ISMIR), Paris, France, pp. 42–49.`\n\n- Figure 2: The color selection is not print or (more importantly) color-blind friendly. I would suggest the authors re-render the Figures. See: https://www.nature.com/articles/nmeth.1618 for a reference\n\n- Section 3.2: It would have been better to introduce the explanation in the order of the tokens in Figure 2 for readability.\n\n- On/Off ... events are interpreted as note-on or note-off.\n\n  It might be better to explain what on-off means (e.g., a note is played/released) for an audience unfamiliar with MIDI specifications.\n\n- audio is split into smaller, non-overlapping segments ...\n\n  Apart from the \"turn-off\" handled by the \"end of tie token,\" does the model performance degrade around the edges of the segment?\n\n- 4. Experiments, ... labeling issues with certain datasets\n\n  A few other dimensions could be:\n    \n  - different analytic purposes \n  - difference between the granularity of the transcriptions across the datasets\n  - transcriber consistency/reliability\n\n  It could be helpful to exemplify such issues -at least qualitatively- in the appendix.\n\n- Section C, Section D, ... \n\n  I think they should be Appendix C, Appendix D.\n\n- Ethical considerations\n\n  I think the text in this section does not describe ethical considerations but technical constraints.\n  Nit: See http://globalnotation.org.uk/ as another alternate representation for encoding micro/finer-grained music.  \n\nSuggestions for future work:\n=====\n\n- entirely out-of-domain\n\n  A nitpick to rather suggest future work: I'm afraid I have to disagree that leaving each dataset out is an **entirely** out-of-domain setting; the datasets share common instruments, genres, rhythmic structure, temperament, etc. Having said that it would be interesting to observe how MT3 behaves as the test data gradually becomes out-of-domain, e.g., instrumental vs voice, inserting compound rhythms, altering the timbre/instruments, zero-shot transcription on a LakhNES-like dataset or traditional music datasets (Meertens Tune Collection, Greek folk tunes in Benetos & Holzapfel's recent publications, CompMusic corpora ...) in the future.\n\n- As the authors argue, the characteristics and the annotation alignment varies between the datasets. The work may open an exciting path towards studying \"data valuation\" for automatic music transcription, e.g., which parts of the data are more informative, inconsistent, or erroneous. ds3labs's (https://ds3lab.inf.ethz.ch/easeml.html) work may be of inspiration.", "summary_of_the_paper": "The authors combine numerous automatic music transcription datasets to devise a training framework. They train an off-the-shelf T5 architecture on the combined data and outperform the current state-of-the-art reported on individual datasets. They report numerous experiments to demonstrate the robustness of the model on low resource datasets & out-of-dataset transcription against task-informed variations of F1-scores, including a novel multi-instrument-based F1-score introduced in the paper.\n\n", "summary_of_the_review": "The paper is well written. The literature review is extensive - even though there can be several additions (see the specific comments in the main review). The work extends the scope of existing work in AMT significantly. The experiments are thorough and the authors declare that they will present the necessary code, experimental setup, and results in the camera-ready version so that the work may be reproducible.\n\nGiven the work's strong points as described above, I would like to recommend the paper for publication in ICLR 2022.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work concerns Automatic Music Transcription (AMT) -- transcribing notes given the audio of the music. The paper demonstrates that a single general-purpose transformer model can perform AMT for many instruments across several different transcription datasets. The method represents the first unified AMT model that can transcribe music audio with an arbitrary number of instruments.

All reviewers rated this paper highly and are excited about seeing it at the conference. One reviewer noted that "This paper seems to be a great milestone in the AMT research. It is probably the first unified AMT model that can take music audio with an arbitrary number of instruments."

The reviewers had some suggestions and comments, which appear to be addressed by the authors.
Decision: Accept (Spotlight)