Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors combine numerous automatic music transcription datasets to devise a training framework. They train an off-the-shelf T5 architecture on the combined data and outperform the current state-of-the-art reported on individual datasets. They report numerous experiments to demonstrate the robustness of the model on low resource datasets & out-of-dataset transcription against task-informed variations of F1-scores, including a novel multi-instrument-based F1-score introduced in the paper.


main_review: Strong points & contributions:

- Combining AMT datasets to provide a unified training framework
- Outperforming the relevant state-of-the-art in AMT
- Widening the scope into multi-instrument transcription
- Systematic analysis of model training under diverse setups
- Comparison of numerous architectures against numerous evaluation metrics. 
- A novel, musically relevant evaluation metric taking instruments into consideration
- Out-of-dataset transcription experiments

Below I suggest minor improvements and future work:

- An ideal AMT system should be capable of transcribing multiple instruments at once.  

  I think the notion of "ideal" is ill-defined. Any music transcription, let it be human or machine annotated, and carried for an engineering or musicological task in mind, should be tailored for the analytic purpose. The purpose does not necessarily encompass robust handling of multiple instruments, time/pitch precision, or else. For example, the most crucial goal may be obtaining a music score like explained in: 

  `Carvalho R. G. C., Smaragdis P. (2017). Towards end-to-end polyphonic music transcription: transforming music audio directly to a score. In 2017 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), New Paltz, NY, USA, pp. 151–55.`

and if the method's output notation is not satisfactory for human users, the results will be far from desired.

- The authors clearly demonstrate the case of AMT having relatively low resources. However, they omit to discuss the (even lower resourced) studies applied to music out of 12 tone-equal-temperament (see suggestions below). The authors should include at least the recent work such as:

  `Holzapfel A., Benetos E. (2019). Automatic music transcription and ethnomusicology: a user study. In Proceedings of the 20th International Society for Music Information Retrieval Conference (ISMIR), Delft, Netherlands, pp. 678–84.`

  `V. S. Viraraghavan, A. Pal, H. Murthy, and R. Aravind, "State-Based Transcription of Components of Carnatic Music," ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2020, pp. 811-815, DOI: 10.1109/ICASSP40776.2020.9054435.`

- The authors should contrast their metrics of choice and the novel metric in the paper with the MV2H metric proposed in:

  `McLeod A., Steedman M. (2018). Evaluating automatic polyphonic music transcription. In Proceedings of the 19th International Society for Music Information Retrieval Conference (ISMIR), Paris, France, pp. 42–49.`

- Figure 2: The color selection is not print or (more importantly) color-blind friendly. I would suggest the authors re-render the Figures. See: https://www.nature.com/articles/nmeth.1618 for a reference

- Section 3.2: It would have been better to introduce the explanation in the order of the tokens in Figure 2 for readability.

- On/Off ... events are interpreted as note-on or note-off.

  It might be better to explain what on-off means (e.g., a note is played/released) for an audience unfamiliar with MIDI specifications.

- audio is split into smaller, non-overlapping segments ...

  Apart from the "turn-off" handled by the "end of tie token," does the model performance degrade around the edges of the segment?

- 4. Experiments, ... labeling issues with certain datasets

  A few other dimensions could be:
    
  - different analytic purposes 
  - difference between the granularity of the transcriptions across the datasets
  - transcriber consistency/reliability

  It could be helpful to exemplify such issues -at least qualitatively- in the appendix.

- Section C, Section D, ... 

  I think they should be Appendix C, Appendix D.

- Ethical considerations

  I think the text in this section does not describe ethical considerations but technical constraints.
  Nit: See http://globalnotation.org.uk/ as another alternate representation for encoding micro/finer-grained music.  

Suggestions for future work:
=====

- entirely out-of-domain

  A nitpick to rather suggest future work: I'm afraid I have to disagree that leaving each dataset out is an **entirely** out-of-domain setting; the datasets share common instruments, genres, rhythmic structure, temperament, etc. Having said that it would be interesting to observe how MT3 behaves as the test data gradually becomes out-of-domain, e.g., instrumental vs voice, inserting compound rhythms, altering the timbre/instruments, zero-shot transcription on a LakhNES-like dataset or traditional music datasets (Meertens Tune Collection, Greek folk tunes in Benetos & Holzapfel's recent publications, CompMusic corpora ...) in the future.

- As the authors argue, the characteristics and the annotation alignment varies between the datasets. The work may open an exciting path towards studying "data valuation" for automatic music transcription, e.g., which parts of the data are more informative, inconsistent, or erroneous. ds3labs's (https://ds3lab.inf.ethz.ch/easeml.html) work may be of inspiration.
summary_of_the_review: The paper is well written. The literature review is extensive - even though there can be several additions (see the specific comments in the main review). The work extends the scope of existing work in AMT significantly. The experiments are thorough and the authors declare that they will present the necessary code, experimental setup, and results in the camera-ready version so that the work may be reproducible.

Given the work's strong points as described above, I would like to recommend the paper for publication in ICLR 2022.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a multi-task multi-track music transcription framework. Music transcription task has mainly been tackled individually for each instrument type. However, in this work, the authors jointly trained the model using several datasets with different instrument types. As a result, the proposed model provides better transcriptional results compared to models trained on individual data sets. The contribution of the paper follows three parts. 1. multi-task transcription model: This is a good direction of tackling low-resourced transcription task. 2. MIDI-like representation for learning multi-instrument piano rolls: Similar ideas are explored in MIDI-based music generation work recently, and it is good to see this direction is also proposed in multi-instrument transcription task. 3. multi-instrument F1 score metric: Since multi-instrument transcription task is somewhat new, having this kind of metric will be a good for following researches. 

main_review: The contribution of the paper is mainly written in the "Summary of the paper" part, I will write some questions in this section.

I think multi-instrument music transcription task can be regarded as two parts which are transcription (addressing note) and classification (instrument). As the performance of the transcription part can be enhanced apart from the instrument classification part when a large transcription data is used for training, I wonder the performance of a model trained without instrument supervision (without instrument parts in token). This may give some insights about the effect of the multi-instrument token.

Since the main target was on dataset-wise experiment to measure zero-shot performance, the main results are all reported in dataset-wise. However, I wonder the performance difference between instrument-wise split. For example, piano, bass, guitar, ... split (no matter each instrument contains mixed dataset). I think this will give some insights on instrument-level analysis.

summary_of_the_review: Overall, the paper is well-written, and I vote for accepting the paper. The authors proposed dataset split, model, and some evaluation metric for somewhat new task.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a universal automatic music transcription (AMT) model agnostic to the number of instrument tracks. The model is trained across various AMT datasets, each of which was used as a different AMT task, and less biased to the volume of individual datasets. This is enabled by the novel MIDI-like output data representation designed to include "instrument" tokens and also supplementary tokens such as "tie" or "EOS" tokens. The authors conducted comprehensive experiments with 6 datasets, calculated not only regular accuracy metrics (Frame F1, Onset F1, Onset-Offset F1) but also multi-instrument F1 which is a new metric for the universal AMT task. They significantly improved the accuracy over all datasets and also showed that the model trained with mixed datasets are generally more effective than those trained with a single dataset. 
main_review: This paper seems to be a great milestone in the AMT research. It is probably the first unified AMT model that can take music audio with an arbitrary number of instruments. Most of all, leveraging the ability of being trained with all types of AMT datasets, it achieves impressive improvement in the accuracy. 

< Good parts >
- Drum is always a special track in MIDI because the same contains different drum samples over different MIDI notes.  It is great to have drum tokens explicitly added.
- The introduction of "end of tie section" token is a nice idea, considering the nature of music data which requires input audio to be segmented with a fixed length in the practical experiment setting. 
- The evaluation of instrument labeling is conducted as three different granularity of instrument grouping. In particularly, MIDI class, based on the instrument family, make a great sense. 
- The generalization test to out-of-domain data with a series of leave-one-dataset-out experiments is also valuable. It emphasizes the strength of the proposed model and well-aligned with the universality. 
- It is interesting to see the performance of the commercial software (Melodyne, which is known to be based on DSP) in comparison to the previous models. 

< Weak parts >
- The main contribution is the MIDI-like data representation for the output. This representation was "hand-designed" rather than learned. I understand that it is natural to design the output form manually and can be seen as a case of knowledge integration to learning models. However, the output representation was simply applied to an existing Transformer model (T5 small) with minor change. In other words, technical novelty is weak. 
- The proposed model was not evaluated with datasets with vocal which is the most essential sound source in popular music. This might be due to the lack of datasets which include vocal tracks in mixtures and their labels. However, the lack of vocal track evaluation limits the capacity of the model.  Since vocal is one of the most expressive instruments, it would be interesting to see how the model works for vocal with background instrumental sounds. 

< Questions / Minor parts >
- The demo examples show that the input audio and output MIDI are temporally aligned well. I wonder if this is because of the absolute positional encoding. What happen if the relative positional encoding?
- page 7: "...if it is has the same pitch..." -->  "if it has the same pitch" 

summary_of_the_review: This paper achieved a great milestone in AMT. The demo examples are very impressive and show great potential as a musical tool or a data generator for symbolic music modeling. The main weakness is that the main contribution is designed by hands, in other words, the technical novelty is low. Also, vocal tracks are not included in the experiment. But, this seems to be a matter of dataset availability.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper introduces a system called MT3 that performs music transcription with various types of datasets. In the experiment, the system showed state-of-the-art performance in all the relevant metrics and on the selected datasets, which are comprehensive. 
main_review: - Should we really call it multi-task? I'm not sold that the difference in target instruments is significant enough to call it multi-task.
- Experiment > Frame F1 - is there any reason for the choice of 62.5 frame per second?
- I might have missed this from the text - How exactly were the single-instrument metrics computed? Considering every note a single instrument? If it's not specified in the paper, please add it. 
- It's great that even Melodyne was included in the experiment.
- Re: Table 3 - please add the grouping strategies in the appendix, or at least at the demo page. I expect this work will be impactful in the field, and as a consequence, the grouping strategies would need to be shared somewhere.
- Appendix A seems to be a bit too simple to me. T1 "small" model - does it really define everything about the model? 

summary_of_the_review: Very solid work. A few more details about the experiment and the model would be great.
review_title: null

Meta-review: This work concerns Automatic Music Transcription (AMT) -- transcribing notes given the audio of the music. The paper demonstrates that a single general-purpose transformer model can perform AMT for many instruments across several different transcription datasets. The method represents the first unified AMT model that can transcribe music audio with an arbitrary number of instruments.

All reviewers rated this paper highly and are excited about seeing it at the conference. One reviewer noted that "This paper seems to be a great milestone in the AMT research. It is probably the first unified AMT model that can take music audio with an arbitrary number of instruments."

The reviewers had some suggestions and comments, which appear to be addressed by the authors.
Decision: Accept (Spotlight)