Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a joint training framework of three components - instrument recognition, transcription and source separation. The instrument recognition module is optional and can be replaced by human inputs. The joint training of the transcription and source separation module was shown to be beneficial. The author tried to work on very challenging data with many instruments playing at the same time such as pop music. Various experiments show that the proposed method is promising for several downstream tasks.
strength_and_weaknesses: Strengths:
* The paper is generally well written and easy to follow
* Although many details are omitted in the main text, significant number of details such as model setup are given in the appendix
* The background and related work are well covered
* Although the idea of joint training of transcription & separation is not new for audio processing, authors made great attempt for very challenging multi-instrument music task

Weakness
* Authors could have referred to some recent work about joint speech recognition & speech separation for multi-speaker overlapped speech
* Although authors claim the proposed model reaches the state-of-the-art, the quality of separation is still poor in many cases (in the provided samples). I suspect further improvement can be obtained if authors could optimize the model architecture / training more carefully. For example, Table 1 shows deeper Transformer has the potential to perform better. There are many convolution / self-attention architectures, especially in computer vision and speech area where the authors could try to adopt
* The comparison of the proposed model and MT3 may not be strictly fair as the model size is significantly larger, the training data is different, etc.
* The quality of some plots in the appendix can be improved
clarity,_quality,_novelty_and_reproducibility: * The paper is clearly written and well presented
* The technical novelty is limited as each individual component exists in prior work
* The author will open source the code if published
summary_of_the_review: Overall, the paper explored the interesting path of joint training of music transcription and source separation. Some experiments are informative. However, the technical novelty less significant compared to the empirical novelty. More in-depth analysis of the model behavior or optimization of model architecture that may be related to the task or the joint training technique would be more interesting to the community.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper presents a novel approach to MIR tasks which uses a modular setup combining instrument recognition, transcription, and source separation models into a joint pipeline. The authors perform experiments to evaluate the proposed approach, but the results are mixed (with the only improvement being in a listening study, and quantitative metrics otherwise lower than baselines when they are compared to). Joint training of the modules hurts (rather than helps) performance. Evaluations are focused on what the authors call "real-world popular music" and are conducted on the Cerberus dataset and a set of songs extracted from other pop databases (the model only shows improvements on the latter).
strength_and_weaknesses: 
# Major comments

* The motivation for the current work in the introduction does not, as I understand it, actually motivate the current work. The authors say that "the number of instruments in a pop song may vary....it is limiting to have a model that transcribed a pre-defined fixed number of musical instruments". However, MT3 also does not transcribe a fixed number of instruments, and transcribes the same set of outputs as the current model. I think a better motivation for this work is (1) the general trend of AI/ML models toward "foundation" models trained simultaneously to perform multiple tasks with shared parameters, and (2) a focus on pop music. The latter, in particular, seems to be a motivating factor for the current work, since the evaluation *only* focuses on such music, while the baselines (Omnizart, MT3) can transcribe other common domains as well (e.g. classical) which are absent from the evaluations.

* In several places, the characterization of Jointist vs. MT3, the current state of the art in transcription, seems unfair. For example, the abstract claims that Jointist "achieves state-of-the-art performance...outperforming ...MT3". However, Table 2 shows a clear advantage of MT3; the only improvement over MT3 is in the subjective study (see comments below). The authors also claim that "jointist is more flexible than MT3 in which human users can pick the target instruments that they are interested in"; however, one could also do this with MT3 outputs by simply retaining only the desired instruments. The latter point, in particular, is important: the ground-truth (or "human-provided)") labels are used for Jointist in many experiments (i.e. many of the entries in Table 2) and lead to the best performance; in contrast, such information could be, but is not, applied to the outputs of MT3 -- which would be a more fair comparison. This form of conditioning (essentially filtering of MT3 outputs for known instruments) would also remove the "NA" values for MT3 in Table 2, which the authors state are due to extraneous predicted instruments in the output (something avoided by their models by design in the cases where ground-truth instrument labels are provided).

* Too much material is relegated to the supplement that belongs in the main text. For example, many important architectural details and experimental results (Figs 2-4, the "new instrument-wise metric" in 4.3.1, experimental descriptions for section 6.1).

* The main improvement of the model seems to be in the human listening study. This is concerning for a few reasons. (1) The listening study results are at odds with the quantitative results in Table 2, where it seems MT3 outperforms all versions of Jointist. If the claim is that MT3 does not generalize well to new datasets, this could be verified with a similar listening study on the Slakh test set. (2) The study itself is not well-described: What are the mean and SD of the subjective scores, and how did these vary by piece and dataset? Who are the participants in the human evaluation (musical experts, non-experts)? What was the motivation for providing only MIDI files and allowing participants to render it themselves, instead of providing a pre-rendered version to control for their DAW, etc.

* The non-transcription experiments (source separation, beat tracking, downbeat/chord/key) do not seem to compare to any baselines (at least not in main text), even though baselines exist for all of these tasks (e.g. the authors cite several for source separation; for beat tracking SOTA might be the "madmom" model).

* Why is only note-offset, not note-onset-offset, F1 used in the transcription experiments?

# Minor comments

* The paper refers to such categories as "real-world popular music", but the Slakh dataset is probably not a globally-representative sample of popular music. It would be better to refer to "Western popular music" or similar (as is evidenced by a single category of "ethnic" in Table 11, which encompasses several globally-popular and distinct instruments). 

* Is there a reason the arrow from f_T -> f_MSS goes into the "side" of f_MSS in Figure 1 (not where the rest of the inputs are)?

* The paper should also cite, and possibly compare to, the preceding work "Unaligned Supervision For Automatic Music Transcription in The Wild" (2022) and "A Lightweight Instrument-Agnostic Model for Polyphonic Note Transcription and Multipitch Estimation" (2022).

* P3: Please describe the denoising procedure adapted from Kong et al.

* The description of MIDI mappings in 4.1 is not clear. (1) Are all MIDI numbers 0-3 mapped to a single MIDI number (i.e. electric piano, etc. mapped to piano)? If so, how is this accounted for in the baselines? (2) How are different drum hits mapped to MIDI pitches in the model?

* Do you plan to open-source trained models and code associated with this work?

* Please describe the weighted F1 metric; I could not find a description of how it was weighted in the paper.

* "the language model in MT3 dominates the acoustic model". I believe there is only one model in MT3 (an encoder-decoder). Please clarify.

* "false positive piano rolls and false negbative piano rolls have undefined F1 scores" - please clarify. How is an entire piano roll FP or FN?

* What does the location of I_cond^(i) in Figure 4 signify? That it is input to both the encoder and decoder of U-net? Please clarify.

* Why not compare to a version that uses a frozen MT3 as the transcription model f_T, since the models are not trained together anyway?

* Table 1 suggests continued and monotonic improvements with depth; why not increase the number of layers beyond 4?

# Typos etc.

There are many typos in the paper; I list a few here but would suggest a thorough revision for clarity.

P3: "onsets-and-frames" -> onsets and frames

P3: "teacher-forced training is used during training"

P4: "differnt"

P4: "tracks per piece" -> instruments per piece (?)

P9 "in par with" -> on par with

P15 "instruments conditions" -> instrument conditions

P15 "full-fledge"
clarity,_quality,_novelty_and_reproducibility: See above. The clarity could be improved considerably by revising the writing, adding relevant details to the main text. Reproducibility is unclear, as the authors did not state whether they plan to release open-source code to accompany the work.
summary_of_the_review: Overall, the paper presents a novel take on an existing idea (training joint transcription + source separation ideas). The Jointist approach is more flexible than previously-proposed models for this task, as it can transcribe arbitrary sets of 128 MIDI instruments plus drums. However, the results seem mixed (the only clear transcription improvement is a small gain on a subjective listening study) some of the choices do not seem particularly motivated, and the characterization and use of existing models (such as MT3) seems unfair (see below) with other experiments (source separation, beat tracking, etc.) apparently missing baselines entirely. The paper also needs considerable revision; in particular, some important information is in the supplement that should be in the main text.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes an end-to-end method so-called ``Jointist'' to simultaneously address 3 audio tasks consisting respectively
in separating the source signals, transcribing the music and recognizing the instruments.

The proposed method is a deep neural network which combines together 3 existing neural network architectures which are
respectively [Won et al. 2021] for instrument recognition, [Hawthorne et al. 2017] for the transcription and
[Jansson et al. 2017] for the source separation.
strength_and_weaknesses: Strength:
-Reproducible results with public code and audio samples

Weaknesses:

-Lack of novelty (reuse 3 existing systems trained separately on the 3 addressed tasks)
-Details of the incremental contributions and methods are not sufficient
-insufficient validation (based on only a unique synthesized dataset Slakh2100)
-The interest of combining simultaneously 3 systems appears to be limited according to results
clarity,_quality,_novelty_and_reproducibility: The paper is hard to read with a lot (from my opinion too many) bibliographical references which are not always adequate.
The original contributions except the combination of 3 existing deep neural network is not highlighted.
Results are limited but all are reproducible. 
summary_of_the_review: The main contribution of the paper is the combination of 3 existing deep learning-based methods into a unique end-to-end system for transcription / instrument recognition and separation.

Despite it could be interesting from a practical point of view to investigate how instrument recognition can help to improve transcription  and separation, the authors failed to convince a reader of the interest of their work for the reasons listed below:

1)from a practical point of view, the experiments are insufficient and don't always compare the proposed framework with the relevant state of the art method for each distinct task (eg. source separation results only present the method of the author(s)). Moreover, there exist several other research datasets for music transcription, instrument recognition and source separation which should also be investigated (at least one other reference dataset for each addressed task with the corresponding state-of-the-art method instead of using the same slakh2100 dataset). 

2) Numerical results are limited and not sufficiently detailed. For example, Recall and Precision and confusion matrices are common metrics that could be used for the Instrument Recognition. There also exist other objective metrics for source separation such as SIR and SAR (cf. BssEval) but also perceptual-relevant metrics such as PEASS which could be used to present the separation results.

3)Important details about the proposed method are missing. It is not clear how the 3 modules can be simultaneously trained. The fig. 1 is insufficient to describe the overall method and don't explain how the 3 networks  are connected together.

To conclude, I think that this paper needs an overall reorganization to be consistent and self-content while highlighting the original contribution from the author(s). 
Each experiment should be conduced more rigorously and should provides arguments about the relevance of combining the 3 systems in a unique end-to-end framework with a fair comparison with the state of the art of each distinct task.
The ideas of combining separation with transcription, and instrument recognition with transcription are not novel. Hence the authors 
should further explain in what they believe that this work is original and preferable than using 3 distinct dedicated methods.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper describes a jointly-trained triplet of models for musical instrument recognition, music transcription, and music source separation, able to handle a large number of instruments.  Experiments demonstrate that training these models jointly results in better performance for each task than training each model in isolation, and the models (the transcription model specifically) is useful for several downstream tasks.  I am not as familiar with the source separation literature, but I believe another contribution of the paper is a source separation model that can work on a much wider variety of instruments.
strength_and_weaknesses: Strengths
-----------
There is one main strength which is that the overarching goal of the paper is solid: solve transcription and source separation jointly for music with an arbitrary number of instruments.  And the system proposed in the paper does appear to be a step forward in solving both problems, as demonstrated by the many experiments in the paper.
 
Weaknesses
--------------
1) It's not especially surprising that combining transcription and source separation would lead to improved performance on both tasks.  However, there are multiple ways one could imagine combining these tasks.  In this paper, transcription seems to be treated as "primary", as the output of the transcription module is fed into the source separation module.  One could just as well imagine the converse, where sources are separated prior to transcription.  Would that setup work better?  Or could there be a third setup that learns a representation useful for both tasks?  These questions are neither asked nor answered by the paper.

2) In terms of the objective metrics, the transcription model in this paper performs *considerably* worse than the MT3 model of Gardner et al.  Since the purpose of music transcription is usually not direct listening, the fact that listeners prefer the Jointist transcriptions is interesting but not especially compelling.  Since you are already performing some experiments with the transcriptions on downstream tasks, it would be helpful to demonstrate that the transcriptions from Jointist also outperform MT3's transcriptions on these tasks, if that is in fact the case.

3) Related to the above, I would be interested in reading an entire paper comparing downbeat/chord/key estimation from audio vs. pianoroll.  But the fact that giving a model access to both pianoroll and audio outperforms just audio is not surprising, nor is the performance improvement large enough to warrant the inclusion of the experiment in *this* paper.  Downbeat/chord/key estimation is an application of *any* transcription model!

4) The introduction suggests a lack of symbolic data for training generative pop music models as a motivation for music transcription.  However, the very dataset used in this paper (Slakh) is derived from a much larger dataset of symbolic music (Lakh MIDI Dataset), much of which is pop.  And there are existing generative models (e.g. MuseNet) trained on such datasets.  I would argue that what's lacking is symbolic data *aligned to real recordings*.

5) Minor issues:
    * Throughout the paper the term "ablation" is used to refer to experiments with multiple conditions where none is obviously an ablation; "ablation" refers to *removal* of components.
    * Some typos: "differnt" in Section 3.3, "black" instead of "block" in Appendix A, "marco" instead of "macro" in Appendix B.1.
    * Figures 6 and 7 would be much more readable if axis labels used linear label counts even if the dimension is log scale.
    * More of a question: in Appendix E, why do some pairs of instruments that seem similar (e.g. recorder and flute) have vastly different transcription performance?  It might be helpful to show something like a confusion matrix here (even though the values aren't exactly classification counts), because I suspect there are pairs of instruments that are frequently mistaken for one another.
clarity,_quality,_novelty_and_reproducibility: The paper is written clearly, is original albeit not revolutionary, and is most likely reproducible modulo the standard fiddling with hyperparameters and whatnot.
summary_of_the_review: The paper doesn't have any especially surprising findings and could explore its main idea more thoroughly (and remove some non-essential sections), but I recommend acceptance nonetheless as it appears methodologically sound and is a valid research contribution to music transcription and source separation.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary: The paper presents a joint training framework for instrument recognition, transcription and source separation. The problem is challenging and the experiments have shown promising results.

Strengths: The work present a promising end-to-end solution to a challenging task.

Weaknesses: As agreed from the reviewers, the novelty of the paper is limited. There have been many existing work on joint training in other/similar fields though not exactly the same setup. 
justification_for_why_not_higher_score: Limited novelty.
Decision: Reject