Jointist: Simultaneous Improvement of Multi-instrument Transcription and Music Source Separation via Joint Training
Authors: Kin Wai Cheuk
Keunwoo Choi
Qiuqiang Kong
Bochen Li
Minz Won
Ju-Chiang Wang
Yun-Ning Hung
Dorien Herremans
Venue: Submitted to ICLR 2023 (2023)
Keywords: multi-task learning
automatic music transcription
music source separation
instrument recognition

Abstract: In this paper, we introduce Jointist, an instrument-aware multi-instrument framework that is capable of transcribing, recognizing, and separating multiple musical instruments from an audio clip.
Jointist consists of an instrument recognition module that conditions the other two modules: a transcription module that outputs instrument-specific piano rolls, and a source separation module that utilizes instrument information and transcription results. The joint training of the transcription and source separation modules serves to improve the performance of both tasks. The instrument module is optional and can be directly controlled by human users. This makes Jointist a flexible user-controllable framework.

Our challenging problem formulation makes the model highly useful in the real world given that modern popular music typically consists of multiple instruments. Its novelty, however, necessitates a new perspective on how to evaluate such a model. In our experiments, we assess the proposed model from various aspects, providing a new evaluation perspective for multi-instrument transcription. Subjective listening test shows that Jointist achieves state-of-the-art performance on popular music, outperforming existing multi-instrument transcription models such as MT3. %We also argue that transcription models can be used as a preprocessing module for other music analysis tasks. We conducted experiments on several downstream tasks, and found that the proposed method improved transcription by more than 1 percentage points (ppt.); source separation by 5 SDR, downbeat detection by 1.8 ppt., chord recognition by 1.4 ppt., and key estimation by 1.4 ppt., when utilizing transcription results obtained from Jointist.  

Under review as a conference paper at ICLR 2023
JOINTIST: SIMULTANEOUS IMPROVEMENT OF MULTI- INSTRUMENT TRANSCRIPTION AND MUSIC SOURCE SEPARATION VIA JOINT TRAINING
Anonymous authors Paper under double-blind review
ABSTRACT
In this paper, we introduce Jointist, an instrument-aware multi-instrument frame- work that is capable of transcribing, recognizing, and separating multiple musical instruments from an audio clip. Jointist consists of an instrument recognition module that conditions the other two modules: a transcription module that outputs instrument-speccin piano rolls, and source separation module that utilizes instru- ment information and transcription results. The joint training of the transcription and source separation modules serves improve the performance of both tasks. The instrument module is optional and can be directly controlled by humaa users. This makes Jointist flexible uscr-controllable framework.
Our challenging problem formulation makes the model highly useful in the real world given that modern popular music typically consists of multiple instruments. Its novelty, however, necessitates a new perspective on how to evaluate such a model. In our experiments, we assess the proposed model from various aspects, providing new evaluation perspective for multi-instrument transcription. Our subjective listening study shows that Jointist achieves state-of -ote-----thhrrtt performance on popular music, outperforming existing multi-instrumert transcription models such as MT3. We conducted experiments on several downstream tasks and found that the proposed method improved transcription by more than percentage points (ppt.), source separation by 5 SDR, downbeat detection by 1.8 ppt., chord recogni- tion by 1.4 ppt., and key estimation by 1.4 ppt., when utilizing transcription results obtained from Jointist.
INTRODUCTION
Transcription, automatic music transcription (AMT), is music analysis task that aims to repre- sent audio recordings using symbolic notations such as scores or MIDI (Musical Instrument Digital Interface) files (Benetos et al. 2013 2018; Piszczalski & Galler 1977). AMT can play important role in music information retrieval (MIR) systems since symbolic information - c.g., pitch, duration, and velocity of notes determines large part of our musical perception. A successful AMT should symbolic format, which could ease the difficulty of many MIR tasks such melody extraction (Ozcan et al.) (2005), chord recogni- tion (Wu & Li 2018)) beat tracking (Vogi et al. (0177), composer classification (Kong et al. 2020a)) (Kim et al. 2020p, and emotion classification (Chou et al. [021). Finally, high-quality AMT systems can be used to build large-scale datasets as done by Kong et al. (2020b). This can, in turn, accelerate the development of neural network- based MIR systems as these are often trained using otherwise scarcely available audio aligned symbolic data (Brunner et al. 2018 Wu et al. 2020a Hawthorne al. 2018). Currently, the only available pop music dataset the Slakh2100 dataset by Manilow al. (2019). The lack of large-scale audio aligned symbolic dataset for pop music impedes the development of other MIR systems that are trained using symbolic music representations.
provide a denoised version of music in a musically-meaninggfff
In the early research on AMTs, the problem is often defined narrowly as transcription of a single target instrument, typically piano (Klapuri & Eronen 1998) or drums (Paulus & Klapuri 2003), and whereby the input audio only includes that instrument. The limitation of this strong and then- unavoidable assumption is clear: the model would not work for modern pop music, which occupies majority of the music that people listen to. In other words, handle realistic use-cases of AMT, it

---

Under review as a conference paper at ICLR 2023
is necessary to develop multi inntiiiinstumment transcription system. Recent examples are Omnizart (Wu et al. 2021 2020b and MT3 (Gardner et al. 2021) which we will discuss in Section 2
The progress towards multi-instrumen transcription, however, has just begun. There are still several challenges related to the development and evaluation of such systems. In particular, the number of instruments in multiple-instrument audio recordings not fixed. The number of instruments in a pop song may vary from a few to over ten. Therefore, it is limiting to have a model that transcribes a pre-defined fixed number of musical instruments in every music piece. Rather, a model that can adapt to a varying number of target instrument(s) would be more robust and useful. This indicates that we may need consider instrument recognition and instrument--pecific behavior during development as well as evaluation.
Motivated by the aforementioned recent trend and the existing issues, we propose Jointist - frame- work that includes instrument recognition, source separation, as well as transcription. We adopt a joint training scheme to maximize the performance of both the transcription and source separation module. Our experiment results demonstrate the utility of transcription models as a pre-processing module MIR systems. The result strengthens perspective of transcription result as a symbolic representation, something distinguished from typical i) audio-based representation (spectrograms) or ii) high-level features (Choi et al. 2017 Castellon et al. 202I
This paper is organized as follows. We first provide brief overview of the background of automatic music transcription in Section 2 Then we introduce our framework, Jointist, in Section 3 We
deseribe the experimental details in Section f and discuss the experimental results in Section [] We
also explore the potential applications of the piano rolls generated by Jointist for other MIR tasks in Section 6 Finally, we conclude the paper in Section 7
2 BACKGROUND
While automatic music transcription (AMT) models for piano music are well developed and are able to achieve high accuracy (Benetos et al. 2018; Sigtia et al. 2015) Kim & Bello 2019; Kelz et al. 2019; Hawthorne et al. 2017]20111 Kong al. 2021). insti-instrument automatic music transcription (MIAMT) is relatively unexplored. MusicNet (Thickstun et al.] 2016: 2017) and Re- conVAT (Cheuk al., 2021a) piano, but their output is a flat piano roll that includes notes from all the instruments in a sin- gle channel. In other words, they are not instrument-awan
MIAMT systems that transeribe musical instruments other than
Omnizart (Wu et al. 2019 2021) is
instrument-awann but it does not scale well when the number of musical instruments increases as discussed in Section 5..3 MT3 Gardner et al. 2021 is currently the state-of-the-ee MIAMT model. It formulates AMT as sequence prediction task where the sequence consists of tokens represent- ing musical notes. By adopting the structure of a natural language processing (NLP) model called sequence-to-seqeeeee architecture can
T5 (Raffel et al.) 2020). MT3 shows that transformer-base perform successful transcription by learning from multiple datasets for various instruments.
Although there have been attempts to jointly train speech separation and recognition model (Shi jct al. 2022). the joint training of transcription model together with a source separation model is still very limited in the MIR domain. For example, Jansson et al. (2019) use the FO estimation to guide the singing voice separation. However, they only demonstrated their method with a monophonic singing track. In this paper, the Jointist framework extends this idea into polyphonic music. While Manilow et al. (2020) extends the joint transcription and source separation training into polyphonic music, their model is limited to up to five sources (Piano, Guitar, Bass, Drums, and Strings) which do not cover the diversity of real-world popular music. Our proposed method is trained and evaluated on 39 instruments, which is enough to handle real-world popular music. [Hung al. (202I) and Chen et al. (2021a) also use joint transcription and source separation training for a small number of instruments. However, they only use transcription as an auxiliary task during training while no transcription is performed during the inference phases. The model in Tanaka et al.] (2020), on the other hand, is only capable of doing transcription. Their model applies a joint spectrogram and pitchgram clustering method to improve the multi-instrument transcription accuracy. A zero- shot transcription and separation model was proposed in Lin et al. (2021) but was only trained and evaluated on 13 classical instruments. Our proposed Jointist framework alleviates most of the problems mentioned above. Unlike existing models, Jointist is instrument-awar, and transcribes and separates only the instruments present in the input mix audio.
2

---

Under review as a conference paper at ICLR 2023
Predicted
STFT
fMSS
Xsvrr (8,7,512)
(B,L)xX
Mel
Xway (B,L)
Xiee (B,T, 229)
fIR
Conditions Lond C)X
fT
Posterlorgrams T,)xXX
Figure 1: The proposed Jointist framework. This framework can transcribe/separrate up to 39 dif- ferent instruments as defined in Table [1] the Appendix. B: batch size, T.: audio length, C: instrument classes, T:: number of time steps, K: number of predicted instruments. The dotted lines represent iterative operations that are repeated K times. Best viewed in color.
JOINTIST
We designed the proposed framework, Jointist, to jointly train music transcription and source sep- aration modules to improve the performance for both tasks. By incorporating an instrument recognition module, our framework instrument-aware and compatible with up to 39 different in- struments shown in Table of the Appendix.
As illustrated in Figure T Jointist consists of three modules: the instrument recognition module fIR (Figure 2) the transcription module (Figure B), and the music source separation module fuss (Figure The fIR and fr share the same mel spectrogram Xeel as the input, while sMSs uses the STFT spectrogram XSIFT the input.
Jointist is flexible framework that can be trained end-to-end or individually. For the ease of eval- uation, trained fIR individually. Note that our framework could still work without fIR. Since both fr and fass accept one-hot vectors I'ond as the instrument condition, users could easily bypass fIR and manually specify the target instruments to transcribed, making Jointist a human control- lable framework. In Section 5.3) we show that joint training of fT and fMss mutually benefits both modules. In the discussions below, we denote Jointist with fIR module as end-to-end Jointist; without SIR module controllable Jointist.
3.1 INSTRUMENT RECOGNITION MODULE
Figure 2 the Appendix shows the model architecture of the instrument recognition module fIR me which inspired by the music tagging transformer proposed by Won et al. (2021a).
Transformer networks /Vaswani et al. (007)) have been shown to work well for a wide range of MIR
tasks (Chen al. 202Tb: Lu et | Huang al. 2018 Park et al. 2019 Won al.) 2019 Guo et al. 2022 Won et al.) [2221a). We therefore designed our instrument recognition model to consist of a convolutional neural network (CNN) front-end and transformer back-end. To prevent overfit- ting, we perform dropout after each convolutional block with rate of 0.2 (Srivastava et al. 2014). We conduct an ablation study as shown in Table I find the optimal number of transformer layers, which turns out to 4. The instrument recognition loss defined as LıR = BCE (Yundd Yeena) where BCE is the binary cross-entropy, Yoond a vector of the predicted instruments, and Yound are the ground truth labels. The predicted Yoood is then converted into i: to be used by fr and fMss later.
3.2 TRANSCRIPTION MODULE
Figure shows the model architecture of the transcription module, ( which is adopted from the onsets and frames model proposed by /Hawthorne et al. (2017) ). This module is conditioned by the one-hot instrument vectors Fe using FiLM (Perez et al. 2018), as proposed by Meseguer- control which target instrument
Brocal & Peeters (2019) for source separation, which allows to transcribe. To ensure model stability, teacher-forced training is used, i.e. the ground truth ! is used the condition. During inference. combine the onset rolls Yonser and the frame rolls Y Ytrame based the idea proposed by Kong al. (2021) to produce the final transcription Y In this procedure, Yo used to filter out noisy Yirame, resulting in cleaner Y Note that i: could be
3

---

Under review as a conference paper at ICLR 2023
provided by either the fiR module or human users. The transcription loss is defined as L1 Σ) Li, where L; BCE(Y,, Yj) is the binary cross entropy and j E {onset, frame).
3.3 SOURCE SEPARATION MODULE
Figure shows the model architecture of the source separation module, fuss (XSTFT. - V which inspired by Jansson et al. (2017); Meseguer-Brocal & Peeters (019 In addition to Iood (either generated by SIR or provided by human users), our /MSs also uses Y of the music. The source separation loss LMss = L Yÿ) is set as the L2 loss between the predicted source waveform Y3 and the ground truth source waveform Y When combining the transcription output Y Frame = (meme,, and concatenation g ((immm))) @ XSTET- Where g is linear layer that maps the 88 midi pitches Y the same dimensions the XSTFT. An ablation study for the two different modes can be found in Figure in the Appendix.
i (cmma) with XSTFT, we explored two different modes: summation g (Y framm) + XSTFT
EXPERIMENTS
4.1 DATASET
Jointist is trained using the Slakh2100 dataset (Manilow al. 2019). This dataset is synthesized from part of the Lakh dataset (Raffel (016) by rendering MIDI files using a high-quality sample- based synthesizer with a sampling rate of 44.1 kHz. The training, validation, and test splits contain 1500, 225, and 375 pieces respectively. The total duration of the Slakh2100 dataset is 145 hours. The number of tracks per piece in the Slakh2100 dataset ranges from 4 to 48, with a median number of 9, making it suitable for multiple-instrumen AMT. In the dataset, 167 different plugins are used to render the audio recordings.
Each plugin also has a MIDI number assigned to it, hence we can use this information to map 167 different plugins onto 39 different instruments as defined in Table [T in the Appendix. We map map MIDI number 0-3 to
the MIDI numbers according their instrument types. For example, piano, although they indicate different piano types. Finally, the original MIDI numbers only cover 0-127 channels. We add one extra channel (128) represent drums that our model can also transcribe this instrument.
We also conduct subjective evaluation to measure the perceptual quality of the transcription outputs, since it has been shown that higher objective metrics do not necessarily correlate to better perceptual quality (Simonetta al.) 2022. [uo al. 2020). We selected 20 full-tracks of pop songs from public datasets: Isophonics (Mauch et al.) 2009). RWC-POP (Goto et al.) 2002). JayChou29 (Deng & Kwok 2017), and USPOP2002 (Berenzweig et al. 2004), considering the diversity of instrumen- tation and composition complexity. The metadata of the 20 songs and the participants are listed in Table & and Figure Appendix
4.2 TRAINING
When training the modules of the Jointist framework, the audio recordings are first resampled to 16 kHz, which is high enough to capture the fundamental frequencies as well some harmonics of the highest pitch Cg on the piano (4,186 Hz) (Hawthorne et al. 2017 Kong et al. (021). Following some conventions on input features (Hawthorne al., 2017 Kong et al. 2021 Cheuk et al. 2021a), for the instrument recognition and transcription, we use log Mel spectrograms with a window size of 2,048 samples, a hop size of 160 samples, and 229 Mel filter banks. For the source separation, we use STFT with window size of 1,024, and a hop size of 160. This configuration leads to spectrograms with 100 frames per second. Due to memory constraints, we randomly sample a clip of 10 seconds of mixed audio to train our models. The Adam optimizer (Kingma & Ba 2014) with learning rate of 0.001 is used to train both fIR and fr. For fMss. a learning rate of 1 x 10 is chosen after preliminary experiments. All three modules are trained two Tesla V100 32GB GPUs with a batch size of six each. PyTorch and Torchaudio (Paszke et al. 2019 Yang et al. 2022 are used to perform all of the experiments well the audio processing.
4

---

Under review as a conference paper at ICLR 2023
The overall training objective L is sum of the losses of the three modules, i.e., L LIR Lr Lass-
4.3
EVALUATION CONFIGURATION
4.3.1 OBJECTIVE EVALUATION
A detailed explanation of each metric is available in Section B in the Appendix. The mAP score for fiR is calculated using the sigmoid output Yoond in Figure In practice, however, we need to apply threshold to Yoond to obtain the one-hot vector that can be used by fr and fMss as the condition. Therefore, we also report the Fl score with a threshold of 0.5 in Table We chose this threshold value ensure the simplicity of the experiment, even though the threshold can be tuned for each instrument to further optimize the metric (Won et al., 2021b). To understand the model performance under different threshold values, we also report the mean average precision (mAP) in Table [
For fT. we propose a new instrument-ve wise metric (Figure 9 and 10 to better capture the model performance for multi-instrument transcription. Existing literature uses mostly flat metrics piece- wise evaluation (Hawthorne et al. 2017 Gardner et al. [021 Kong et al. 2021; Cheuk et al. 222 la), Although this can provide a general idea of how good the transcription is, it does not show which musical instrument the model is particularly good or bad at. Since frame-wise metrics do not reflect the perceptual transcription accuracy (Cheuk et al. 2021b), we report only the note-wise (N.) and note-wise with offset (N&O) metrics in Table 2 We study the performance of both end-to-end and human controllable Jointist.
For fMss, we also report the instrument- -ise metrics to better understand the model performance for each instrument (Figure 13 and Table
4.3.2 SUBJECTIVE EVALUATION
As mentioned in the Introduction, successful AMT would provide clear and musically-plausible symbolic representation of music. This suggests that synthesized audio from a transcribed MIDI file should resemble the original recording, given that the transcription was accurate. Therefore, participants were asked to use DAW (Digital Audio Workstation), preferably GarageBand on Mac OS, listen the audio synthesized by the default software instruments and review the MIDI note display of each instrument present in the piece.
We focus on the comparison between Jointist and MT3 for the subjective evaluation. To generate the test MIDI files for the 20 selected songs, we used end-to-end Jointist and replicated the MT3 model following (Gardner al. [ We propose four evaluation aspects (detailed explanation available in Appendix B.4): instrument integrity, instrument-wise note continuity, overall note accuracy, and overall listening experience. Participants imported the original audio of piece in the DAW, alone with one of two anonymous MIDI transcriptions (by either Jointist or MT3), so that the signals are synchronized, allowing the participants to examine back and forth between the audio and MIDI. After that, they were asked to rate the two transcriptions according to the aspects mentioned above on a scale from to as explained in Appendix [B.5] Participants were asked to pay attention important instruments such drums, bass, guitar, and piano which plays more important role in building the perceptions of genre, rhythm, harmony, and melody of a song.
RESULTS
5.1 INSTRUMENT RECOGNITION
Table shows the mAP and F1 scores of the instrument recognition module fIR when a different number of transformer encoder layers are used. Both the mAP and F1 scores improve as the number of layers increases. The best mAP and Fl scores are reached when using four transformer encoder layers. Due to the instrument class imbalance in the 11akh2100 dataset, our fIR performance is relatively low for instrument classes with insufficient training samples such as clarinet, violin, or harp as shown in Figure in Appendix E Therefore, the weighted FI/mAP is higher than the
http:///tttcccoooooomommmm//moma transcription with transformers.ipynb
5

---

Under review as a conference paper at ICLR 2023
mAP
F1
#Layers (#Parameters)
Macro
Weighted
Macro
Weighted
(78.0M)
71.8
91.6
61.5
85.4
(78.8M)
72.1
92.1
65.0
86.2
(79.6M)
73.7
92.2
63.7
86.9
(80.3M)
77.4
92.6
70.3
87.6
Table 1: The accuracy of instrument recognition by the number transformer layers.
Flat F1
Piece, F1
Inst. F1
Model
N.
N&O
N
N&O
N
N&O
Wu et al. (019)
26.6
13.4
11.5
6.30
4.30
1.90
Gardner al. (2021
76.0
57.0
N.A.
N.A.
N.A.
N.A.
57.9
25.1
59.7
27.2
48.7
23.2
58.0
25.4
N.A
N.A
N.A
pTS(s)
58.4
26.2
61.2
28,5
50.6
24.7
ipTS(s)
58.4
26.5
N.A
N.A
N.A
N.A
TS(s)
47.9
18.6
45.7
18.4
34.9
15.8
PTS(c)
58.4
26.3
61.3
28.6
50.8
24.8
TS(c)
47.7
18.6
46.0
18.0
35.5
16.2
Table 2: Transcription accuracy for existing state-of-the-art models and our proposed Jointist. "T" and 'S" specifies the trained fi and fMss modules respectively. The prefix 'p-' represents that the transcription module is pretrained, 'i-" represents that fIR is used obtain the instrument conditions, models without a prefix represent full control by human users in which their target instruments are exactly the ground truth instruments. (s) and (c) indicate whether the piano rolls are summed or concatenated, respectively, to the spectrograms.
macro F1/mAP. Nevertheless, we believe that our SIR is good enough to generate reliable instrument conditions for popular instruments such as drums, bass, and piano.
Note that human users can easily bypass this module by providing the one-hot instrument conditions directly to Jointist. Even if human users do not want to take over fIR. Jointist could still work by activating all 39 instrument conditions. Therefore, fIR an optional module of the framework and hence keep its evaluation minimal.
5.2 TRANSCRIPTION
Table 2] shows the note-wise (N.) and note-wise with offset (N&O) transcription F1 scores for dif- ferent models. In addition to training only the fr (T), we also explore the possibility of training both fi and SMss jointly (TS) and study effect the transcription accuracy.
When evaluating the transcription module fr. there will be cases where the fIR predicts instruments that are not in the mix which results in undefined F1 scores for that particular instrument. Given the fact that human users can bypass fIR, we assume that the ground truth instrument labels 4ind are the target instruments human users
transcribe.
When training fr standalone for 1,000 epochs, we can achieve an instrument-wise Fl score of 23.2. We presumed that joint training of fr and fass would result in better performance as the modules would help each other. Surprisingly, the joint training of fr and fMss from scratch results in lower instrument- wise score, 15.8. Only when we use pretrained fT for 500 epochs and then continue training both JT and fMss jointly, do we get higher F1 score, 24.7. We hypothesize two reasons for this phenomenon. First, we believe that it is due the noisy output y generated from fr in the early stage of joint training which confuses the fMss. The same pattern can be observed from the Source-to- to--sstortion Ratio (SDR) of the fMss which will be discussed in Section 5.3 Second, multi-task training often results a lower performance than training a model for single task, partly due to the difficulty in balancing multiple objectives. Our model might be example of such case.

---

Under review as a conference paper at ICLR 2023
Inst. Integrity
Note Continuity
Overall Note Acc.
Overall Lis. Exp.
MT3
2.66 + 1.04
2.64 + 1.04
2.76 + 1.05
2.60 + 1.06
Jointist
2.87 0.96
2.92 1.03
2.93 1.02
2.91 + 1.03
p-value
0.004
0.0001
0.02
2.7e-05
Table 3: Mean subjective scores (from to 5) based on 20 participants, each rated 20 songs. The bottom row presents the p-values using T-test for the comparison between MT3 and Jointist.
When comparing the conditioning strategy, the difference between the summation and the concate- nation modes is very subtle. The summation mode outperforms the concatenation mode by only 0.1 in terms of piece-wise Fl score: while the concatenation mode outperforms the summation mode by 0.001 in terms of flat F1 well as the wiisr F1. We believe that the model has learned to utilize piano rolls to enhance the mel spectrograms. And therefore, summing up both the piano rolls after the linear projection with the mel spectrograms is enough to achieve this objective, and therefore no obvious improvement is observed when using the concatenation mode.
Compared to the existing methods, our model (24.8) outperforms Omnizart (Wu et al. 2019) (1.9) by large margin in terms of Instrument-wiss F1. The instrument- wise transcription analysis in Figure 9 12 of Appendix El indicates that Jointist can handle heavily skewed data distributions (Figure 6 much better than Omnizart. While Omnizart only performs well for instruments with abundant labels available such drums and bass, Jointist still maintains competitive transcription accuracy as the instrument label availability decreases. This result proves that the proposed model with control mechanism (Jointist) is more scalable than the model (Omnizart) with fixed number of output instruments.
Although our proposed framework (58.4) did not outperform the MT3 model (76.0) on the note- wise Fl score, Jointist outperforms MT3 in all aspects of our subjective evaluation (see Table 3). We notice that, while MT3 is able to capture clean and well defined notes in classical music, struggles in popular music with complex timbre. Sometimes, the language model in MT3 dominates the acoustic model, producing a series of unexpected notes of transcription that are non-existent in the recording (examples can be found in our demo page'). MT3 is also less consistent in continuing the notes for an instrument, where a sequence of notes in the recording can jump back and forth between irrelevant instruments in the transcription, causing a notably lower perpetual score in "note continuity" in the subjective evaluation. Jointist is more robust than MT3 in these aspects. Moreover, Jointist is more flexible than MT3 in which human users can pick the target instruments that they are interested in. For example, if only a 'piano' condition is given, Jointist will only transcribe for piano. Whereas MT3 will simply transcribe all of the instrument found in the audio clip.
To test the robustness of proposed framework, we compare the flat transcription result between JIR generated [ (prefix 'i-' in Table [ and human specified conditions (assuming the target instruments are the instruments that users want to transcribe). We obtain a similar flat Fl score between the two approaches, which implies that fiR is good enough pick up all of the necessary instruments for the transcription. Because false positive piano rolls and false negative piano rolls have undefined Fl scores, we are unable report the piece-wise and instrument-wis Fl scores for fIR generated T The end-to-end transcription samples generated by Jointist are available onling")
5.3 SOURCE SEPARATION
Table shows that when fMss (TS) is jointly trained with fr, it outperforms a standalone trained fass (S). Similar to the discussion Section 5.2 training TS from scratch does not yield the best
result due to the noisy transcription output in the early stage which confuses the fass module. It can be seen that when using pretrained Sr. sMss able to achieve a higher SDR (pTS). Our
experimental results in both Section (.2tionn5.2mmm and Section 5.3 show that the joint training of fr and fMss helps both modules to escape local minima and therefore, achieve better performance compared to training them independently. A full instrument-wis SDR analysis is available in Figure 13 of Appendix E and the separated audio samples produced by Jointist are available online!.
7

---

Under review as a conference paper at ICLR 2023
Model
Instrument
Piece
Source
1.52
3.24
3.03
TS
1,86
3.55
3.32
pTS
2,01
3.72
3,50
U Uper bound TS
4.06
4.96
4.81
Table 4: Source-to-Distorrion Ratio (SDR) for different models. represents transcriptor, S rep- resents separator. The 'p-" prefix represent pretrained model. The last row shows the upper bound when we use the ground truth piano rolls for source separation.
Spectrograms XSIFT and posteriorgrams Y be combined using either "sum" (s) "cat" (c). But there is only a minor difference in instnumen//pccce/surre SDR between the former and latter 2.0013.72/3.50 dB vs. 1.92/3.75/3.52 dB). Instead of y frame the binary piano rolls rell can also be used as the condition for our fMss, however, this does not yield better result (Table 2 in Appendix F. We also explore the upper bound of source separation performance when the ground truth Y is used (last of Table 4). When /MSs has access accurate transcription results, its SDR can be greatly improved. This shows that AMT is an important MIR task that could potentially benefit other downstream tasks such music source separation. the next Section, will explore applications of the transcription output produced by Jointist.
6 APPLICATIONS OF JOINTIST
6.1 DOWNBEAT, CHORD, AND KEY ESTIMATIONS
intuitive to believe that symbolic information is helpful for beat and downbeat tracking, chord estimation, and key estimation. Literature has indicated that the timing of notes is highly related to beats (Matthew E. P. Davies (20TT. Downbeats, on the other hand, correspond to bar boundaries and are often accompanied by harmonic changes (Durand et al. 2016). The pitch information included in piano rolls offers explicit information about the musical key and chords (Pauws) 2004 Humphrey & Bello 2015). It thus also provides strong cues for modeling downbeats since they are correlated with harmonic changes. Given these insights, we attempt to apply our proposed hybrid representation to improve the performance these tasks. Detail descriptions of the experimental setup and datasets used are available in Appendix G
Table 5 presents the evaluation results for each task. We report the frame- wise Major/Minor score for chord, F1 score for downbeat tracking, and the song-level accuracy. We observe that the model with the hybrid representation can consistently outperform the one that uses only spectrograms, and this across all three tasks. This can be attributed to the advantage of piano rolls that provide explicit rhythmic and harmonic information to SpecTNN,, which frequency-aware (i.e., not shift invariant along the frequency axis).
6.2 MUSIC CLASSIFICATION
We experimented see the piano rolls produced by Jointist are also useful for music classification. The
dataset (Law et al.] (009) is a widely used benchmark in automatic music tagging research. We used m2lk tracks with top 50 tags following previous work by Won et al. 2020b
Eval Data
Input
Downbeat (F1)
Chord (MajMin)
Key (Acc)
Audio Only
72.8
79.8
72.8
Isophonics
Hybrid
74.6
81.2
75.2
Audio Only
62.0
76.6
RWC-POP
Hybrid
66.3
78.5
Table 5: Comparison of with and without pianoroll representation on each task on two datasets.
8

---

Under review as a conference paper at ICLR 2023
Input
Model
ROC-AUC
PR-AUC
Pons & Scrra
2019)
91.06
44,93
Audio
Lec ct al. (201 7
90.58
44.22
Only
Won el al.
2020b)
91.29
46.14
Won al.
2020a
91.27
46.11
Piano Roll Only Hybrid
Transformer Transformer
89.38
40.63
90.90
44.00
Table 6: Music tagging results on MagnaTTag-ATune Dataset. The audio-only baseline systems are MusiCNN Pons & Serra (2019), Sample-level CNN Lee al. (2017). Short-chunk ResNet Won et al. (220bb) and Harmonic CNN Won et al. (2020a).
Since music classification using symbolic data is a less explored area, we design a new architecture that is based on the music tagging transformer (Won et al. 2021a). The size of our piano roll input (B, 39, 2913, 88), where B is the batch size, 39 is the number of instruments, 2913 is the number of time steps, and 88 is the number of MIDI note bins. The CNN front-end has 3 convolutional blocks with residual connections, and the back-end transformer is identical to the music tagging transformer (Won al. (0211a. The area under the receiver operating characteristic curve (ROC- AUC) and the area under the precision-recal curve (PR-AUC) are reported as evaluation metrics.
As shown in Table 6 the hybrid model that uses both audio and piano roll features does not out- perform the audio-only model. We thus further experimented to see whether the piano roll as the only input is enough for music classification. Surprisingly, using piano roll-only yields competi- tive results. This opens up new possibilities to apply music classification to pure symbolic music datasets (or audio datasets converted symbolic using frameworks like Jointist). Introducing the symbolic features as the extra modality for classifier models also allows us to discover new aspects. We observed the following phenomena during our experiments:
The model performance for piano roll-only is comparable to audio-only methods when an instru- ment tag is one of the 39 instruments for which Jointist was explicitly trained (e.g., cello, violin, sitar), or genre is highly correlated with the Slakh2100 dataset (e.g., rock, techno).
The piano roll-only model performance drops when tag is not related to the 39 instruments (e.g.. female vocal, male vocal; compared guitar, piano), or when the tag is related acoustic characteristics that cannot be captured by piano rolls (e.g., quiet)
While these conclusions are interesting, this is exploratory work and an in-depth exploration is left as future work.
CONCLUSION
In this paper, we introduced Jointist, a flexible framework for instrument recognition, transcription, and source separation. In this framework, human users can easily provide the target instruments that they are interested in, such that Jointist transcribes and separates only these target instruments Joint training is used in Jointist mutually benefit both transcription and source separation performance. In our extensive experiments, we show that Jointist outperforms existing multi-instrumme music transcription models. In our listening study, we confirm that the transcription result produced by Jointist is on par with state-of-the-ar models as such MT3 (Hawthorne et al. (021). We then explored different practical use cases of our framework and show that transcriptions resulting from proposed framework can be useful to improve model performance for tasks such downbeat tracking, chord, and key estimation.
automatic
In the future, Jointist may be further improved, for instance, by replacing the ConvRNN with Trans- formers as done in Hawthhrme et al. (202T). From our experiments, the power of using symbolic representations for tasks such as tagging was shown, hence we hope more attempts to sym- bolic representations, complementary to or even replace audio representations, to progress towards more complete, multi-task music analysis system.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Emmanouil Benetos, Simon Dixon, Dimitrios Giannoulis, Holger Kirchhoff, and Anssi Klapuri. Au- tomatic music transcription: challenges and future directions. Journal of Intelligent Information Systems, 41(3):407-434, 2013.
Emmanouil Benetos, Simon Dixon, Zhiyao Duan, and Sebastian Ewert. Automatic music transerip- tion: An overview. IEEE Signal Processing Magazine, 36(1):20-30, 2018.
Adam Berenzweig, Beth Logan, Daniel PW Ellis, and Brian Whitman. A large-scale evaluation of acoustic and subjective music-similarity measures. Computer Music Journal, pp. 63-76, 2004.
S. Böck and M. EP Davies. Deconstruct, analyse, reconstruct: How to improve tempo, beat, and downbeat estimation. In 21th International Society for Music Information Retrieval Conference, ISMIR 2020, 2020.
Gino Brunner, Andres Konrad, Yuyi Wang, and Roger Wattenhofer. Midi-vae; Modeling dynamics and instrumentation of music with applications style transfer. arXiv preprint 1r0Xiv::809.000 2018.
John Ashley Burgoyne, Jonathan Wild, and Ichiro Fujinaga. An expert ground truth set for audio chord recognition and music analysis. In ISMIR, volume 11, pp. 633-638, 2011.
Rodrigo Castellon, Chris Donahue, and Percy Liang. Codified audio language modeling learns useful representations for music information retrieval. arXiv preprint arXiv:2107.05677 2021.
Ke Chen, Xingjian Du, Bilei Zhu, Zejun Ma, Taylor berg-kirkpatrick, and Shlomo Dubnov. Zero- shot audio source separation through query-based learning from weakly-labeled data. arXiv preprint arXiv:21 2021a.
Ting Chen, Saurabh Saxena, Lala Li, David J Fleet, and Geoffrey Hinton. Pix2seq: A language modeling framework for object detection. arXiv preprint arXiv:2 10852, 2021b.
Tian Cheng. Simon Dixon, and Matthias Mauch. A deterministic annealing em algorithm for auto- matic music transcription. International Conference of the Society of Music Information Retrieval, pp. 475-480, 2013.
Kin Wai Cheuk, Dorien Herremans, and Li Su. Reconvat: A semi-supervised automatic music transcription framework for low-resource real-world data. In Proceedings of the 29th ACM Inter- national Conference on Multimedia, pp. 3918-3926, 2021a.
Kin Wai Cheuk, Yin-Jyun Luo, Emmanouil Benetos, and Dorien Herremans. Revisiting the onsets and frames model with additive attention. In Proceedings of the International Joint Conference on Neural Networks, pp. In press. IEEE, 2021b. doi: 10. 11111999.200...0000
Keunwoo Choi, György Fazekas, Mark Sandler, and Kyunghyun Cho. Transfer learning for music classification and regression tasks. In 18th International Society for Music Information Retrieval Conference, ISMIR 2017, pp. 141-149. International Society for Music Information Retrieval, 2017.
Yi-Hui Chou, Chen, Chin-Jui Chang, Joann Ching, Yi-Hsuan Yang, et al. Midibert-piano: Large- scale pre-training for symbolic music understanding. arXiv preprint arXiv:2107.052233 2021.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555, 2014.
Jun-qi Deng and Yu-Kwong Kwok. Large vocabulary automatic chord estimation with
chance training scheme. In Proc. ISMIR pp. 531-536, 2017.
Simon Durand, Juan Bello, Bertrand David, and Gaël Richard. Feature adapted convolutional neural networks for downheat tracking. In 2016 IEEE international conference on acousties, speech and signal processing (ICASSP), pp. 296-300. IEEE, 2016.
Josh Gardner, Ian Simon, Ethan Manilow, Curtis Hawthorne, and Jesse Engel. Mt3: Multi-task multitrack music transcription. arXiv preprint arXiv:21 2021.
10

---

Under review as a conference paper at ICLR 2023
Masataka Goto, Hiroki Hashiguchi, Takuichi Nishimura, and Ryuichi Oka. RWC Music Database: Popular, classical and jazz music databases. In Prc. ISMIR, volume 2, pp. 287-288, 2002.
Fabien Gouyon. computational approuch rhythm RessriptiinnAAuio features for the compula- tion of rhythm periodicity functions and their use in tempo induction and music content process- ing. Universitat Pompeu Fabra, 2006.
Rui Guo, Ivor Simpson, Chris Kiefer, Thor Magnusson, and Dorien Herremans. Musiac: An extensi- ble generative framework for music infilling applications with multi-level control. International Conference Computational Intelligence in Music, Sound, Art and Design (Part of EvoStar), pp. 341-356. Springer, 2022.
Stephen W Hainsworth and Malcolm Macleod. Particle filtering applied to musical tempo track- ing. EURASIP Journal on Advances in Signal Processing, 2004(15):1-1,, 2004.
Curtis Hawthorne, Erich Elsen, Jialin Song, Adam Roberts, Ian Simon, Colin Raffel, Jesse Engel, Sageev Oore, and Douglas Eck. Onsets and frames: Dual-objective piano transcription. arXiv preprint arXiv:1710.1115: 2017.
Curtis Hawthorne, Andriy Stasyuk, Adam Roberts, Ian Simon, Cheng-Zhi Anna Huang, Sander Dieleman, Erich Elsen, Jesse Engel, and Douglas Eck. Enabling factorized piano music modeling and generation with the maestro dataset. arXiv preprint arXiv:/810.12247, 2018.
Curtis Hawthorne, lan Simon, Rigel Swavely, Ethan Manilow, and Jesse Engel. Sequence-to- sequence piano transcription with transformers. arXiv preprint arXiv:2107.09 142, 2021.
Andre Holzapfel, Matthew EP Davies, José R Zapata, João Lobato Oliveira, and Fabien Gouyon. Selective sampling for beat tracking evaluation. IEEE Transactions on Audio, Speech, and Lan- guage Processing, 20(9):2539-2544 2012.
Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Ian Simon, Curtis Hawthorne, Noam Shazeer, Andrew M Dai, Matthew Hoffman, Monica Dinculescu, and Douglas Eck. Music transformer: Generating music with long-term structure. In International Conference on Learning Representations, 2018
Eric J Humphrey and Juan Pablo Bello. Four timely insights on automatic chord estimation. In Proc. ISMIR, volume 10, pp. 673-679, 2015.
Yun-Ning Hung, Gordon Wichern, and Jonathan Le Roux. Transcription is all you need: Learning to separate musical mixtures with score as supervision. In ICASSP 2021 1021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 46-50. IEEE, 2021.
Yun-Ning Hung, Ju-Ch Chianin Wang, Xuchen Song, Wei-Tsung Lu, and Minz Won. Modeling beats and downbeats with a time-frequency transformer. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 401-40S IEEF, 2022.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448 448-456. PMLR, 2015.
Andreas Jansson, Eric Humphrey, Nicola Montecchio, Rachel Bittner, Aparna Kumar, and Tillman Weyde. Singing voice separation with deep u-net convolutional networks. 18th International Society for Music Information Retrieval Conference, ISMIR 2017, 2017.
Andreas Jansson, Rachel M. Bittner, Sebastian Ewert, and Tillman Weyde. Joint singing voice sep- aration and f0 estimation with deep u-net architectures. In 2019 27th European Signal Processing Conference (EUSIPCO), pp. 1-5, 2019. doi: wTT.:///SSSIC.OO..9992..22
Rainer Kelz, Sebastian Böck, and Gerhard Widmer. Deep polyphonic adsr piano note transcription. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Process- ing (ICASSP), pp. 246-250. IEFF, 2019.
11

---

Under review as a conference paper at ICLR 2023
Jong Wook Kim and Juan Pablo Bello. Adversarial learning for improved onsets and frames music transcription. International Society forMusic Information Retrieval Conference, pp. 670-677, 2019.
Sunghyeon Kim, Hyeyoon Lee, Sunjong Park, Jinho Lee, and Keunwoo Choi. Deep composer
classification using symbolic representation. ISMIR Late Breaking Demo Session, 2020.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. Proceedings of the 3rd International Conference Learning Representations (ICLR), 2014.
Anssi Klapuri and Antti Eronen. Automatic transcription of music. In Proceedings of the Stockholm Music Acoustics Conference, pp. 6-9. Citeseer, 1998.
Qiugiang Kong, Keunwoo Choi, and Yuxuan Wang. Large-scale midi-based composer elassification. arXiv preprint arXiv:2010. 14805, 2020a.
Qiuqiang Kong, Bochen Li, Jitong Chen, and Yuxuan Wang. Giantmidi-piano. large-scale midi dataset for classical piano music. arXiv preprint arXiv:2010.07061, 2020b.
Qiuqiang Kong, Bochen Li, Xuchen Song, Yuan Wan, and Yuxuan Wang. High-resolution piano transcription with pedals by regressing onset and offset times. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3707-3717, 2021.
Florian Krebs, Sebastian Böck, and Gerhard Widmer. Rhythmic pattern modeling for beat and downbeat tracking in musical audio. In ISMIR, pp. 227-232. Citeseer, 2013.
Edith Law, Kris West, Michael Mandel, Mert Bay, and Stephen Downie. Evaluation of algorithms using games: The case of music tagging. In Conference of the International Society for Music Information Retrieval Conference (ISMIR), 2009.
Huu Le, Rasmus Kjær Høier, Che-Tsung Lin, and Christopher Zach. Adaste: An adaptive straight- through estimator to train binary neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 460-469, 2022.
Jongpil Lee, Jiyoung Park, Keunhyoung Luke Kim, and Juhan Nam. Sample-level deep con- volutional neural networks for music auto-tagging using raw waveforms. arXiv preprint arXiv:1703.01789, 2017.
Liwei Lin, Qiuqiang Kong, Junyan Jiang, and Gus Xia. unified model for zero-shot music source separation, transcription and synthesis. In International Society for Music Information Retrieval (ISMIR), 2021.
Wei-Tsung Lu, Ju- Ju-Chiang Wang, Minz Won, Keunwoo Choi, and Xuchen Song. Spectnt: a time- frequency transformer for music audio. arXiv preprint arXiv:2110.09127, 2021.
Yin-Jyun Luo, Yuen-Jen Lin, and Li Su. Toward expressive singing voice correction: On perceptual validity of evaluation metries for vocal melody extraction. arXiv preprint arXiv:2010.12196. 2020.
Ethan Manilow, Gordon Wichern, Prem Seetharaman, and Jonathan Le Roux. Cutting music source separation some Slakh: A dataset study the impact of training data quality and quantity. In Proc. IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA). IEEE, 2019.
Ethan Manilow, Prem Seetharaman, and Bryan Pardo. Simultaneous separation and transcription of mixtures with multiple polyphonic and percussive instruments. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 771-775 IEEE, 2020.
Ugo Marchand and Geoffroy Peeters. Swing ratio estimation. In Digital Audio Effects 2015 (Dafx15), 2015.
Magdalena Fuentes Matthew E. P. Davies, Sebastian ock. Tempo, Beat and Downbeat Es- 2021. URL https://
timation. http:://cumbbbbabbbbbbbbiiibbbbb Cemppbberroommbbtt.............d giubb..
12

---

Under review as a conference paper at ICLR 2023
M. Mauch, C. Cannam, M. Davies, S. Dixon, C. Harte, S. Kolozali, D. Tidhar, and M. Sandler. OMRAS2 metadata project 2009. ISMIR Late Breaking and Demo, 2009.
Gabriel Meseguer-Browe Brocal and Geoffroy Peeters. Conditioned-u-nett Introducing a control mecha- nism in the u-net for multiple source separations. arXiv preprint arXiv: 2019.
Giyasettin Ozcan, Cihan Isikhan, and Adil Alpkocak. Melody extraction on midi music files. In Seventh IEEE International Symposium on Multimedia (ISM 005), pp. 8 8-pp. leee, 2005.
Jonggwon Park, Kyoyun Choi, Sungwook Jeon, Dokyun Kim, and Jonghun Park. A bi-directional transformer for musical chord recognition. arXiv preprint 19Xiv:/907.02200 2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury. Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 32, 2019.
Jouni Paulus and Anssi Klapuri. Model-based event labeling in the transcription of percussive audio signals. In Proc. Int. Conf. Digital Audio Effects (DAFX), pp. 73-77. Citeseer, 2003.
Steffen Pauws. Musical key extraction from audio. 5th International Society for Music Informa- fion Retrieval Conference, 2004.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 2011.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with general conditioning layer. In Proceedings of the AAAI Conference Artificial Intelligence, volume 32, 2018.
Martin Piszczalski and Bernard A Galler. Automatic music transcription. Computer Music Journal, pp. 24-31, 1977.
Jordi Pons and Xavier Serra. musicnn: Pre-trained convolutional neural networks for music audio tagging. arXiv preprint arXiv: 1909199699444 2019.
Colin Raffel. Learning- Learning-based based methods for comparing sequences, with applications audio-to-midi aligument and matching. Columbia University, 2016.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:1-67, 2020.
Jing Shi, Xuankai Chang, Shinji Watanabe, and Bo Xu. Train from scratch: Single-stage joint training of speech separation and recognition. Computer Speech & Language, 76:101387, 2022.
ISSN 0885-2308.
Siddharth Sigtia, Emmanouil Benetos, and Simon Dixon. An end-to-end neural network for poly- phonic piano music transcription. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 24:927-939, 2015.
Federico Simonetta, Federico Avanzini, and Stavros Ntalampiras. A perceptual measure for evalu- ating the resynthesis of automatic music transcriptions. Multimedia Tools and Applications, pp.
1-21,2022
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 1995 2014.
Keitaro Tanaka, Takayuki Nakatsuka, Ryo Nishikimi, Kazuyoshi Yoshii, and Shigeo Morishima. music transcription based on deep spherical clustering of spectrograms and pitchgrams. In International Society for Music Information Retrieval Conference (ISMIR), Mon- freal, Canada, 2020.
13

---

Under review as a conference paper at ICLR 2023
John Thickstun, Zaid Harchaoui, and Sham M. Kakade. Learning features of music from scratch. In ICLR, volume abs/161 1.09827, 2016.
John Thickstun, Zaid Harchaoui, Dean P. Foster, and Sham M. Kakade. Invariances and data aug- mentation for supervised music transcription. 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2241-2245, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- fion processing systems, 30, 2017.
Emmanuel Vincent, Rémi Gribonval, and Cédric Févotte. Performance measurement in blind audio source separation. IEEE transactions audio, speech, and language processing, 14(4): 1462- 1469, 2006.
Richard Vogl, Matthias Dorfer, Gerhard Widmer, and Peter Knees, Drum transcription via joint beat and drum modeling using convolutional recurrent neural networks. 18th International Society for Music Information Retrieval Conference, ISMIR 2017, pp. 150-157, 2017.
Ju- Lu-Chiang Wang, Yun-Ning Hung, and Jordan Smith. To catch chorus, verse, intro, anything else: Analyzing song with structural functions. In Proc. ICASSP, pp. 416-420 420,, 2022.
Minz Won, Sanghyuk Chun, and Xavier Serra. Toward interpretable music tagging with self- attention. arXiv preprint 19XXiv:900.000
2019.
Minz. Won, Sanghyuk Chun, Oriol Nieto, and Xavier Serre. Data-driven harmonic filters for au- dio representation learning. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 536-540. IEEE, 2020a.
Minz Won, Andres Ferraro, Dmitry Bogdanov, and Xavier Serra. Evaluation of cnn-based automatic music tagging models. arXiv preprint arXiv:2006.0075), 2020b.
Minz Won, Keunwoo Choi, and Xavier Serra. Semi-supervised music tagging transformer. Confer- ence of the International Society for Music Information Retrieval Conference (ISMIR), 2021a.
Minz Won, Janne Spijkervet, and Keunwoo Choi. Music Classification: Beyond Supervised Learn- ing, Towards Real-world Applications.
2021b.
doi: 10.528817emmdo.55737779 URI. tutorial
1o/
Xianchao Wu, Chengyuan Wang, and Qinying Lei. Transformer-xl based music generation with multiple sequences of time-valued notes. arXiv preprint arXiv:2007.07244, 2020a.
Yiming Wu and Wei Li. Automatic audio chord recognition with midi-trained deep feature and bistm-crf sequence decoding model. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 27(2):355-366 -666555--666666 2018
Yu-Te Wu, Berlin Chen, and Li Su. Polyphonic music transcription with semantic segmentation. In ICASSP 2019- 2019 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 166-170, 2019. doi: 10. 10/1/////S 8.866665
Yu-Te Wu, Berlin Chen, and Li Su. Multi-instrument automatic music transcription with self- attention-based instance segmentation. IEEE/ACM Trunsuctions on Audio, Speech, and Language Processing, 28:2796-2809, 2020b.
Yu-Te Wu, Yin-Jyun Luo, Tsung-Ping Chen, I Wei, Jui- Juig Hsu, Yi-Chin Chuang, Li Su, et al. Omnizart: A general toolbox for automatic music transcription. arXiv preprint arXiv:2106.00497, 2021.
Yao- Yuan Yang, Moto Hira, Zhaoheng Ni, Artyom Astafurov, Caroline Chen, Christian Puhrsch, David Pollack, Dmitriy Genzel, Donny Greenberg, Edward Z Yang. et al. Torchaudio: Building blocks for audio and speech processing. In ICASSP 2022- 2022 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 6982-6902 6986. IEFF, 2022.
14

---

Under review as a conference paper at ICLR 2023
Penghang Yin, Jiancheng Lyu, Shuai Zhang, Stanley J. Osher, Yingyong Qi, and Jack Xin. Under- standing straight-througe estimator in training activation quantized neural nets. In International Conference Learning Representations, 2019. URLhttps: / wpwnreeiew.net/eorouueee 1d=Skh4 jRcKk
15

---

Under review as a conference paper at ICLR 2023
A
MODEL ARCHITECTURES
CLS taken
spectrogram Xme]
feature maps
Transformer
one-hot conditions
Encoder
Linear
Sigmoid
cond
one-hot
10
10
cond
encoding
Figure 2: Model architecture for the fIR module which consists of CNN backbone and trans- former encoder. The classification (CLS) token in the transformer encoder is used to extract the musical instruments present in the spectrogram. The CNN front end has six convolutional blocks with output channels (64, 128, 256, 512, 1024, 2048]. Each convolutional block has two convolu- tional layers with kernel size (3,3), stride (1,1), and padding (1,1), followed by average pooling (2,2). Sigmoid activation is used to convert the output into probability. Finally, a threshold (p >> 0.5) applied to the sigmoid output and obtain the one-hot vectors as the instrument conditions. Note that due to the one-hot nature of the instrument condition, human users can easily take over fIR and specify the target instruments they are interested. Otherwise, all 39 instruments conditions could be used to perform full-fledge transcription, make sure of fiR to automatically determine the instruments present in the audio. This makes Jointist flexible framework.
ft
framm
CNN
Frame
P posteriorgram
Prame
Proll plano roll
spectrogram Xmel
BatchNorm
bgg
Threshold
1'ond
Onset CNN
onset
fP
Figure 3: Model architecture for the fr module. It consists of a batch normalization layer (loffe & Szegedy 2015), a frame model ffrime, and an onset model fameet Both ffame and fr follow the same architecture which consist of four conditional convolutional blocks that takes in j via a FiLM layer, followed by a linear layer with ReLU activation, one layer of bidirectional GRU (biGRU) (Chung et al. 2014) of 256 hidden dimension, and linear layer with sigmoid activation which projects the hidden dimension into 88 notes. Each conditional convolutional block contains two 2D convolutional layers with batch normalization and ReLU activation followed by average pooling size of (2,2) 2).
The outputs from firme and gonset concatenated together, resulting in tensor of 172 dimension. Then this tensor is passed to another biGRU layer with 256 hidden dimensions followed by linear layer with sigmoid activate which projects the hidden features back into 88 the posteriorgram V frame-
16

---

Under review as a conference paper at ICLR 2023
Source Iz
Transcription Module ft
Phase Information
STFT
iSTFT
p
XSTFT
Mask
Multiply
Linear
BatchNorm
Sigmoid
4
g
Conv Block
lonnd
U-net Encoder
U-net Decoder
Skip Connection
Conv Block
Bottleneck Feature
Figure 4: Model architecture for the fMss module has a U-net as the main component which takes XSIFT as input. FiLM is added to the U-net such that the output source Y can be controlled by î Poond A linear layer used to project Y XSTFT- Then the U-net encoder takes in the merged feature of the batch normalized XSIFT and g as the input. Different merging methods such summation and concatenation well spectrogram patch has been studied in Table 9 In the spectrogram patch approach, we added (sum) our posteri- orgram feature g directly into
fr(X (i into tensor with the same shape as
The U-net encoder consists of six convolution blocks with output channels [32, 64, 128, 256, 384, 384]. Each convolution block contains two bias-less convolutional layers with kernel size (3,3), stride size (1,1), padding (1.1) and average pooling size (2,2) followed by batch normalization layer. The output of the convolutional layers are combined with FiL.M condition as Meseguer- Brocal & Peeters (2019). There is an extra convolution block after the encoder which has both input and output channel equal to 384, kernel size of (2,2) and padding (1,1).
The U-net decoder follows a similar architecture the U-net encoder to reverse the process. It consists of six Transposed convolution blocks with output channels [384, 384, 256, 128, 64, 32]. Each convolution block contains one bias-less transposed convolutional layers with kernel size (3,3), stride size (1.1), padding (0,0) followed by a batch normalization layer. Similarly, there a convo- lution block after the decoder which has both input and output channel equal 32, kernel size of (1,1) and padding of (0,0).
The output of the U-net Decoder a spectrogram mask with values between [0.1]. The original mag- nitude STFT (mix) is multiplied with this mask to produce the separated magnitude STFT (source). To invert it back the waveform, we use inverse STFT which requires also the phase information. Since our separated magnitude STFT has no phase, we use the original phase information in the mix STFT.
We also conducted an ablation using the continuous value posteriorgrams Y frame E [0, versus the binary value piano rolls Y {0, 1} in Table 9 and we found that using posteriorgrams is generally better than piano rolls.
17

---

Under review as a conference paper at ICLR 2023
B EVALUATION METRICS
B. MEAN AVERAGE PRECISION
We first calculate the instrument-wise average precision (AP)) for each audio sample s using scikit- learn package (Pedregosa et al. 2011) which is defined
AP; E -
(1)
where where P and Rt are the precision and recall at the k-th threshold for the instrument defined in I1
Then the mean average precision (m.Pi)) for instrument across the test set with size N can be calculated using
MAPi
AP
N
This metric measures the instrument recognition performance across different threshold values, and hence it is not affected by an incorrectly selected threshold value. Due to the skewed distribution of different instrument labels (as shown in Figure [), report both the macro and weighted mAP.
The macro mAP calculate the average mAP score across different instrument unweighted as
E/= MAP
macro mAP
according Table 11 Macro mAP susceptible to instruments such Bassoon, Cello, and Recorder with only few labels and hence is not a reliable metries when the data distribution is heavily skewed. But gives us an idea on whether model is doing well across all instrument classes. low macro score implies that model performance poorly on a specific instrument classes.
where the total number of instruments which
To account for the skewed data distribution, also report weighted mAP which is calculated
weighted mAP E/1 (MAP'
I w(i)
I
where wo() is the weighting for the i-th instrument proportional its label count in the test set. This metric is more reliable than the macro metric, since it weights more towards classes with more labels and less towards classes with less labels. In other words, one prediction mistake in instrument classes with few labels would not seriously affect the weighted mAP score.
With both metrics reported together with Figure we understand that our model performance is highly correlated to the label availability as shown in Figure 6
B.2 NOTE-WILH TRANSCRIPTION FI SCORE
We use the standard transcript on. oo..oonpppree.e preci si sioonnecaalllaeooo __f1_overiappppp function from mir_eval calculate the note-wise Fl score (notated as F2) for each instrument present in each audio sample s. We follow the standard note-wise Fl setting where both the predicted pitch needs to be same as the ground truth label and the predicted onset needs be within 50ms of the ground truth onset in order to be considered as a correct transcription.
Our wiece-wise note-wise F1 score is computed
Fl piece =
(5)
where I, a variable representing the number of ground truth instruments present the audio sample and S is constant representing the total number of audio samples in the test set. While
18

---

Under review as a conference paper at ICLR 2023
this metrics reflect how well model performance in general, it provides no information which instrument our model is particularly good at or bad at.
To measure the performance different instruments proposed the instrument-wis note-wise Fl score which defined as
FI instrument = = - 5
Σ F
(6)
where S; is variable representing the number of audio samples that contains instrument and T= 39 is constant representing the number of musical instruments supported by our model as listed Table II] Note that although instrument-win evaluation was mentioned (Cheng et al. 2013), they have always 10 instruments in each audio sample which makes their evaluation method much simpler than ours. our case, we have varying number of instruments per samples. Hence, the instrument-win wise evaluation for Slakh2100 is not as straight forward as (Cheng et al.) (2013).
B.3 SOURCE-TO-DISTORNE RATIO (SDR)
We followed the standard SDR defined in Vincent et al. 2006).
Starger |
SDR = 10
Strget
8|
where Staget the ground truth waveform and is the predicted waveform. Since we have different number of instruments 7s per audio sample in Slakh2100, we need to define new metrics to handle this case. For the sake of disccusion below, we define SDR,, the SDR for instrument of sample
source-wise SDR is defined as the mean SDR for all sources Na present in the dataset,
1 ΣΣΣ SDR
SDR soarce
(8)
Nall
This metric is the most straight forward yet the least informative one.
To understand how well model performs for each audio sample (music piece), define piece- wise SDR as follows
- Ns N! Σ SDR
SDR
where N/. is the number of instruments for audio sample (music piece) and Ns is the total number of audio samples (music pieces) in the test set. This metric allows us to understand the model performance for each piece.
Similarly, to understand the model performance for each instrument, instrument-wise SDR is used. It is defined as
SDR instrument
1 Σ N
Σ SDR
(10)
where N3 the number of pieces containing instrument and 39 in Equation
B.4 SUBJECTIVE METRICS
Instrument integrity: How accurate is the recognition of important instruments? If an impor- tant instrument is missed or falsely included as a dominant instrument, it is regarded as poor in instrument integrity.
19

---

Under review as a conference paper at ICLR 2023
Instrument-wise note continuity: How is the note continuity of each important instrument? For instance, a sequence of piano notes are assigned by the system to guitar, organ, and piano from segment to segment, it is regarded as poor note continuity for piano.
Overall note accuracy: How accurate are the notes of important instruments when listening to them as whole? Do they miss important notes or insert false notes?
Overall listening experience: What is the overall quality of the transcription?
B.5
RATING CRITERIA FOR SUBJECTIVE EVALUATION OF TRANSCRIPTION
Table list the score criteria for the four aspects proposed in Section
Score
Criterion for Instrument Integrity
5
Perfect.
4
Some unimportant instruments are missed or falsely recognized, and they do not affect the overall listening experience.
3
Some important instruments missed or falscly recognized, and they moderately affect the overall listening experience.
2
More important instruments missed or falsely recognized, and they obviously de- grade the overall listening experience. Instruments recognized completely wrong.
Score
Criterion for Instrument-wise Note Continuity
5
Perfect.
4
Some unimportant instruments' notes are less continuous, and they do not affect the overall listening experience.
3
Some important instruments' notes not continuous, and they moderately affect the overall listening experience.
2
More important instruments' notes not continuous, and they obviously degrade the overall listening experience.
The notes are assigned to instruments random, where no rule can be concluded.
Score
Criterion for Overall Note Accuracy
5
Perfect.
4
Some unimportant instruments' notes less accurate, and they do not affect the overall listening experience.
3
Some important instruments' notes are inaccurate, and they moderately affect the overall listening experience.
2
More important instruments' notes inaccurate, and they obviously degrade the over- all listening experience.
The notes are completely messy, and one cannot identify the song from the notes.
Score
Criterion Overall Listening Experience
5
Excellent
4
Good
3
Fair
2
Poor
1
Awful
Table 7: Rating criteria for the subjective transcription quality.
C METADATA FOR SUBJECTIVE EVALUATION
20

---

Under review as a conference paper at ICLR 2023
20.0%
25.0%
5.0%
Composer
5.0%
Amateur inst. player
Pro Inst. player
10.0%
MIR researcher
35.0%
Music producer
Other
Figure 5: Music background for the subjective evaluation participants.
Artist
Song Title (or ID)
Isophonics
Beatles
Let Be
Beatles
Lucy In The Sky With Diamonds
Beatles
Yellow Submarine
Michael Jackson
Black or White
Michael Jackson
I Want You Back
Michael Jackson
Off the Wall
Queen
Bohemian Rhapsody
Queen
I Want to Break Free
RWC-POP
RWC
RM-PP04
RWC
RM-P033
RWC
RM-P047
RWC
RM-PO83
RWC
RM-P096
JayChou29
Jay Chou
chao-ren-bu-hui-f
Jay Chou
gei-wo-yishhou-e-de-hhii
Jay Chou
ju-hua-tai
USPOP2002
PapaRoach
Last Resort
Radiohead Ricky Martin
Karma Police
Livin La Vida Loca
Spice Girls
Become One
Table 8: Metadata of the selected 20 full-tracks for subjective evaluation. The audio of Isophon- ics, JayChou29, and USPOP2002 may be available at (https: :tttps://githunn com hoom/rrist111/ chords - recognition
21

---

Under review as a conference paper at ICLR 2023
D DATA DISTRIBUTION
Instrument counts (train set)
10
10
10a
Drums
Bass
Electric Guitar
Piano
Strings
Acoustic Guitar
Electric Piano
Voice
Synth Pad
Saxophone
Organ
Synth Lead
Chromatic Percussion
Brass
Flute
Pipe
Clavinet
Harpsichord
Trumpet
Harmonica
Trombone
Horn
Clarinet
Oboe
Violin
Accordion
Recorder
Cello
Piccola
Timpani
Harp
Tuba
Bassoon
Viola
Contrabass
Sound Effects
Percussive
Ethnic
Synth Effects
10a
102
10a
Figure 6: Instrument label count for the train set of Slakh2100. The label count in log scale.
22

---

Under review as a conference paper at ICLR 2023
Instrument counts (test set)
10
10
102
Drums
Bass
Piano
Electric Guitar
Strings
Acoustic Guitar
Electric Piano
Synth Pad
Organ
Saxophone
Voice
Synth Lead
Chromatic Percussion
Brass
Pipe
Harpsichord
Harmonica
Flute
Trombone
Trumpet
Clavinet
Oboe
Horn
Accordion
Tuba
Harp
Violin
Timpani
Viola
Recorder
Piccolo
Clarinet
Sound Effects
Percussive
Ethnic
Synth Effects
Bassoon
Contrabass
Cello
10º
101
10a
Figure 7: Instrument label count for the test set of Slakh2100. The label count is in log scale.
23

---

Under review as a conference paper at ICLR 2023
E
INSTRUMENT-ISE PERFORMANCE
F1
0.0
0.2
0.4
0.6
0.8
1.0
Recorder
Clarinet
Bass
Drums
Acoustic Guitar
Electric Guitar
Brass
Piano
Strings
Electric Piano
Trumpet
Voice
Chromatic Percussion
Synth Pad
Synth Lead
Saxophone
Accordion
Trombone
Organ
Horn
Tuba
Viola
Pipe
Flute
Harmonica
Oboe
Harp
Vioiin
Clavinet
Harpsichord
Sound Effects
Percussive
Ethnic
Synth Effects
Piccolo
Bassoon
Timpani
Contrabass
Cello
0.0
0.4
0.8
Figure 8: Instrument-wise Fl score for the fIR module in the test set of Slakh2100.
24

---

Under review as a conference paper at ICLR 2023
Transcription notewise F1
0.0
0.2
0.4
0.6
0.8
1.0
Bass
Recorder
Drums
Harpsichord
Acoustic Guitar
Tuba
Contrabass
Clarinet
Electric Piano
Trumpet
Piano
Voice
Electric Guitar
Trombone
Oboe
Accordion
Clavinet
Saxophone
Strings
Brass
Pipe
Violin
Harp
Harmonica
Synth Lead
Chromatic Percussion
Organ
Flute
Bassoon
Piccolo
Hom
Synth Pad
Cello
Viola
Timpani
0.0
0.2
0.4
0.6
0.8
1.0
Figure 9: Instrument-wise note Fl score for the fr module in the test set of Slakh2100. The red line represents the average Fl score.
25

---

Under review as a conference paper at ICLR 2023
Transcription notewise offset F1
0.0
0.2
0.4
0.6
0.8
1.0
Bass
Contrabass
Recorder
Trumpet
Trombone
Accordion
Clarinet
Tuba
Brass
Clavinet
Saxophone
Strings
Synth Lead
Violin
Voice
Electric Guitar
Oboe
Harpsichord
Organ
Cello
Electric Piano
Harmonica
Pipe
Hor
Piano
Acoustic Guitar
Flute
Harp
Synth
Chromatic Percussion
Bassoon
Piccolo
Timpani
0.0
0.2
0.4
0.6
0.8
1.0
Figure 10: Instrument-wee wise note with offset Fl score for the fr module in the test set of Slakh2100, The red line represents the average Fl score.
26

---

Under review as a conference paper at ICLR 2023
Transcription notewise F1 (baseline)
0.4
0.6
0.8
Drums
Bass
Piano
Acoustic Guitar
Electric Guitar
Electric Piano
Brass
Saxophone
Trumpet
Strings
Synth Lead
Clavinet
Recorder
Voice
Harpsichord
Trombane
Organ
Synth
Flute
Accordion
Pipe
Violin
Cello
Clarinet
Chromatic Percussion
Piccolo
Contrabass
Bassoon
Oboe
Harp
Viola
Tuba
Harmonica
Timpani
Hom
0.4
0.6
0.8
Figure 11: Instrument-witd wise note Fl scores (with and without offset) for the baseline model Wu et al. 1202 in the test set of 100. The red line represents the average Fl score.
27

---

Under review as a conference paper at ICLR 2023
Transcription notewise-offset F1 (baseline)
0.0
0.2
0.4
0.6
0.8
1.0
Brass
Electric Guitar
Saxophone
Piano
Trumpet
Electric Piano
Acoustic Guitar
Strings
Trombone
Clavinet
Synth Lead
Voice
Organ
Synth Pad
Harpsichord
Recorder
Pipe
Accordion
Chromatic Percussion
Flute
Piccalo
Contrabass
Bassoon
Cello
Oboe
Harp
Violin
Viola
Tuba
Harmonica
Timpani
Horn
Clarinet
0.0
0.2
0.4
0.6
0.8
1.0
Figure 12: Instrument-wise note F1 scores (with and without offset) for the baseline model Wu et al. 202T in the test set of 100. The red line represents the average Fl score.
28

---

Under review as a conference paper at ICLR 2023
Instrument-wise SDR
-15
-10
10
Recorder
Bass
Clarinet
Strings
Flectric Guitar
Voice
Acoustic
Contrabass
Pipe
Harp
Harpsichord
Hube
Electric Plano
Organ
Harmonica
Accordion
Clavinet
Plano
Trmppe
Synth Load
Trombone
Brass
Chromatic Percussion
Hom
Piccolo
Synth Pad
Saxophone
Bassoon
Cello
Oboe
Timpani
15
100
10
Figure 13: Instrument-wise source-to-die nource-toddittortion ratio (SDR) Fl score for the fr module in the test of 100. The red line represents the average Fl score.

---

Under review as a conference paper at ICLR 2023
F
ABLATION STUDY
Full length SDR
10s SDR
feature merge
Model
inst
piece
source
inst
piece
source
TS (posterior)
1.86
3.55
3.32
3.47
5.60
4.54
PTS (posterior)
2.01
3.72
3.50
3.69
5.86
4.81
sum
TS+STE (binary)
1.30
3.30
3.06
2.40
5.13
4.16
PTS+STE (binary)
1.67
3.30
3.06
3.29
5.36
4.41
TS (posterior)
1.80
3.53
3.31
3.21
5.50
4.51
pTS (posterior)
1.92
3.75
3.52
3.50
6.03
4.90
concat
TS+STE (binary)
1.89
3.60
3.37
3.63
5.70
4.66
PTS+STE (binary)
1.72
3.47
3.22
3.18
5.78
4.61
TS (posterior)
1.79
3.46
3.24
3.35
5.30
4.43
pTS (posterior)
1.91
3.61
3.40
3.58
5.52
4.65
spec patch
TS+STE (binary)
1.74
3.46
3.24
3.50
5.28
4.42
PTS+STE (binary)
1.71
3.47
3.23
3.28
5.40
4.53
Table 9: Ablation study for the source-to-distorie ratio (SDR) achieved by fMss when fr (T) and fasss (S) are jointly trained together. Similar Table 2 the prefix 'p-" represents that the transcription module is pretrained. In this study, we also assume that the ground truth instruments are the target instruments that human users intended to transcribe as in Table
Three merging modes between the spectrograms XSIFT and the transcription features 9 are studied (Figure 4p. In the 'sum' mode, the XSTFT after the batch normalization layer simply adds to g. The resulting merged tensor has the same dimension as XSTFT and g. In the 'concat' mode, XSTFT and g concatenated together forming tensor with frequency dimension double the size of XSTFT and g. The 'concat' mode generally performs better than the 'sum' in exchange of a higher computation complexity. We also explored 'spec patch' mode where g is added XSTPT directly before the batch normalization. The model predicted mask is applied to this modified spectrogram XSIFI + in this merge mode. We want to know if g can be used to enhance the spectral features in XSIFT corresponding to the target instrument i. It turns out this merging mode performs the worst among the three modes. Since the difference SDR is not very significant between the 'sum' and "concat' modes, we stick to the 'sum' mode.
In Table 2 it is shown that when using the ground truth piano roll, which is binary in na- ture, the source separation performance improved by a large margin. To understand whether the binary nature of piano roll or the fully accurate transcribed notes contribute the boosted perfor- mance, we also experimented with two forms of transcription output: posteriorgram Yiamo E [o. 1] and piano roll Yi E {0, 13. When converting Yimme into Y thresholding required, which destroys the gradient information in it. To keep the gradient information, we apply straight- estimation (STE) Yin et al.] (2019); Le et al. (002)) as the thresholding method to preserve the gradient information. The results indicate that using binary piano rolls Yi has advantage the posteriorgrams y frame
G EXPERIMENTAL SETUP FOR DOWNBEAT, CHORD, AND KEY ESTIMATIONS
For audio processing, we use 6-channel harmonic spectrograms (128 frequency bins) from Won al. (200aa). We simplify the piano rolls produced by Jointist into two channels (instrument index 0-37 as channel 0 and index 38 as channel 1). We then use a 1-D convolution layer to project the piano rolls into the same frequency dimension the spectrograms. The resulting hybrid representation is a concatenation of both the spectrograms and piano rolls, i.e., 8-channel.
SpecTNT (Lu et al. 2021), ofate-of-the-ar Transformer architecture, the temporal musical events audio recordings (Hung et al. 2022) Wang et al.) (022). The times- tamp and label annotations converted into temporal activation curves for the learning targets for
chosen for modeling
30

---

Under review as a conference paper at ICLR 2023
SpecTNTT as done in Lu et al. (2021): Hung et al. (002 Table 10 below summarizes our SpecTNT configurations for the three tasks.
We train the beat and downbeat tracking tasks jointly following Hung et al. (2022). but focus downbeat evaluation, which more challenging than beat tracking (Böck & Davies 2020 Durand al. 2016). We consider 24 classes of major and minor triad chords for chord estimation, and 24 classes major and minor keys for key estimation, plus a "none" class for both tasks. The key and chord tasks are trained and evaluated separately.
For downbeat tracking, we use 7 datasets: Ballroom (Krebs et al. 2013), Hainsworth (Hainsworth
& Macleod 2004), SMC (Gouyon 2006). Simac (Holzapfel al. 2012), GTZAN (Marchand & Pecters 2015). Isophonics (Mauch et al. 22009) and RWC-POP (Goto et al. 2002). We use both the Isophonics and RWC-POP for evaluation while the remaining datasets are used for training. For chord estimation, we use Billboard (Burgoyne et al. 2011 Isophonics, and RWC-POP. We use both Isophonies and RWC-POP for evaluation while the remaining datasets are used for training. For key estimation, we use Isophonics for evaluation and Billboard for training.
Task
Input length
(k:, d)
(hg, ha)
Downbeat
6 seconds
(128, 96)
(4, 8)
Chord
12 seconds
(64, 256)
(8, 8)
Key
36 seconds
(128, 32)
(8, 4)
Table 10: SpecTNT parameters used in each task, where k and d denote spectral and temporal feature dimensions; while h2 and ha represent the number of heads for the spectral and temporal Transformer encoders, respectively.
Duu missing audio files for Billboard, redownloaded the missing files from the Internet..
31

---

Under review as a conference paper at ICLR 2023
MIDI Index
MIDI Instrument
MIDI Program
Mapping
Our Index
0-3
Piano
Grand/Brigh//Hokyyttoon Piano
Piano
4-5
Piano
Electric Piano 1-2
Electric Piano
6
Piano
Harpsichord
Harpsichord
Piano
Clavinet
Clavinet
Celesta, Glockenspiel, Music box,
8-15
Chr. Percussion
Vibraphone, Marimba, Xylophone,
Chr. Percussion
4
Tubular Bells, Dulcimer
Drawhar, Percussive,
16-20
Organ
Rock, Church,
Organ
Reed Organ
21
Organ
Accordion
Accordion
22
Organ
Harmonica
Harmonica
23
Organ
Tango Accordion
Accordion
24-25
Guitar
Acoustic Guitar (nylon, steel)
Acoustic Guitar
Electric Guitar (jazz, clean, muted,
26-31
Guitar
Electric Guitar
overdriven, distorted, harmonics)
32-39
Bass
AcousticFFlectnccSlllySyyt Bass
Bass
10
40
Strings
Violin
Violin
41
Strings
Viola
Viola
12
42
Strings
Cello
Cello
13
43
Strings
Contrabass
Contrabass
14
44
Strings
Tremolo Strings
Strings
15
Strings
Pizzicato Strings
Strings
46
Strings
Orchestral Harp
Harp
16
47
Strings
Timpani
Timpani
17
48-51
Ensemble
Acoustic/Synth String Ensemble
Strings
52-54
Ensemble
Aahs/Oohs/Synth Voice
Voice
18
55
Ensemble
Orchestra Hit
Strings
15
Brass
Trumpet
Trumpet
57
Brass
Trombone
Trombone
20
58
Brass
Tuba
Tuba
21
Brass
Muted Trumpet
Trumpet
60
Brass
French Horn
Horn
22
61-63
Brass
Acoustic/Synth Brass
Brass
23
64-67
Reed
Soprano, Alto, Tenor, Baritone Sax
Saxophone
68
Reed
Oboe
Oboe
25
69
Reed
English Hom
Horn
22
Reed
Bassoon
Bassoon
71
Reed
Clarinet
Clarinet
27
72
Pipe
Piccolo
Piccolo
28
73
Pipe
Flute
Flute
74
Pipe
Recorder
Recorder
30
Pan Flute, Blown bottle, Shakuhachi,
75-79
Pipe
Pipe
31
Whistle, Ocarina
80-87
Synth Lead
Lead 1-8
Synth Lead
32
88-95
Synth Pad
Pad 1-8
Synth Pad
33
96- 103
Synth Fffects
1-8
Synth Effects
Sitar, Banjo. Shamisen, Koto, Kalimba. Bagpipe, Fiddle. Shana
104-111
Ethnic
Ethnic
35
Tinkle Bell, Agogo, Steel Drums, Woodblock,
112-119
Percussive
Taiko Drum, Melodic Tom,
Percussive
36
Synth Drum
Guitar Fret Noise, Breath Noise,
120-127
Sound Effects
Seashore, Bird Tweet, Telephone Ring,
Sound Effects
Helicopter, Applause, Gunshot
128
Drums
Drums
Drums
38
Table 11: The instrument mapping used in our experiments. Our mapping is less detailed than the MIDI Program Number, but is finer than the MIDI Instrument code, thus resulting in 39 different instruments.
32


Reviews:
[{"review_id": "uHgD5gG0x1l", "reviewer": "Reviewer_ZUz4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an end-to-end method so-called ``Jointist'' to simultaneously address 3 audio tasks consisting respectively\nin separating the source signals, transcribing the music and recognizing the instruments.\n\nThe proposed method is a deep neural network which combines together 3 existing neural network architectures which are\nrespectively [Won et al. 2021] for instrument recognition, [Hawthorne et al. 2017] for the transcription and\n[Jansson et al. 2017] for the source separation.", "summary_of_the_review": "The main contribution of the paper is the combination of 3 existing deep learning-based methods into a unique end-to-end system for transcription / instrument recognition and separation.\n\nDespite it could be interesting from a practical point of view to investigate how instrument recognition can help to improve transcription  and separation, the authors failed to convince a reader of the interest of their work for the reasons listed below:\n\n1)from a practical point of view, the experiments are insufficient and don't always compare the proposed framework with the relevant state of the art method for each distinct task (eg. source separation results only present the method of the author(s)). Moreover, there exist several other research datasets for music transcription, instrument recognition and source separation which should also be investigated (at least one other reference dataset for each addressed task with the corresponding state-of-the-art method instead of using the same slakh2100 dataset). \n\n2) Numerical results are limited and not sufficiently detailed. For example, Recall and Precision and confusion matrices are common metrics that could be used for the Instrument Recognition. There also exist other objective metrics for source separation such as SIR and SAR (cf. BssEval) but also perceptual-relevant metrics such as PEASS which could be used to present the separation results.\n\n3)Important details about the proposed method are missing. It is not clear how the 3 modules can be simultaneously trained. The fig. 1 is insufficient to describe the overall method and don't explain how the 3 networks  are connected together.\n\nTo conclude, I think that this paper needs an overall reorganization to be consistent and self-content while highlighting the original contribution from the author(s). \nEach experiment should be conduced more rigorously and should provides arguments about the relevance of combining the 3 systems in a unique end-to-end framework with a fair comparison with the state of the art of each distinct task.\nThe ideas of combining separation with transcription, and instrument recognition with transcription are not novel. Hence the authors \nshould further explain in what they believe that this work is original and preferable than using 3 distinct dedicated methods.\n", "strength_and_weaknesses": "Strength:\n-Reproducible results with public code and audio samples\n\nWeaknesses:\n\n-Lack of novelty (reuse 3 existing systems trained separately on the 3 addressed tasks)\n-Details of the incremental contributions and methods are not sufficient\n-insufficient validation (based on only a unique synthesized dataset Slakh2100)\n-The interest of combining simultaneously 3 systems appears to be limited according to results", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "sL4ESuijX_1", "reviewer": "Reviewer_ZLUi", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a joint training framework of three components - instrument recognition, transcription and source separation. The instrument recognition module is optional and can be replaced by human inputs. The joint training of the transcription and source separation module was shown to be beneficial. The author tried to work on very challenging data with many instruments playing at the same time such as pop music. Various experiments show that the proposed method is promising for several downstream tasks.", "summary_of_the_review": "Overall, the paper explored the interesting path of joint training of music transcription and source separation. Some experiments are informative. However, the technical novelty less significant compared to the empirical novelty. More in-depth analysis of the model behavior or optimization of model architecture that may be related to the task or the joint training technique would be more interesting to the community.", "strength_and_weaknesses": "Strengths:\n* The paper is generally well written and easy to follow\n* Although many details are omitted in the main text, significant number of details such as model setup are given in the appendix\n* The background and related work are well covered\n* Although the idea of joint training of transcription & separation is not new for audio processing, authors made great attempt for very challenging multi-instrument music task\n\nWeakness\n* Authors could have referred to some recent work about joint speech recognition & speech separation for multi-speaker overlapped speech\n* Although authors claim the proposed model reaches the state-of-the-art, the quality of separation is still poor in many cases (in the provided samples). I suspect further improvement can be obtained if authors could optimize the model architecture / training more carefully. For example, Table 1 shows deeper Transformer has the potential to perform better. There are many convolution / self-attention architectures, especially in computer vision and speech area where the authors could try to adopt\n* The comparison of the proposed model and MT3 may not be strictly fair as the model size is significantly larger, the training data is different, etc.\n* The quality of some plots in the appendix can be improved", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "n0ZdGxyhzmW", "reviewer": "Reviewer_YdWq", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a novel approach to MIR tasks which uses a modular setup combining instrument recognition, transcription, and source separation models into a joint pipeline. The authors perform experiments to evaluate the proposed approach, but the results are mixed (with the only improvement being in a listening study, and quantitative metrics otherwise lower than baselines when they are compared to). Joint training of the modules hurts (rather than helps) performance. Evaluations are focused on what the authors call \"real-world popular music\" and are conducted on the Cerberus dataset and a set of songs extracted from other pop databases (the model only shows improvements on the latter).", "summary_of_the_review": "Overall, the paper presents a novel take on an existing idea (training joint transcription + source separation ideas). The Jointist approach is more flexible than previously-proposed models for this task, as it can transcribe arbitrary sets of 128 MIDI instruments plus drums. However, the results seem mixed (the only clear transcription improvement is a small gain on a subjective listening study) some of the choices do not seem particularly motivated, and the characterization and use of existing models (such as MT3) seems unfair (see below) with other experiments (source separation, beat tracking, etc.) apparently missing baselines entirely. The paper also needs considerable revision; in particular, some important information is in the supplement that should be in the main text.", "strength_and_weaknesses": "\n# Major comments\n\n* The motivation for the current work in the introduction does not, as I understand it, actually motivate the current work. The authors say that \"the number of instruments in a pop song may vary....it is limiting to have a model that transcribed a pre-defined fixed number of musical instruments\". However, MT3 also does not transcribe a fixed number of instruments, and transcribes the same set of outputs as the current model. I think a better motivation for this work is (1) the general trend of AI/ML models toward \"foundation\" models trained simultaneously to perform multiple tasks with shared parameters, and (2) a focus on pop music. The latter, in particular, seems to be a motivating factor for the current work, since the evaluation *only* focuses on such music, while the baselines (Omnizart, MT3) can transcribe other common domains as well (e.g. classical) which are absent from the evaluations.\n\n* In several places, the characterization of Jointist vs. MT3, the current state of the art in transcription, seems unfair. For example, the abstract claims that Jointist \"achieves state-of-the-art performance...outperforming ...MT3\". However, Table 2 shows a clear advantage of MT3; the only improvement over MT3 is in the subjective study (see comments below). The authors also claim that \"jointist is more flexible than MT3 in which human users can pick the target instruments that they are interested in\"; however, one could also do this with MT3 outputs by simply retaining only the desired instruments. The latter point, in particular, is important: the ground-truth (or \"human-provided)\") labels are used for Jointist in many experiments (i.e. many of the entries in Table 2) and lead to the best performance; in contrast, such information could be, but is not, applied to the outputs of MT3 -- which would be a more fair comparison. This form of conditioning (essentially filtering of MT3 outputs for known instruments) would also remove the \"NA\" values for MT3 in Table 2, which the authors state are due to extraneous predicted instruments in the output (something avoided by their models by design in the cases where ground-truth instrument labels are provided).\n\n* Too much material is relegated to the supplement that belongs in the main text. For example, many important architectural details and experimental results (Figs 2-4, the \"new instrument-wise metric\" in 4.3.1, experimental descriptions for section 6.1).\n\n* The main improvement of the model seems to be in the human listening study. This is concerning for a few reasons. (1) The listening study results are at odds with the quantitative results in Table 2, where it seems MT3 outperforms all versions of Jointist. If the claim is that MT3 does not generalize well to new datasets, this could be verified with a similar listening study on the Slakh test set. (2) The study itself is not well-described: What are the mean and SD of the subjective scores, and how did these vary by piece and dataset? Who are the participants in the human evaluation (musical experts, non-experts)? What was the motivation for providing only MIDI files and allowing participants to render it themselves, instead of providing a pre-rendered version to control for their DAW, etc.\n\n* The non-transcription experiments (source separation, beat tracking, downbeat/chord/key) do not seem to compare to any baselines (at least not in main text), even though baselines exist for all of these tasks (e.g. the authors cite several for source separation; for beat tracking SOTA might be the \"madmom\" model).\n\n* Why is only note-offset, not note-onset-offset, F1 used in the transcription experiments?\n\n# Minor comments\n\n* The paper refers to such categories as \"real-world popular music\", but the Slakh dataset is probably not a globally-representative sample of popular music. It would be better to refer to \"Western popular music\" or similar (as is evidenced by a single category of \"ethnic\" in Table 11, which encompasses several globally-popular and distinct instruments). \n\n* Is there a reason the arrow from f_T -> f_MSS goes into the \"side\" of f_MSS in Figure 1 (not where the rest of the inputs are)?\n\n* The paper should also cite, and possibly compare to, the preceding work \"Unaligned Supervision For Automatic Music Transcription in The Wild\" (2022) and \"A Lightweight Instrument-Agnostic Model for Polyphonic Note Transcription and Multipitch Estimation\" (2022).\n\n* P3: Please describe the denoising procedure adapted from Kong et al.\n\n* The description of MIDI mappings in 4.1 is not clear. (1) Are all MIDI numbers 0-3 mapped to a single MIDI number (i.e. electric piano, etc. mapped to piano)? If so, how is this accounted for in the baselines? (2) How are different drum hits mapped to MIDI pitches in the model?\n\n* Do you plan to open-source trained models and code associated with this work?\n\n* Please describe the weighted F1 metric; I could not find a description of how it was weighted in the paper.\n\n* \"the language model in MT3 dominates the acoustic model\". I believe there is only one model in MT3 (an encoder-decoder). Please clarify.\n\n* \"false positive piano rolls and false negbative piano rolls have undefined F1 scores\" - please clarify. How is an entire piano roll FP or FN?\n\n* What does the location of I_cond^(i) in Figure 4 signify? That it is input to both the encoder and decoder of U-net? Please clarify.\n\n* Why not compare to a version that uses a frozen MT3 as the transcription model f_T, since the models are not trained together anyway?\n\n* Table 1 suggests continued and monotonic improvements with depth; why not increase the number of layers beyond 4?\n\n# Typos etc.\n\nThere are many typos in the paper; I list a few here but would suggest a thorough revision for clarity.\n\nP3: \"onsets-and-frames\" -> onsets and frames\n\nP3: \"teacher-forced training is used during training\"\n\nP4: \"differnt\"\n\nP4: \"tracks per piece\" -> instruments per piece (?)\n\nP9 \"in par with\" -> on par with\n\nP15 \"instruments conditions\" -> instrument conditions\n\nP15 \"full-fledge\"", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Ls0FyfTsMV", "reviewer": "Reviewer_jSBB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper describes a jointly-trained triplet of models for musical instrument recognition, music transcription, and music source separation, able to handle a large number of instruments.  Experiments demonstrate that training these models jointly results in better performance for each task than training each model in isolation, and the models (the transcription model specifically) is useful for several downstream tasks.  I am not as familiar with the source separation literature, but I believe another contribution of the paper is a source separation model that can work on a much wider variety of instruments.", "summary_of_the_review": "The paper doesn't have any especially surprising findings and could explore its main idea more thoroughly (and remove some non-essential sections), but I recommend acceptance nonetheless as it appears methodologically sound and is a valid research contribution to music transcription and source separation.", "strength_and_weaknesses": "Strengths\n-----------\nThere is one main strength which is that the overarching goal of the paper is solid: solve transcription and source separation jointly for music with an arbitrary number of instruments.  And the system proposed in the paper does appear to be a step forward in solving both problems, as demonstrated by the many experiments in the paper.\n \nWeaknesses\n--------------\n1) It's not especially surprising that combining transcription and source separation would lead to improved performance on both tasks.  However, there are multiple ways one could imagine combining these tasks.  In this paper, transcription seems to be treated as \"primary\", as the output of the transcription module is fed into the source separation module.  One could just as well imagine the converse, where sources are separated prior to transcription.  Would that setup work better?  Or could there be a third setup that learns a representation useful for both tasks?  These questions are neither asked nor answered by the paper.\n\n2) In terms of the objective metrics, the transcription model in this paper performs *considerably* worse than the MT3 model of Gardner et al.  Since the purpose of music transcription is usually not direct listening, the fact that listeners prefer the Jointist transcriptions is interesting but not especially compelling.  Since you are already performing some experiments with the transcriptions on downstream tasks, it would be helpful to demonstrate that the transcriptions from Jointist also outperform MT3's transcriptions on these tasks, if that is in fact the case.\n\n3) Related to the above, I would be interested in reading an entire paper comparing downbeat/chord/key estimation from audio vs. pianoroll.  But the fact that giving a model access to both pianoroll and audio outperforms just audio is not surprising, nor is the performance improvement large enough to warrant the inclusion of the experiment in *this* paper.  Downbeat/chord/key estimation is an application of *any* transcription model!\n\n4) The introduction suggests a lack of symbolic data for training generative pop music models as a motivation for music transcription.  However, the very dataset used in this paper (Slakh) is derived from a much larger dataset of symbolic music (Lakh MIDI Dataset), much of which is pop.  And there are existing generative models (e.g. MuseNet) trained on such datasets.  I would argue that what's lacking is symbolic data *aligned to real recordings*.\n\n5) Minor issues:\n    * Throughout the paper the term \"ablation\" is used to refer to experiments with multiple conditions where none is obviously an ablation; \"ablation\" refers to *removal* of components.\n    * Some typos: \"differnt\" in Section 3.3, \"black\" instead of \"block\" in Appendix A, \"marco\" instead of \"macro\" in Appendix B.1.\n    * Figures 6 and 7 would be much more readable if axis labels used linear label counts even if the dimension is log scale.\n    * More of a question: in Appendix E, why do some pairs of instruments that seem similar (e.g. recorder and flute) have vastly different transcription performance?  It might be helpful to show something like a confusion matrix here (even though the values aren't exactly classification counts), because I suspect there are pairs of instruments that are frequently mistaken for one another.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject