MAJL: A Model-Agnostic Joint Learning Framework for Music Source Separation and Pitch Estimation

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-04 12:29:38

Abstract: Music source separation and pitch estimation are two vital tasks in music information retrieval. Typically, the input of pitch estimation is obtained from the output of music source separation. Therefore, existing methods have tried to perform these two tasks simultaneously, so as to leverage the mutually beneficial relationship between both tasks. However, these methods still face two critical challenges that limit the improvement of both tasks: the lack of labeled data and joint learning optimization.
To address these challenges, we propose a Model-Agnostic Joint Learning (MAJL) framework for both tasks.
MAJL is a generic framework and can use variant models for each task. It includes a two-stage training method and a dynamic weighting method named Dynamic Weights on Hard Samples (DWHS), which addresses the lack of labeled data and joint learning optimization, respectively. Experimental results on public music datasets show that MAJL outperforms state-of-the-art methods on both tasks, with significant improvements of 0.92 in Signal-to-Distortion Ratio (SDR) for music source separation and 2.71% in Raw Pitch Accuracy (RPA) for pitch estimation. Furthermore, comprehensive studies not only validate the effectiveness of each component of MAJL, but also indicate the great generality of MAJL in adapting to different model architectures.

Main paper: 
 introduction: 
 The digital music industry has been growing rapidly in the last few years due to the mass publication of music through smartphone apps, enabling hundreds of millions of people to access a song via large music platforms. This has created huge music streaming companies such as Spotify (worth $37B) and QQ Music (worth $10B). The digital music industry in the US has a market of close to $10B in 2022 and has been growing at more than 10% in each of the last five years [39] and that in China has a market of $5B and has been growing at 30∼50% in each of the last five years [26] .Music Information Retrieval (MIR) is a pivotal research domain that supports the functionality of large music platforms. Within MIR, music source separation (MSS) and pitch estimation (PE) emerge as critical tasks with far-reaching implications. MSS facilitates several downstream tasks, such as lyrics extraction [15] and music transcription [2] , accentuating its pivotal role. PE has great significance for various MIR applications, including contentbased music recommendation and query/search by singing [60] . Notably, real-world musical compositions are often mixture music, typically as .wav or .mp3 files, which do not inherently provide the pitch information of music data. To extract target sources along with their corresponding pitches, simultaneous execution of both MSS and PE tasks becomes imperative.The MSS task entails generating isolated stems for vocals, bass, or drums from raw audio or spectrograms of mixture music, as shown in the top row of Figure 1(a) . The PE task involves extracting fundamental frequencies (𝑓 0) from clean music audio or spectrograms, as shown in the second row of Figure 1(a) . Following previous studies [18, 32, 61] , we default to using pitch estimation to refer to single pitch estimation unless explicitly stated otherwise in this paper. It is important to note that the clean music in the PE task corresponds to the predicted source in the MSS task because both are music data from a specific instrument. Music source separation and pitch estimation are closely related in music information retrieval, where the input of pitch estimation is typically obtained from the output of music source separation, as shown in Figure 1(a) . Consequently, various studies have aimed to simultaneously tackle these two tasks, leveraging on their mutually beneficial relationship. One line of studies (e.g., DNN+UPDUDP [15] and HR-ED [16] ) uses pipeline methods, as shown in Figure 1(a) . However, these methods train MSS and PE models independently on different music datasets, leading to a mismatch between the data distributions at training and testing time. This mismatch limits the improvement of pitch estimation from mixture music. Another line of studies (e.g., HWJH [23] , HS-W 𝑝 [44] , and S→P→S→P [28] ) employs joint learning methods, as shown in Figure 1(b) . Although these joint models combine MSS and PE tasks by summing the respective losses, the distinct objectives of each task pose a challenge to achieve simultaneous improvements. Moreover, these models are often designed for specific tasks, lacking scalability, thus limiting their ability to improve performance even when better MSS or PE models become available. Although the above studies have tried to perform MSS and PE tasks simultaneously, there are still two major challenges have to be addressed as described below.Challenge 1: Lack of labeled data. The field of music information retrieval suffers from a large scarcity of annotated datasets, particularly due to the laborious and professional nature of obtaining target sources and corresponding pitches, even for experienced musicians. As a result, only a small amount of music datasets (e.g., MIR-1K [22] and MedleyDB1 [4] ), offer both target sources and corresponding pitches. But the total duration of MIR-1K is only 2.25 h, and the total duration of MedleyDB is 3.21 h. We call such kind of dataset as fully-labeled dataset. In contrast, datasets specific to MSS or PE are much larger than the fully-labeled dataset. For example, MUSDB18 [50] is a music source separation dataset and the total duration of MUSDB18 is about 5 times as that of MIR-1K. While MIR_ST500 [58] is a dataset for pitch estimation from mixture music and the total duration of MIR_ST500 is about 14 times as that of MIR-1K. We call such kind of datasets as single-labeled dataset. The small amount of fully-labeled datasets prevent models from effectively learning the relationship between MSS and PE tasks.Challenge 2: Joint learning optimization. Existing joint learning methods (e.g., HS-W 𝑝 [44] and S→P→S→P [28] ) design joint models and perform joint learning by simply summing up the losses of both tasks. However, these methods do not address the following problems: (i) Error propagation. Due to the cascade relationship between MSS and PE, poor predictions of music source separation can lead to poor pitch estimation results. This error propagation problem is critical in joint learning of both tasks. (ii) Misalignment between different objectives. The objectives of MSS and PE differ from each other. As a result, simply adding the losses of these two tasks cannot guarantee simultaneous improvements in MSS and PE. We believe that existing joint learning methods failed to align different objectives because they treated all samples equally. These two problems make the joint learning of MSS and PE challenging.To address these challenges, we propose a Model-Agnostic Joint Learning (MAJL) framework for music source separation and pitch estimation. Our framework can adopt existing MSS and PE models and improve the performance of both tasks when better models become available. MAJL contains a two-stage training method and the Dynamic Weights on Hard Samples (DWHS), which is designed to address Challenge 1 and Challenge 2, respectively.To address Challenge 1, we design a two-stage training method to leverage large single-labeled datasets. This method comprises an initialization stage (Stage I) and a semi-supervised training stage (Stage II). In Stage I, the model is trained using the available fullylabeled data, then utilizing this trained model to generate pseudo labels and corresponding confidence values for the single-labeled data. The confidence value reflects the quality of pseudo labels. In Stage II, we retrain the model from scratch, using the music data that consists of fully-labeled data, single-labeled data, and pseudo labels generated during Stage I. Additionally, a threshold-based filter is applied to exclude low-confidence single-labeled data, ensuring the quality of pseudo labels. This two-stage training method effectively leverages extensive single-labeled datasets and high-quality pseudo labels, effectively addressing the lack of labeled data.For Challenge 2, we design a dynamic weighting method called Dynamic Weights on Hard Samples (DWHS), to solve the problems of error propagation and misalignment between different objectives in joint learning. Addressing the error propagation problem entails identifying the module making poor predictions in our framework. Furthermore, to solve the problem of misalignment, we need to identify hard samples for both the MSS and PE tasks. The DWHS entails extracting pitch results from target and predicted sources, termed 𝑡𝑎𝑟𝑔𝑒𝑡_𝑠𝑜𝑢𝑟𝑐𝑒2𝑃𝑖𝑡𝑐ℎ and 𝑝𝑟𝑒𝑑𝑖𝑐𝑡𝑒𝑑_𝑠𝑜𝑢𝑟𝑐𝑒2𝑃𝑖𝑡𝑐ℎ, respectively. By comparing these pitch results, we not only identify modules producing poor predictions but also identify hard samples for both tasks. This allows us to allocate appropriate weights to hard samples across tasks, thereby mitigating error propagation and aligning different objectives within joint learning.In this paper, our contributions are summarized as follows:• We propose a Model-Agnostic Joint Learning (MAJL) framework for music source separation and pitch estimation. MAJL is a generic framework, which can further improve the performance of both tasks when better MSS or PE models are available. Moreover, our framework outperforms previous methods, leading to significant improvements in the performance of both tasks. • We design a two-stage training method to solve the lack of labeled data. The two-stage training method combines music source separation and pitch estimation tasks at the data aspect. By leveraging large single-labeled datasets, our method extends fully-labeled data, allowing the model to better learn the relationship between both tasks. • To address the challenge of joint learning optimization, we design a novel dynamic weighting method, named Dynamic Weights on Hard Samples (DWHS). The DWHS can handle error propagation and misalignment between different objectives by identifying hard samples and setting appropriate weights for these samples. 
related work: 
 Music Source Separation (MSS) is a crucial task in MIR, aiming to isolate individual sources from mixture music. Many deep learning methods have been proposed for MSS, generally categorized into common models and side-information informed models.Common models operate solely on hidden features extracted from the time or frequency domains. For example, Spleeter [21] , U-Net [29] , CWS-PResUNet [40] and ResUNetDecouple+ [36] predict target sources using frequency domain features. In contrast, Demucs [10] , Wave-U-Net [57] and its follow-ups [41, 47] leverage time domain features. Other methods such as KUIELAB-MDX-Net [33] , Hybrid Demucs [9] and HT Demucs [53] fuse both domain features to enhance the performance of MSS. In contrast, the side information informed models use additional information, such as lyrics, pitches, or spatial information to improve MSS. For example, JOINT3 [54] employs phoneme-level lyrics alignment, while Soundprism [12, 14] and SPAIN-NET [48] leverage pitches and spatial information, respectively. These methods exclusively address the MSS task, which do not support simultaneous PE task. Pitch Estimation (PE) is a fundamental task in MIR, aimed at extracting fundamental frequency (𝑓 0). PE can be broadly classified into two sub-tasks: PE from clean music and PE from mixture music.For clean music, existing methods encompass heuristic-based and data-driven methods. Heuristic-based methods like ACF [13] , YIN [8] , SWIPE [5] , and pYIN [42] leverage candidate-generating functions to predict pitches. Conversely, data-driven methods, including CREPE [32] , DeepF0 [56] , and HARMOF0 [61] , rely on supervised training of models for PE. While these methods achieve accurate pitch results from clean music, their performance is constrained when applied to mixture music due to the presence of other existing sources. For mixture music, existing methods comprise pipeline and end-to-end methods. Pipeline methods involve utilizing MSS models (e.g., Spleeter [21] and U-Net [29] ) to extract target sources from the mixture music, and then using PE models to predict corresponding pitches. However, a mismatch between the data distributions at training and testing times often limits the performance of PE from mixture music. End-to-end methods (e.g., DSM-HCQT [3] , CNN-Raw [11] , and JDC [37] ) are designed to directly predict pitches from mixture music. Nevertheless, these methods encounter performance limitations as a result of the presence of other sources in mixture music. 
problem formulation: 
 In this section, we formulate the problems of Music Source Separation (MSS) and Pitch Estimation (PE), along with a previously proposed naive joint learning method referred to as the Joint Cascade Framework (JCF) in this paper.Music Source Separation (MSS). The task of MSS is to extract a target source from mixture music signals. The mixture music signals can be represented as either the raw audio waveform 𝑥 or its corresponding spectrogram 𝑋 𝑇 ×𝐹 , where 𝑇 is the number of audio frames and 𝐹 is the number of frequency bins. It should be noted that the spectrogram is computed using the short-time Fourier transform (STFT) as a feature representation of the original music signal. The output of MSS is the target source 𝑠, and the spectrogram of target source is represented as 𝑆 𝑇 ×𝐹 . Thus, the MSS task can be formulated as F 𝑚𝑠𝑠 : 𝑥 (𝑋 𝑇 ×𝐹 ) → 𝑠.Pitch Estimation (PE). The task of PE aims to estimate the pitch sequence of clean music from a raw audio waveform or its spectrogram representation. Following previous studies (e.g., 360 in CREPE [32] and 352 in HARMOF0 [61] ), each pitch is typically represented as a 𝑁 -dimensional one-hot vector 𝑦. As a result, the output of PE is a sequence of pitch vectors 𝑌 𝑇 ×𝑁 , where 𝑁 is the number of pitch values. Furthermore, the input of PE is typically obtained from the output of MSS. Thus, the PE task can be formulated asF 𝑝𝑒 : 𝑠 (𝑆 𝑇 ×𝐹 ) → 𝑌 𝑇 ×𝑁 .Joint Cascade Framework (JCF). The JCF is designed to leverage the cascade relationship between MSS and PE, enabling joint learning of both tasks. It (cf. Figure 3 ) comprises a Music Source Separation Module (MSS Module) and a Pitch Estimation Module (PE Module). Firstly, the features of mixture music are input into the MSS Module to obtain predicted sources. Then, the PE Module extracts corresponding pitches from the predicted sources.In line with previous studies [29, 32, 36, 59, 61] , the training of JCF involves using the Mean Absolute Error (MAE) loss for MSS and the Binary Cross Entropy (BCE) loss for PE, respectively. Therefore, the loss function for MSS is defined as:EQUATION)where 𝑠 is the target sources, ŝ is the predicted sources and 𝐿 is the length of mixture music. And the loss function for PE is defined as:EQUATION)where 𝑁 is the number of pitch values, 𝑦 is the ground truth of pitch results and ŷ is the predicted pitch value. Thus, the total loss for naive joint learning of MSS and PE is:EQUATIONThe JCF is unable to solve the two challenges mentioned in Section 1. Therefore, we propose our Model-Agnostic Joint Learning (MAJL) framework for both tasks, building upon and extending the JCF. 
method: 
 As shown in Figure 2 , the Model-Agnostic Joint Learning (MAJL) framework contains two important components: two-stage training method and Dynamic Weights on Hard Samples (DWHS). Besides, the music source separation module (MSS Module) and pitch estimation module (PE Module) within MAJL can be easily replaced with existing MSS and PE models. Then the trained framework is employed to generate pseudo labels for target sources or corresponding pitches. Additionally, we compute confidence values for each pitch result and each frame of target sources. For predicted pitches, a pitch value is considered present when 𝑚𝑎𝑥 ( ŷ) ≥ 0.5; otherwise, a pitch value is considered absent. Then the confidence value (confi) is defined as:EQUATIONIt should be noted that this confidence value can also be applied to predicted sources due to the inherent cascade relationship between music source separation and pitch estimation.To maintain a consistent format for confidence values across different dataset types (fully-labeled and single-labeled datasets), we set the confidence values of true labels to 1. Therefore, for fullylabeled datasets (e.g., MIR-1K and MedleyDB), the confidence values of MSS (confi 𝑚𝑠𝑠 ) and PE (confi 𝑝𝑒 ) are defined as:confi 𝑚𝑠𝑠 = 1 confi 𝑝𝑒 = 1 (5)Then, for MSS datasets (e.g., MUSDB18), which belongs to singlelabeled datasets, the confidence values of MSS (confi 𝑚𝑠𝑠 ) and PE (confi 𝑝𝑒 ) are defined as:EQUATIONWhile for PE datasets (e.g., MIR_ST500), which also belongs to single-labeled datasets, the confidence values of MSS (confi 𝑚𝑠𝑠 ) and PE (confi 𝑝𝑒 ) are defined as:confi 𝑚𝑠𝑠 = confi confi 𝑝𝑒 = 1 (7)Semi-supervised Training Stage (Stage II): After the initialization stage, we obtain pseudo labels and confidence values from single-labeled datasets. We then combine fully-labeled music data, single-labeled music data, and pseudo-labels to create a synthetic dataset. To filter pseudo-labels from single-labeled music data, we set a threshold (𝑡ℎ). The detailed filtering process involves applying weights for MSS and PE in the loss computation. Subsequently, we retrain our framework from scratch using the synthetic dataset. Thus, the loss function of stage II is written as:EQUATIONHere, confi 𝑚𝑠𝑠 equals 1 if confi 𝑚𝑠𝑠 ≥ 𝑡ℎ, and 0 otherwise. confi 𝑝𝑒 is calculated using the same way as confi 𝑚𝑠𝑠 .4.2.1 Analysis of Different Cases in DWHS. The naive joint learning method can not ensure simultaneous improvements in both tasks due to the problems of error propagation and misalignment between distinct objectives. To address these problems concurrently, we should identify hard samples and assign appropriate weights to each sample in both tasks. This can be achieved by comparing the predicted pitches from target sources with those from predicted sources (c.f. Figure 3 ). By this comparison, we can determine which module is delivering poor predictions and identify samples that are hard for either MSS or PE. A comprehensive analysis of different cases arising from this comparison is summarized in Table 1 . Detailed explanations of each case in Table 1 are provided as follows.For Case 1, both the predicted pitches from predicted sources and those from target sources are correct. This result indicates that there is no issue with the MSS Module, the PE Module or the quality of data. For Case 2, the predicted pitches from predicted sources are correct, while those from target sources are incorrect. This result indicates the presence of noisy pitch labels in the data. To mitigate the impact of noisy labels, the weights of such samples for PE should be within the range of 0 to 1. For Case 3, the predicted pitches from predicted sources are incorrect, while those from target sources are correct. This result indicates that the predicted sources are quite different from the original target sources, making the data hard for the MSS. To emphasize learning on hard samples, the weights assigned to these samples in the MSS should be greater than 1. For Case 4, both the predicted pitches from target sources and those from predicted sources are incorrect. This result highlights poor predictions by the PE Module, indicating that the music data is hard for the PE. To emphasize learning on hard samples, the weights assigned to these samples in the PE should be greater than 1.. By leveraging the above analysis, we can assign different weights to each sample based on identified cases, thereby aligning the focus of two tasks during joint learning. The most direct method involves setting different weights for different cases as outlined in Table 1 . However, this approach incurs high training costs due to the difficulty of manually determining proper weights for each case. Therefore, we introduce the DWHS, which automatically extracts the appropriate weights for different cases.The DWHS utilizes a simple network called the Dynamic Weight Module (DWM) to determine dynamic weights for the MSS and PE tasks under different cases. As illustrated in Figure 4 , the inputs for the DWM consist of 𝑝𝑟𝑒𝑑𝑖𝑐𝑡𝑒𝑑_𝑠𝑜𝑢𝑟𝑐𝑒2𝑃𝑖𝑡𝑐ℎ, 𝑃𝑖𝑡𝑐ℎ𝑒𝑠, and 𝑡𝑎𝑟𝑔𝑒𝑡_𝑠𝑜𝑢𝑟𝑐𝑒2𝑃𝑖𝑡𝑐ℎ from Figure 3 , maintaining the same format as Table 1 : Analysis for different cases in DWHS and the weights that should be set for different cases by the DWHS.Analysis The Weights Should be Assigned MSS Module PE Module Data𝜔 𝑚𝑠𝑠 𝜔 𝑝𝑒 1 Correct Correct ✓ ✓ ✓ 𝜔 𝑚𝑠𝑠 = 1 𝜔 𝑝𝑒 = 1 2 Correct Incorrect ✓ ✓ × 𝜔 𝑚𝑠𝑠 = 1 0 ≤ 𝜔 𝑝𝑒 < 1 3 Incorrect Correct × ✓ ✓ 𝜔 𝑚𝑠𝑠 ≥ 1 𝜔 𝑝𝑒 = 1 4 Incorrect Incorrect ✓ × ✓ 𝜔 𝑚𝑠𝑠 = 1 𝜔 𝑝𝑒 ≥ 1𝑌 𝑇 ×𝑁 . These inputs are concatenated and passed through two CNN layers with ReLU activation, configured as depicted in Figure 4 with 3 × 3 kernels. Following the CNN layers, there is a flatten layer, a fully connected layer with ReLU activation, and finally, a fully connected layer with sigmoid activation that generates dynamic weights for both tasks. This process enables dynamic assignment of weights without the need for manual specification of specific weights. Specifically, the weights (𝜔 𝑚𝑠𝑠 and 𝜔 𝑝𝑒 ) corresponding to different cases outlined in Table 1 are automatically extracted by the Dynamic Weight Module (DWM) of the DWHS. To ensure the weights extracted by DWM for different cases align with the specified weights shown in Table 1 , we design the loss function for the DWM of DWHS to address four specific cases. For Case 1, the Mean Absolute Error (MAE) loss is employed to ensure the predicted weights are around 1. Then the loss function for DWM of DWHS in Case 1 is defined as:EQUATIONFor Case 2, the MAE loss is utilized to ensure the predicted weights of MSS are around 1. For the predicted weights of PE, the Bayesian Personalized Ranking (BPR) loss [52] is employed to ensure lower weights for noisy music data. Then the loss function for DWM of DWHS in Case 2 is defined as:EQUATIONFor Case 3, the BPR loss is used to ensure hard samples for MSS receive higher weights. Simultaneously, the MAE loss is used to ensure the predicted weights of PE are around 1. Then the loss function for DWM of DWHS in Case 3 is defined as:EQUATIONFor Case 4, the MAE loss is employed to ensure the predicted weights of MSS are around 1. Additionally, the BPR loss is used to ensure hard samples for PE receive higher weights. Then the loss function for DWM of DWHS in Case 4 is defined as:EQUATIONThus, the total loss of DWHS is written as:EQUATIONIt is important to note that if there is no music data belonging to a specific case, the loss function for that case is automatically set to 0. For example, if there are no music data belonging to Case 1, then the L 𝑑𝑤ℎ𝑠_1 becomes 0. Thus, with the DWHS, the loss function of stage I is written as:EQUATIONAnd the loss function of stage II is written as:EQUATION)where confi 𝑚𝑠𝑠 and confi 𝑝𝑒 are the same as those in Eq. 8.The above two-stage training method and Dynamic Weights on Hard Samples are two important components of MAJL aimed at addressing the lack of labeled data (Challenge 1) and the joint learning of optimization (Challenge 2), respectively. The effectiveness of MAJL and each component is evaluated in Section 6. 
experimental setup: 
 Datasets. We evaluate our framework using four public datasets: MIR-1K [22] , MedleyDB [4] , MIR_ST500 [58] , and MUSDB18 [50] . MIR-1K and MedleyDB provide both mixture and clean vocal tracks, along with pitch labels for vocal parts, making them fully-labeled datasets. In contrast, MIR_ST500 and MUSDB18 provide PE and MSS labels, respectively, classifying them as single-labeled datasets. It should be noted that all pitch labels are transformed into frequency bins represented in Hz format the same as MIR-1K. Evaluation Metrics. Following previous studies, we use four metrics to evaluate our framework. For MSS, we use Signal-to-Distortion Ratio (SDR) [36] , Global Normalized Signal-to-Distortion Ratio (GNSDR) [22] to evaluate the quality of predicted sources. A higher SDR or a higher GNSDR indicates better separation results, and vice versa. For PE, we use Raw Pitch Accuracy (RPA) and Raw Chroma Accuracy (RCA) [32] to evaluate the accuracy of predicted pitches. Implementation Details. The raw audio is sampled at 16kHz and then transformed into spectrograms using the short-time Fourier transform (STFT) with a Hann window size of 2048 and a hop length of 320 (20ms). During the training of MAJL, we use a batch size of 16 and the Adam optimizer [34] . The learning rate is initialized to 0.001 and then reduced by 0.98 of the previous learning rate every 10 epochs. For MIR-1K [22] and MedleyDB [4] datasets, we randomly split these datasets into training (80%) and testing (20%) sets. The splitting way for MIR_ST500 and MUSDB18 datasets is introduced in [58] and [50] , respectively. During experiments, we only consider the target source of vocals due to the lack of fully-labeled data from other sources such as bass and drums. 
experimental results: 
 In this section, we present the experimental results of our framework to show the superiority of MAJL. We firstly compare MAJL with baselines on different datasets in Section 6.1. Then we explore the generality of MAJL through an investigation of its various modules in Section 6.2. Following these experiments, we visualize and analyze the weights of DWHS to understand the effectiveness of DWHS in Section 6.3. Finally, a study is conducted to investigate the threshold (𝑡ℎ) used in the two-stage training method in Section 6.4.In this experiment, we conduct a comprehensive comparison of MAJL with several baselines, encompassing End-to-End methods, pipeline methods, and joint learning methods. We evaluate the performance of MAJL on both fully-labeled datasets and singlelabeled datasets. These experimental results not only demonstrate the effectiveness of MAJL in joint learning of both tasks, but also highlight its superiority in either the MSS task or the PE task.To show the superiority of our framework, we perform a comparison with several baselines. For the MSS and PE modules, we choose ResUNetDecouple+ and CREPE, respectively, since they achieve the best performance among all other MSS and PE modules, as shown in Table 5 .Our framework shows the effectiveness in both tasks, achieving state-of-the-art performance for both tasks, as summarized in Table 2. (i): Compared to the End-to-End methods, our framework tackles both tasks simultaneously, leading to significant improvement in the PE task. (ii): Moreover, our framework outperforms existing joint learning and pipeline methods. Specifically, MAJL-Stage I outperforms the previous best model (Pipeline method with ResUNetDecouple+ and CREPE) by 0.27 in SDR and 1.77% in RPA on the MIR-1K dataset. These results demonstrate that our DWHS is effective in enhancing the performance of both tasks. (iii): Furthermore, by incorporating additional single-labeled music data, the performance of both tasks is further improved. For example, MAJL demonstrates a 0.65 improvement in SDR and 0.94% in RPA over MAJL-Stage I when incorporating these additional datasets (MUSDB18 and MIR_ST500) on the MIR-1K dataset, validating the effectiveness of our two-stage training method. (iv): Besides, the experimental results on the MedleyDB dataset is similar with those on the MIR-1K dataset, showing the effectiveness of MAJL. Further analysis of the DWHS and the two-stage training method of our framework is described in Section 6.3 and Section 6.4, respectively.To further validate the effectiveness of MAJL in enhancing both tasks, we conduct an evaluation on the test sets of single-labeled datasets. The model under evaluation in this experiment is the same as the one discussed in Section 6.1.1. In the Stage II of training MAJL, additional singlelabeled music data from both MUSDB18 and MIR_ST500 is utilized. Additionally, the test sets of the MSS and PE tasks are derived from MUSDB18 and MIR_ST500, respectively. The results on MUSDB18 and MIR_ST500 are summarized in Table 3 and Table 4 , respectively. The results indicate that our framework effectively learns the relationship between both tasks, enhancing the performance of each individual task. The results on the MUSDB18 dataset, presented in Table 3 , show that our framework achieves state-of-the-art performance for the MSS task. Specifically, MAJL trained on the MIR-1K dataset in the Stage I outperforms the previous best MSS model Methods RPA(%) RCA(%) CNN-Raw [11] 76.58 77.26 JDC [37] 80.09 80.38 U-Net [29] with CREPE [32] 81.61 82.36 ResUNetDecouple+ [36] with CREPE [32] 81.96 82.43 U-Net [29] with HARMOF0 [61] 82.00 82.35 ResUNetDecouple+ [36] with HARMOF0 [61] 82 (KUIELAB-MDX-Net) by 1.13 in SDR. This result illustrates our framework has the capability to effectively leverage the PE task to improve the performance of the MSS task. Similarly, the results on the MIR_ST500 dataset, as shown in Table 4 , demonstrate that our framework achieves state-of-the-art performance for the pitch estimation from mixture music. In particular, MAJL trained on the MIR-1K dataset in the Stage I outperforms the previous best method (ResUNetDecouple+ with HARMOF0) by 1.71% in RPA. This result indicates that our framework can effectively leverage the MSS task to enhance the performance of the PE task. In summary, these results highlight that our framework effectively leverages the mutually beneficial relationship between both tasks, thereby improving their individual performances.In this experiment, we investigate the effect of using different modules in our framework to demonstrate its generality. Our framework can employ various model architectures for the MSS Module and the PE Module. Therefore, we evaluate the performance of our framework using different combinations of these modules, including Re-sUNetDecouple+ [36] , U-Net [29] , HARMOF0 [61] and CREPE [32] .For this experiment, we utilize the MIR-1K dataset in Stage I because MAJL-Stage I, trained with the MIR-1K dataset, demonstrates superior performance on the MUSDB18 and MIR_ST500 datasets, thereby enabling the generation of better pseudo-labels. The results obtained with different combinations of these MSS and PE modules are summarized in Table 5 . According to the results in this table, we can find three main observations as follows.Firstly, this experiment shows the great generality of our framework. As shown in Table 5 , MAJL consistently outperforms both the corresponding pipeline and naive joint learning methods across all module combinations. Particularly, when utilizing U-Net and CREPE, our framework achieves an improvement of 0.94 in SDR and 2.95% in RPA compared to the corresponding pipeline method. Similarly, our framework outperforms the corresponding naive joint learning method by 0.59 in SDR and 2.54% in RPA. These results validate the model-agnostic nature of our framework, showing its consistent performance enhancement across various music source separation and pitch estimation models.Secondly, the two-stage training method and DWHS are robust with different MSS and PE modules. Specifically, MAJL using Re-sUNetDecouple+ and CREPE achieves the best performance. Notably, compared to MAJL-Stage I, MAJL achieves improvements of 0.65 in SDR and 0.94% in RPA, demonstrating the effectiveness of the two-stage training method in leveraging large single-labeled Table 5 : Performance with different combinations of modules on the MIR-1K dataset. U, R, H, C represents the model architecture U-Net [29] , ResUNetDecouple+ [36] , HARMOF0 [61] and CREPE [32] , respectively. MAJL here uses both MIR_ST500 and MUSDB18 as single-labeled data. Lastly, our framework effectively learn the relationship between MSS and PE tasks, making both tasks beneficial for the each other. For example, MAJL using ResUNetDecouple+ and CREPE outperforms MAJL using U-Net and CREPE by 0.61 in SDR and 1.88% in RPA. This improvement arises from the better performance of ResUNetDecouple+ in the MSS task, where the effectiveness of the MSS task is beneficial for the PE task. Similarly, MAJL using ResUNetDecouple+ and CREPE outperforms MAJL using ResUNet-Decouple+ and HARMOF0 by 0.38 in SDR and 1.60% in RPA. This is because CREPE performs better than HARMOF0 at the PE task, and the PE task is beneficial for the MSS task. Thus, our framework also has the potential to further improve the performance of both tasks when better MSS and PE models become available.The above results demonstrate the robustness and great generality of our framework, since our framework achieves the best performance across all module combinations. Moreover, our MAJL framework effectively learns the relationship between MSS and PE tasks, resulting in enhanced performance for both tasks.To provide an intuitive representation of the weights extracted by the DWHS, we visualize the changes in these weights over iterations. In this experiment, we employ ResUNetDecouple+ as the MSS Module and CREPE as the PE Module, consistent with the previous experiment detailed in Section 6. In this figure, we observe that the weight assigned to the PE task for noisy music data is set close to 0, effectively mitigating the negative impact on the PE task. Additionally, for Case 3 and Case 4, the weights assigned to MSS and PE exceed 1, emphasizing the importance of handling hard samples. Other weights are approximately 1. All these weights are automatically set based on the analysis presented in Table 1 . Furthermore, the results presented in Table 2 demonstrate that the DWHS method outperforms the corresponding naive joint learning method. These findings highlight that the DWHS method can adaptively determine appropriate weights for both noisy and hard samples, leading to enhanced performance in both MSS and PE tasks.In this experiment, we use the MUSDB18 and MIR_ST500 datasets as single-labeled music data to explore the impact of the threshold (𝑡ℎ) used in the two-stage training method. The ResUNetDecouple+ is used as the MSS Module and CREPE is used as the PE Module, consistent with the previous experiment in Section 6.1. The MAJL is initially trained on the MIR-1K dataset in Stage I, as it demonstrates superior performance compared to MAJL trained on the MedleyDB dataset, as evidenced by the results presented in Table 3 and Table 4 .As shown in Figure 6 , the results corresponding to different thresholds demonstrate the effectiveness of the two-stage training method, and that the chosen threshold influences the performance of MSS and PE tasks. Specifically, when incorporating both datasets (MUSDB18 and MIR_ST500) to the fully-labeled music data, the performance of both tasks firstly improved and then decreased as the threshold increased. This trend indicates that the quality of pseudo-labels affects the performance of both tasks, with lowerquality pseudo-labels leading to a decrease in the performance of both tasks. In addition, incorporating the MUSDB18 or MIR_ST500 dataset to the fully-labeled music data follows a similar trend in the performance of both tasks. Moreover, when the threshold was set 0.5 0.6 0.7 0.8 0.9 12.0 to 0.7, adding both datasets (MUSDB18 and MIR_ST500) to the fullylabeled music data achieves the best performance, outperforming MAJL-Stage I by 0.65 in SDR and 0.94% in RPA. These results show that the two-stage training method can leverage large single-labeled music data to enhance the performance of both tasks. 
future work: 
 Our experiments have primarily focused on vocals, given that vocals are a common target source for both MSS and PE tasks , as highlighted in previous studies [28, 44] . Furthermore, the availability of music data containing other target sources and corresponding pitches was limited. However, it is important to note that our framework is applicable to a variety of musical instruments, including drums, bass and so on.Expanding our framework to encompass other instruments requires the acquisition or creation of annotated data for both MSS and PE tasks. Overcoming this challenge may involve adapting transfer learning techniques from related domains or exploring further unsupervised training methods. In conclusion, the future of our research holds the potential to adapt and expand our Model-Agnostic Joint Learning (MAJL) framework to encompass a broader range of musical instruments. These directions align with the ongoing evolution of music production techniques and computational audio analysis, making our framework important in advancing the field of music information retrieval. 
conclusion: 
 In this paper, we have proposed a model-agnostic joint learning (MAJL) framework to address the challenges in joint learning of music source separation and pitch estimation. MAJL is generic for both tasks in music information retrieval, offering the capability to adapt improved models for both tasks, further improving their performance. By designing a two-stage training method and a dynamic weighting method named Dynamic Weights on Hard Samples (DWHS), our framework effectively addresses the challenges of the lack of labeled data and the joint learning optimization, respectively. Through leveraging extensive single-labeled music data, MAJL learns the mutually beneficial relationship between music source separation and pitch estimation tasks, leading to improved performance for both tasks. Our experimental results show that the proposed framework achieves a significant improvement in both tasks, with 0.92 in SDR for the music source separation task and 2.71% in RPA for the pitch estimation task.  
 
 Appendix: The MedleyDB dataset comprises a total duration of 5.56 h and encompasses a wide range of musical instruments. Within this dataset, a subset of 3.21 h contains clean vocals along with their corresponding pitches. Notably, the duration of recordings for each other instrument is less than 1 h. Thus, we focus on clean vocals here. 


Decision: This paper proposes a model-agnostic framework for music source separation and pitch estimation. It includes a two-stage training method and a dynamic weighting method to address the lack of labeled data and the joint learning optimization issue. Experimental results show that the framework achieves a significant improvement in both tasks. All reviewers appreciate the novel contributions of the paper, and acknowledge that this paper is well-written. However, all reviewers agree that this research is single modal, not multimodal research. This is evidenced by the fact that the vast majority of the references are from the signal processing communities, including ICASSP, TASLP and ISMIR. This paper will fit beautifully to above venues. I am puzzled why the authors have decided to submit the paper to MM. In consideration of all review comments as well as the authors' rebuttal, I am unable to recommend an accept, unfortunately. However, I would strongly encourage the authors to take the review comments into consideration, and to improve the manuscript for a more suitable venue such as TASLP.

*** TPC Addendum ***
The paper has received positive scores and they have gotten better with the rebuttal. While the work is not completely multi-modal it does have significant overlaps with research interests of the multimedia community. Hence, the TPC is recommending a poster accept to allow for the conversation to continue at the conference.

Reviews: review: paper_summary: paper_summary: This paper proposed a model-agnostic joint learning for music source separation and pitch estimation, which is interesting for the audience of ACM Multimedia. However, there are several concerns make me believe that this paper is not ready for publication. The detailed comments can be found below.
strength_weakness: strength_weakness: The writing of this paper is good. The presentation of this paper is clear.
limitations: limitations: i) The most important issue of this paper is a lack of basic background knowledge of music information retrieval. The concept of pitch estimation (PE) and melody extraction is not clear in this paper. Pitch estimation commonly predicts pitch contours of each part of the polyphonic music and singing melody extraction predicts the fundamental frequency of the music. However, the paper uses pitch estimation algorithm to estimate fundamental frequency. Thus, actually the paper is doing singing melody extraction. It can be confirmed from Table 2 and Table 4, only fundamental frequencies are predicted. 

ii) Moreover, many important related works are missing. For example, other than HT Demucs, the rest of baseline methods are published before the year of 2022. There are many newly published music source separation methods [a,b] are worthy to mention and compare with. Since this paper is actually doing singing melody extraction, many singing melody extraction methods [c,d] are not cited and compared.

iii) The novelty of this paper is limited, as semi-supervised learning for melody extraction has been proposed [d,e].

[a] Schulze-Forster, Kilian, et al. "Unsupervised music source separation using differentiable parametric source models." IEEE/ACM Transactions on Audio, Speech, and Language Processing 31 (2023): 1276-1289.

[b] Luo, Yi, and Jianwei Yu. "Music source separation with band-split RNN." IEEE/ACM Transactions on Audio, Speech, and Language Processing (2023).

[c] Gao, Yuan, et al. "MTANet: Multi-band time-frequency attention network for singing melody extraction from polyphonic music." Proc. INTERSPEECH 2023. 2023.

[d] Yu, Shuai. "MCSSME: Multi-Task Contrastive Learning for Semi-supervised Singing Melody Extraction from Polyphonic Music." In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 1, pp. 365-373. 2024.

[e] Kum, Sangeun, Jing-Hua Lin, Li Su, and Juhan Nam. "Semi-supervised learning using teacher-student models for vocal melody extraction." arXiv preprint arXiv:2008.06358 (2020).
paper_summary: paper_summary: To address the challenges of the lack of the labeled data and join learning optimization in music source separation and pitch estimation, the authors proposed a MAJL framework which includes a two-stage training method and a dynamic weighting method (DWHS). The two-stage training method effectively leverages extensive single-labeled datasets and high-quality pseudo labels. The DWHS can handle error propagation and misalignment between different objectives by identifying hard samples and setting appropriate weights for these samples. Experimental results on public music datasets show that MAJL outperforms state-of-the-art methods on both tasks.
strength_weakness: strength_weakness: - The use of single-labeled datasets and the dynamic weighting method for hard samples based on several cases are highly novel. The loss function part of the dynamic weight module is particula interesting.
- It is good to show the effectiveness of the proposed method by comparing it with state-of-the-art methods in an exhaustive manner.
- The validity of the proposed framework is confirmed by checking the performance when using different model structures in MSS and PE.
limitations: limitations: - It was not clear why equation (13) added up the loss functions for all cases. How is equation (14) optimized for each case?
- It is unfortunate that the experiment is limited to vocal parts. If the authors claim this method is for music source separation, I would have liked to have seen experimental results showing that it can be used to separate other musical instruments as well.
- It would be good to have a discussion on the relationship between data volume in single-labeled datasets and fully-labeled datasets and performance.
- This research is limited to the modality of music and cannot be called a multimodal research.
paper_summary: paper_summary: This paper introduces a model-agnostic joint learning (MAJL) framework aimed at tackling the inherent challenges in simultaneously learning music source separation and pitch estimation. MAJL offers a versatile solution applicable to both tasks, enabling the incorporation of enhanced models tailored to each task. The implementation of a two-stage training approach and a novel dynamic weighting mechanism termed Dynamic Weights on Hard Samples (DWHS) effectively address the challenges of the lack of labeled data and the joint learning optimization.
strength_weakness: strength_weakness: 1. The paper is well written, well organized and provides a good rationale for their motivation with a detailed literature review. 
2. This paper clearly clarifies two challenges when performing MSS and PE tasks simultaneously.
3. The proposed MAJL framework simultaneously tackle MSS and PE tasks and achieves the state-of-the-art performance.
4. The experiments were adequately designed.
limitations: limitations: 1. Typo: “SVS” in the legend of Figure 5.
paper_summary: paper_summary: This paper proposes a model-agnostic framework for music source separation and pitch estimation. A two-stage training method and a dynamic weighting method are introduced to address the lack of labeled data and the joint learning optimization issue. Experimental results show that the framework achieves a significant improvement in both tasks.
strength_weakness: strength_weakness: 1. This paper is well-written and the motivation and description of the proposed method are very clear.
2. The experiment is sufficient and demonstrates the effectiveness of proposed framework in source separation and pitch estimate tasks. 
3. The model-agnostic joint learning framework can learn the mutually beneficial relationship between music source separation and pitch estimation tasks by leveraging extensive single-labeled music data
limitations: limitations: This paper lacks of novelty, the model agnostic framework mainly relies on joint training which is not a very novel method.