HKDSME: Heterogeneous Knowledge Distillation for Semi-supervised Singing Melody Extraction Using Harmonic Supervision

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 11:58:46

Abstract: Singing melody extraction is a key task in the field of music information retrieval (MIR). However, decades of research works have uncovered two difficult issues. \emph{First}, binary classification on frequency-domain audio features (e.g., spectrogram) is regarded as the primary method, which ignores the potential associations of musical information at different frequency bins, as well as their varying significance for output decisions. \emph{Second}, the existing semi-supervised singing melody extraction models ignore the accuracy of the generated pseudo labels by semi-supervised models, which largely limits the further improvements of the model. To solve the two issues, in this paper, we propose a \underline{h}eterogeneous \underline{k}nowledge \underline{d}istillation framework for \underline{s}emi-supervised singing \underline{m}elody \underline{e}xtraction using harmonic supervision, termed as \emph{HKDSME}. We begin by proposing a four-class classification paradigm for determining the results of singing melody extraction using harmonic supervision. This enables the model to capture more information regarding melodic relations in spectrograms. To improve the accuracy issue of pseudo labels, we then build a semi-supervised method by leveraging the extracted harmonics as a consistent regularization. Different from previous methods, it judges the availability of unlabeled data in terms of the inner positional relations of extracted harmonics. 
  To further build a light-weight semi-supervised model, we propose a heterogeneous knowledge distillation (HKD) module, which enables the prior knowledge transfers between heterogeneous models. We also propose a novel confidence guided loss, which incorporates with the proposed HKD module to reduce the wrong pseudo labels.
  We evaluate our proposed method using several well-known public available datasets, and the findings demonstrate the efficacy of our proposed method.

Main paper: 
 introduction: 
 Singing melody extraction is a challenging task in the field of music information retrieval (MIR). It aims to extract the fundamental frequency (f0) contour from polyphonic music. Recently it has become an active research topic with a lot of downstream applications, such as cover song identification [40, 53] , query-by-humming [44] , voice separation [23] , and music recommendation [27] . Singing melody contour obtained from extraction models can be utilized as an audio feature of musical information to enhance the performance of these downstream tasks.With the trend of artificial intelligence, deep learning models play an important role in the development of singing melody extraction techniques. A number of deep learning based methods [3, 21, 42, 49, 50] have been proposed for supervised singing melody extraction. Then, in an attempt to solve the problem of data insufficiency, semi-supervised singing melody extraction methods have become a cutting-edge direction. Some pioneer works [29, 48] adopt semisupervised learning to utilize unlabeled music tracks as training data for the singing melody extraction task, seeking to improve the melody extraction performance. However, these works either do not contain the process of data selection for unlabeled data or use data augmentation based consistency regularization methods.Despite these remarkable successes in singing melody extraction, in this paper, we try to further improve the performance of melody extraction models from two perspectives: a rethinking of supervised training paradigm and an innovation in the approach of semi-supervised singing melody extraction.For supervised learning methods of the singing melody extraction task, the prior works employ the binary classification paradigm on all audio pixels in the spectrogram, to predict whether each of them is classified as the f0 or not. However, this paradigm exposes one potential issue in the singing melody extraction task: all pixels in the spectrogram are treated equally as either melodic pixel or non-melodic pixel, while in the actual analysis of music signals, pixels often have different importance. One of the most representative examples is the harmonic and sub-harmonic information of the singing melody, which has been proven crucial for singing melody extraction [1, 26, 52] . They are distributed above and below the singing melody line according to a fixed ratio, and synchronize with the changes of the singing melody. If we treat these pixels equally as just melodic or non-melodic pixels, then not only can we not make use of the relationship between these harmonics/sub-harmonics and the singing melody, but they may even in turn be misidentified as the singing melody (known as octave error), because they possess similar patterns and activation values as the singing melody.For semi-supervised singing melody extraction, while data augmentation is treated as a popular consistency regularization method, the accuracy of pseudo labels is still a challenging issue. Yu et al. [48] has claimed that the singing melody extraction task is very sensitive to the data augmentation, many data augmentation based consistency regularization methods can not obtain satisfied results, such as MixMatch [4] , MeanTeacher [43] . We seek for some method that uses the correlations inside features as a consistency regularization to alleviate this issue. And we expect that such methods can be both light-weight and accurate to further increase the efficiency of the model's utilization of unlabeled data in the semi-supervised learning scenario.Following the above analysis, in this paper, we first propose a new paradigm, which uses harmonic supervision for supervised singing melody extraction. To be specific, we first leverage harmonic and sub-harmonic information as additional labels, then we perform a four-class classification: melodic pixels, harmonic pixels, sub-harmonic pixels and non-melodic pixels. The new paradigm encourages the model to learn the positional correlations in the spectrogram, alleviating the octave error issue.Extending from supervised learning, we propose to leverage the extracted harmonic and sub-harmonic positional information as a consistency regularization for semi-supervised learning. Different from previous methods, it judges the availability of unlabeled data in terms of the inner positional relations of extracted harmonics. We expect to see how the performance of the task is improved due to the proposed harmonic consistency regularization via semi-supervised learning.To further build a light-weight semi-supervised model, we also propose a heterogeneous knowledge distillation (HKD) module, which enables the prior knowledge transfer between heterogeneous models. We also propose a novel confidence-guided labeling loss to better train the proposed HKD module.The contribution of this is summarized as follow:• A new training paradigm is proposed for supervised singing melody extraction task, which uses harmonic supervision to classify the audio pixels in the spectrogram into four classes. The proposed harmonic supervision method encourages the model to learn positional correlations in the spectrogram, reducing the octave errors. • Extending from supervised singing melody extraction, we apply harmonic supervision to the semi-supervised scenario as a consistency regularization method to improve the accuracy of pseudo labels. • To further build a light-weight semi-supervised model, a heterogeneous knowledge distillation module is proposed to enable the prior knowledge can be transferred between heterogeneous models. We also propose a novel confidenceguided labeling loss to better train the proposed HKD module. • We use MIR-1K dataset and part of music tracks of the Med-leyDB dataset as labeled data for training the model and we evaluate the performance on the well-known ADC2004, MIREX 05, iKala and another part of MedleyDB. The experimental results demonstrates the superiority of our method compared with other state-of-the-art ones. 
related works 2.1 supervised singing melody extraction: 
 Deep learning models for the singing melody extraction task undergone various model architectures throughout its history. Kum et al. [31] proposes a multi-column deep neural network to learn a nonlinear mapping between frame and melody. Subsequently, many convolutional neural network (CNN) based approaches have been developed to better capture spectral-temporal information [9, 14, 16, 33, 42] . In addition, the use of musical prior knowledge and structural priors has further broadened the design of melody extraction models [11, 17, 21, 30, 35] . The relationship between frequencies can be further captured through multi-dilation or attention networks [13, 15, 50, 52] , or harmonic constant-Q transform (HCQT) [5] . The separate prediction of octave and pitch-class is proposed in [8] to further enhance the octave accuracy and chroma accuracy of the melody extraction. These models further improve the melody extraction performance.Although semi-supervised learning is a crucial method to handle the unlabeled data in the era of artificial intelligence, merely little works [29, 48, 51] have been studied for singing melody extraction. As far as we know, Kum et al. [29] employed a pretrained teacher model to generate pseudo labels on unlabeled data. However, since there is a lack of the process of data selection, it is prone to generate wrongly predicted pseudo labels, which would decrease the performance of the model. Yu et al. [51] proposed a few-shot learning algorithm to address the imbalance distribution of the samples due to the scarce of labeling data. Unfortunately, this algorithm can not be used in the scenario of semi-supervised learning. In the field of machine learning, consistency regularization has become an active research direction [10, 24, 34, 38, 55] . A lot of popular consistency regularization methods [2, 4, 32, 43, 47] have been proposed to deal with the unlabeled data. Unfortunately, Yu et al. [48] has claimed that the singing melody extraction task is too sensitive to obtain satisfied results. In this work, we propose a harmonic supervision based consistency regularization method.Our work is related to knowledge distillation, which is initially proposed in [20] . Hinton et al. [20] proposed to use the output class probabilities of a static cumbersome model as soft targets to teach a light-weight student model. Zagoruyko et al. [54] proposed a method that encourage the student model to generate the same attention as the teacher model. Wu et al. [46] proposed a mutual learning method to learn complementary features in semisupervised learning. However, these methods mentioned above are either logit-based or intermediate-feature based methods, which are too heavy to train the model. In this paper, we propose a confidence guided loss incorporated with HKD module to dynamically adjust the amount of information learned from teacher model. 
methodology: 
 The overview of the proposed framework is presented in Fig. 2 . We choose to use FTANet [50] as the feature extractor of HKDSME.The proposed framework HKDSME consists of three modules: harmonic supervision (HS) module, harmonic consistency regularization (HCR) module and heterogeneous knowledge distillation (HKD) module. At the same time, we perform HS module, HCR module and HKD module between the two groups of the generated music representations. We will introduce each components in the following subsections.In this paper, the inputs are from both labeled and unlabeled data.For the input data, the music signal can be denoted as𝐷 = {𝐷 𝑙 , 𝐷 𝑢 }. 𝐷 𝑙 = {(𝑥 1 , 𝑦 1 ), (𝑥 2 , 𝑦 2 ), ...(𝑥 𝑚 , 𝑦 𝑚 )} and 𝐷 𝑢 = {𝑢 1 , 𝑢 2 , ..., 𝑢 𝑁 } denote the labeled music data and unlabeled music data, respectively. 𝑀 and 𝑁 are the number of labeled and unlabeled data. 𝑇 denotes the whole training dataset. The learning objective function is constructed in the following form:EQUATIONwhere 𝐿 𝑙 is the loss function of the supervised learning and 𝐿 𝑢 is the loss function of the unsupervised learning. 𝜔 is a non-negative parameter, 𝜃 represents the parameters of our proposed framework. The aim of the harmonic supervision (HS) module is to encourage the model to learn the positional correlation relationship in the spectrogram. Distinct from the existing binary classification paradigm, the HS module force the model to predict not only melodic and non-melodic pixels in the spectrogram, but harmonic and subharmonic pixels. To be specific, we employ a convolution layer with the kernel size of 1 × 1, and the number of output channel of the convolution layer is set to 4. The convolution layer is followed by the batch normalization and Softmax Layers. The output of the module is four feature maps: 1) a feature map with melodic pixels and non-melodic pixels, 2) a feature map with harmonic pixels and non-melodic pixels, 3) a feature map with sub-harmonic pixels and non-melodic pixels and 4) a feature map with non-voiced pixels and non-melodic pixels as shown in Fig. 3 . Note that non-voiced pixel, as an auxiliary flag, indicate that there is no singing voice at that time step. To be clear, given an input spectrogram 𝑆, the output 𝑃 (the melody contour, harmoic/sub-harmonic contour and non-voice contour) can be predicted as follow:EQUATIONwhere 𝑓 (•) denotes the feature extractor (e.g., FTANet without classification head) for singing melody extraction, 𝐻𝑆 (•) denotes the proposed harmonic supervision module and 𝐹 𝑀 = {𝐹 𝑀 𝑖 |𝑖 ∈ {1, 2, 3, 4}}. Since the HS module follows the plug-and-play fashion, it can be used as classification head on all of the existing deep learning based singing melody extraction models.The aim of the harmonic consistency regularization (HCR) module is to select the generated pseudo labels based on the essential for 𝑡 ← 1 to 𝑇 do7: if 3𝑦 1 3 𝑓 0 𝑡 == 2𝑦 1 2 𝑓 0 𝑡 == 𝑦 𝑓 0 𝑡 == 1 2 𝑦 2𝑓 0 𝑡 == 1 3 𝑦 3𝑓 0 𝑡 then 8:𝑁𝑢𝑚 + + end if 21: end for 22: return 𝐷 characteristics of the musical audio and reduce the computation. Given an unlabeled spectrogram, we first employ HS module to predict the following key values related to the singing melody: f0, 2f0, 12 f0, 3f0, 1 3 f0. After obtaining the values, we can validate the predictions mentioned above. If the predicted values are according to the corresponding ratio, we can add the input into the training data for next iteration of the training. Otherwise, there are two kinds of cases need to be discussed: i) If the values are all wrong, the input data will be discarded directly. ii) If not all of the values are right, the input data we will give a low confidence score to the input data. For example, if there are four out of five values are right, we will give a confidence score of 80% to the input data 1 . The detailed procedures are presented in Alg. 1. By this way, we can judge the availability of the unlabeled data. In addition, since this method does not need perform data augmentation, not only can we reduce the cost of computation by data augmentation, but we can avoid the performance decrease comes from perturbations to the sensitive spectrogram by data augmentation.In order to build a light-weight semi-supervised model for singing melody extraction, we propose a heterogeneous knowledge distillation (HKD) module. The detailed HKD module is presented in Fig. 4 . To achieve this, we first employ a large pre-trained model (e.g., FTANet) as the teacher model, and a light-weight model (e.g., MSNet) as the student model. Then we choose several layers from the student model, and we divide the chosen layers into three groups: head layers, intermediate layers and bottom layers. Unlike previous methods [19, 25, 36, 45] , we employ a three-layer small neural network to transform the feature maps from the chosen layers and the feature map of the last layer from the teacher model into the shared hidden space. Then the information from teacher and student models can be fused and used for training and testing.Formally, given an input spectrogram 𝑆, we first feed 𝑠 into the teacher model and obtain the prediction 𝑝 𝑡 𝑖 . We then feed 𝑠 into the student model and obtain 𝑝 𝑠 𝑖 . In order to avoid the collapse of the model, we freeze the parameters in the teacher model. After obtaining 𝑝 𝑡 𝑖 and 𝑝 𝑠 𝑖 , There are three loss functions are calculated. We first perform cross entropy loss function between 𝑝 𝑠 𝑖 and 𝑦 𝑖 :EQUATIONwhere |𝐷 | denotes the number of training data and 𝐶𝐸 (•) is the cross entropy loss function. Then, KL divergence is employed to calculate the difference between predictions from teacher and student model:EQUATIONwhere 𝐾𝐿(•) denotes the KL divergence loss function. By using the two loss functions the outputs among teacher, student and the ground truth could be aligned.Although we can get a high performance from the teacher model, the teacher model will still generate wrong pseudo labels. To improve the accuracy of the pseudo labels generated by the teacher model, we propose a novel confidence-guided labeling (CGL) loss to achieve this. Since the teacher model will output a probability value in addition to the prediction, the proposed CGL loss is based on how much confidence the teacher model shows. If the prediction from teacher model is not that confident, then the output from student model may stick to its own predictions. Otherwise, the student should obey the prediction from teacher model. The CGL loss can be calculated:EQUATIONwhere 𝑐 denotes the index of frequency bin corresponding to the f0, ĉ denotes the remaining frequency bins are not f0, 𝛼 is a hyperparameter to scale the term (1 + 𝑔 𝑡 𝑐 ), 𝑔 𝑡 𝑐 ∈ (0, 1) denotes the confidence shows by the teacher model and 𝑔 𝑠 𝑐 ∈ (0, 1) denotes the confidence shows by the student model. The overall loss function of the proposed HKDSME framework can be calculated:𝐿 = 𝐿 𝐶𝐸 + 𝐿 𝐾𝐿 + 𝐿 𝐶𝐺𝐿 (6) 
experiments 4.1 datasets: 
 We train and evaluate our proposed HKDSME framework on several public datasets, the descriptions of the datasets we used are listed in Table 1 . For the training data, we first choose 1000 popular music tracks from MIR-1K [22] and 35 popular music tracks from MedleyDB [6] with melody annotated. Then we also choose 700 popular music tracks from FMA dataset [12] without labels. We also use 30 popular music tracks from RWC dataset [18] . For the testing data, we use four well-known testing datasets for this task: 12 tracks from ADC2004, 9 tracks from MIREX052 , 12 tracks from MedleyDB and 262 tracks from iKala [7] .Following the convention in the literature [39] , we use the following metrics for performance evaluation: overall accuracy (OA), raw pitch accuracy (RPA), raw chroma accuracy (RCA), voicing recall (VR) and voicing false alarm (VFA). We use mir eval library [37] with the default setting to calculate the metrics. For each metric other than VFA, the higher score, the higher performance. In the literature, OA is often considered more important than other metrics. The proposed framework is implemented using PyTorch 3 . All experiments are conducted on a machine with two NVIDIA RTX 3090 GPUs. For a fair comparison, we train the baseline models using the same training data. Following [21] , we choose to use a set of input representations. It contains three parts: (1) the generalized cepstrum(GC) [28] , (2) the generalized cepstrum of spectrum (GCoS) [41] , (3) the Combined Frequency and Periodicity (CFP) spectrum [42] . In this work, the audio files are resampled to 8 kHz and merged into one mono channel following [50] . Data representations are computed with a Hanning window of 768 samples and hop size of 80 samples. To adapt the pitch ranges required in singing melody extraction, following [21] , we set hyper-parameters in computing the CFP for our model. For vocal melody extraction, the number of frequency bins is set to 320, with 60 bins per octave, and the frequency range is from 31 Hz (B0) to 1250 Hz (D#6).We compare our framework with seven state-of-the-art (SOTA) methods for singing melody extraction: (1) DSM [5] , (2) MSNet [21] , (3) MD+MR [15] , (4) Teacher-student [29] , (5) FTANet [50] , (6) HGNet [49] , (7) MCSSME [48] . To demonstrate the effectiveness of our proposed method, we train the proposed framework HKDSME and compare our method with other baseline methods.The quantitative results are shown in Table 2 and Table 3 . It is 3 https://pytorch.org observed that with assisted unlabeled music data, our proposed HKDSME achieves the best performance on four public testing sets in general. For comparison with other baselines, when focusing on OA, the proposed method outperforms FTANet by 10.6% in ADC2004, by 1.5% in MIREX 05, by 3.0% in Medley DB and by 1.7% in iKala, relatively. When comparing with semi-supervised methods on OA, the proposed method outperforms Teacher-student by 9.0% in ADC2004, by 4.9% in MIREX 05, by 7.6% in Medley DB and by 6.5% in iKala, relatively. It is worthy to mention that the effectiveness of our method is from the harmonic consistency regularization and harmonic knowledge distillation compared with the performance of a semi-supervised model Teacher-student [29] .To investigate what types of errors are solved by the proposed model, a case study is performed on several music tracks chosen from ADC2004 and MIREX 05 datasets. We choose FTANet [50] to compare with due to its effectiveness and popularity. As depicted in Fig. 5 , we can observe that there are fewer octave errors in our model than in FTANet in general. We can also observe that there are some errors that are wrongly predicted near the right frequency bin in diagram (a) near the time of 1000 ms, which are correctly predicted in diagram (c). Through the visualization of the predicted melody contour, we can say that the performance gains of the proposed model can be attributed to solving the octave errors and other To investigate the effectiveness of the key components in our framework, we conduct ablation studies and the quantitative results are presented in Table 4 . We first remove the harmonic supervision module and use binary classification to train the framework. As observed in Table 4 , the performances of OA decreased by 4.7% in ADC2004 and 4.4% in MIREX 05. We then remove the HCR module and use data augmentation to select unlabeled data, the perfor- 0.7% in MIREX 05. The results show that the proposed harmonic supervision paradigm contributes most to HKDSME. And the proposed HCR and HKD modules can also improve the performances in the scenario of using very limited labeled data. Overall, the key components of our framework HKDSME are tightly incorporated and collaboratively devote to remarkable results. Effects of harmonic supervision. To investigate how the supervision improves the performance, we vary the harmonic numbers of harmonic supervision. The quantitative results is presented 5 . To compare with the binary classification, we also add the results of f0 in the table. Specifically, we perform five combinations of harmonics to justify the effectiveness of our proposed HS module. We first try the combination of f0, nf0 and 1 𝑛 f0, the results show that using the above combination will improve both of the peformances on ADC2004 and MIREX 05. It is worthy to mention that the improvements on MIREX 05 are better than in ADC2004, that is because MIEX05 are popular music tracks, which are prone to be misidentified by sub/harmonic information during extracting f0. Then we try the combination of f0 and harmonics, the results show that on both datasets, the OAs are improved by 2.0% on ADC2004, and by 1.6% on MIREX 05, when compared with binary classification. We also try the combination of f0 and sub-harmonics, the OAs are improved by 2.0% on ADC2004, and by 2.0% on MIREX 05. Although not shown in the table, we tried more sub/harmonics, the results are nearly the same as the best performance in Tab. 5Effects of 𝛼. We vary the value of 𝛼 for the CGL loss, and the results is shown in Table 6 . To observe the effects of 𝛼, we vary the 𝛼 from 0.5 to 2.5. We can observe from Tab. 6, when 𝛼 is set to 0.5, we can obtain the best performance. We can also observe that the results are nearly the same, which indicates that the proposed framework HKDSME is not very sensitive to 𝛼. 
conclusion: 
 In this paper, we proposed a heterogeneous knowledge distillation framework for semi-supervised singing melody extraction using harmonic supervision, called HKDSME. Specifically, we proposed a new training paradigm that uses harmonic supervision to classify the audio pixels in the spectrogram into four classes. Extending from harmonic supervision, we proposed to use the extracted harmonics to judge the availability of unlabeled data in terms of inner positional relations. To build a light-weight semi-supervised model, we proposed a heterogeneous knowledge distillation, this module enables the prior knowledge can transfer between different architectures of the models. To further improve the accuracy of pseudo labels, we also proposed a confidence-guided labeling loss function. HKDSME evaluates on a set of well-known public melody extraction datasets with promising performances. The experimental results demonstrate the effectiveness of the HKDSME framework for singing melody extraction from polyphonic music using very limited labeled data scenarios. This work has also provided another verification of the feasibility for integrating harmonic supervision and heterogeneous knowledge distillation for semi-supervised singing melody extraction.  
 
 Appendix: Obviously, more complex scoring criteria can be explored for the task in future.https://labrosa.ee.columbia.edu/projects/melody 


Decision: In general, this is a well-motivated and well-written paper of good quality.

+ Based on important observations that pixels from spectrograms have different importance, the authors designed new training objectives to add harmonic and subharmonic information, referred to as Harmonic Supervision, in classifying pixels on the input spectrogram.
+ The authors proposed a Heterogeneous Knowledge Distillation method, which involves selecting layers from the student model and projecting the outputs according to a specially designed loss function.
+ The authors introduced Harmonic Consistency Regularization, which carefully filters pseudo-labeled data based on the physical relationships of F0, 2F0, 3F0, 1/2F0, and 1/3F0.

- Numerous typos need to be fixed before publication.

Reviews: review: paper_summary: paper_summary: The paper introduces a heterogeneous knowledge distillation framework, HKDSME, for semi-supervised singing melody extraction using harmonic supervision. It adopts a novel training paradigm to classify the audio pixels in the spectrogram into four classes, as well as a light-weight semi-supervised model. Experiment results demonstrate that HKDSME outperforms previous works for singing melody extraction with limited labeled data.
strength_weakness: strength_weakness: 1. The proposed approach is well motivated. Singing melody extraction from polyphonic music is a challenging task. Since previous studies treated all pixels in the spectrogram equally and ignored the accuracy of pseudo labels generated by extraction models, HKDSME appears as a logic and promising solution.
2. The paper is well structured and well written.
3. The proposed harmonic supervision module follows the plug-and-play fashion and is applicable to all singing melody extraction models.
4. The heterogeneous knowledge distillation module is lightweight and can make full use of limited labeled data.
limitations: limitations: Editing and Typesetting: Check the paper for spelling mistakes, punctuation usage, word collocations, and other errors to ensure the writing is smooth and refined. Some examples (non-exhaustive list):
- line 32-33: "… which **enables the prior knowledge transfers** between heterogeneous models …"
- line 104-106: "… They are distributed above and below the singing melody line according to a fixed ratio, and **synchronize** with the changes of the singing melody …"
- line 200-203: "… The experimental **results demonstrates** the superiority of our method compared with other state-of-the-art ones …"
- line 282: "… a method that **encourage** the student model …"
- line 376: "… the HS module **force** the model to …"
- line 387: "… as an auxiliary flag, **indicate** that there is no singing voice …"
- ...
Lack of Generated Results: Adding the results generated by different models in the supplementary materials can make the conclusions more convincing.
paper_summary: paper_summary: This paper proposed a singing pitch extraction model for polyphonic music. Specifically, a new training objective is adopted that adds harmonic and subharmonic into classification. A new regularization is proposed to exploit inter-label relationships. Further, this objective is extended in a semi-supervised setting. Knowledge distillation is also adopted for heterogeneous models, with proposed confidence guided labeling loss, to make the model lightweight. Experiments show superior results than SOTA pitch tracking methods. Visualization of the learned representation shows a high correlation with frequency of melody.
strength_weakness: strength_weakness: - The author made important observations that pixels from spectrograms have different importance, and designed new training objectives to add harmonic and subharmonic information in classification. Further, a regulation objective is designed for pitch and its harmonic, which is very intuitive in singing voice where harmonics is a very common scenario.

- A knowledge distillation approach is proposed with a combination of cross entropy loss, KL divergence, and a group of task-related soft labels. Task-related loss is computed for multiple depth of the student models’ output.

- The visualization of features shows the learned feature is highly correlated with melody’s frequency. It indicates the proposed approach could be extended to various pitch-related audio analysis tasks.
limitations: limitations: **Major Issues**
- In Section 3.1, L_l and L_u are not explained. Not sure how they are computed.

**Minor Issues**
- From Table 5, the student model (HKDSME) has better performance than the teacher model (w/o HKD). The reviewer believes this is not a common scenario, so it can be better to include further comparison and discussion of behavior of teacher model and student model.
- TONet [8] seems not included in the comparison (although the proposed model has better performance than that of TONet).
paper_summary: paper_summary: This paper proposed HKDSME, a semi-supervised framework using heterogeneous knowledge distillation and harmonic supervision for singing melody extraction. Experiments show that HKDSME effectively extracts melodies from polyphonic music with minimal labeled data.
strength_weakness: strength_weakness: 1. The novelty of this approach lies in utilizing harmonic, sub-harmonic, and fundamental frequencies (F0), referred to as Harmonic Supervision, to classify pixels on the input spectrogram.
2. The authors proposed a Heterogeneous Knowledge Distillation method, which involves selecting layers from the student model and projecting the outputs according to a specially designed loss function.
3. The authors introduced Harmonic Consistency Regularization, which carefully filters pseudo-labeled data based on the physical relationships of F0, 2F0, 3F0, 1/2F0, and 1/3F0.
limitations: limitations: Please note the following typos:
1. On line 335, the last data pair subscript should be an uppercase 'M'.
2. In Algorithm 1, the labeled dataset is denoted with 'n', while the unlabeled dataset is denoted with 'M'. This notation is inconsistent with line 337.
paper_summary: paper_summary: This paper presents HKDSME, which leverages heterogeneous knowledge distillation (HKD) and harmonic supervision (HS) for semi-supervised singing melody extraction. For the HS, the authors propose a multi-target classification strategy to help deep learning models to tell the singing part from other information in the spectrograms (e.g. harmonics and sub-harmonics). Then, the authors propose a harmonic consistency regularisation (HSR) module to select the predicted labels by HS and achieve higher-quality data augmentation accordingly. Finally, in the singing melody extraction part, HKD is utilised to build a light-weight melody extractor (student model) with the reference to a large pretrained model (teacher model). Experimental results and ablation studies advocate the effectiveness of the proposed HKDSME framework, as well as the usefulness of each component in the framework.
strength_weakness: strength_weakness: * This paper is clearly-written and easy to follow.
* The methodology is well motivated and elaborately designed. Using multiple targets and HKD is an intuitive and clever idea to address existing problems.
limitations: limitations: * Line 686: Instead of using "several music tracks", report the actual number of tracks, name & genre of tracks etc. (e.g. are you using the same genres) to highlight the objectivity of your track selection. The same goes for Line 742.
* Section 4.4: instead of relying solely on figure 5, some real statistics should be reported (e.g. percentages of each error, p values from statistical comparisons between groups) to support the advantage of the proposed method over other baselines.
* In addition to the evaluation metrics, some **demonstrations** (including the extracted melodies from HKDSME and other baselines) are recommended to help the reviewers understand the contribution and effectiveness of this work. Currently there are not any in the paper or the supplementary materials.
* Some minor nitpicks that may need double checks:
	* Line 177: enables the prior knowledge **to** transfer...
	* Line 335: Should $m$ in $(x_m, y_m)$ in $D_l$ be capitalised as $(x_M, y_M)$? In line 336 the authors denote $M$ as the total number of labelled data.
	* In footnote 1, "Obviously" does not appear academically appropriate.
	* In Line 499, does the lower case $s$ refer to the input spectrogram? (if that's true, should it be capitalised?)
	* Line 547: I think it should be "$\hat{c}$ denotes the remaining frequency bins **that** are not $\texttt{f0}$?".
	* Line 576: "In the literature...", I would recommend adding some citations here and briefly introducing why OA is the most important.