# Automatic Synthesis Technology of Music Teaching Melodies Based on Recurrent Neural Network

Authors: Yingxue Zhang; Zhe Li
Journal: Scientific Programming
 (2021)

---

## Abstract

Computer music creation boasts broad application prospects. It generally relies on artificial intelligence (AI) and machine learning (ML) to generate the music score that matches the original mono-symbol score model or memorize/recognize the rhythms and beats of the music. However, there are very few music melody synthesis models based on artificial neural networks (ANNs). Some ANN-based models cannot adapt to the transposition invariance of original rhythm training set. To overcome the defect, this paper tries to develop an automatic synthesis technology of music teaching melodies based on recurrent neural network (RNN). Firstly, a strategy was proposed to extract the acoustic features from music melody. Next, the sequence-sequence model was adopted to synthetize general music melodies. After that, an RNN was established to synthetize music melody with singing melody, such as to find the suitable singing segments for the music melody in teaching scenario. The RNN can synthetize music melody with a short delay solely based on static acoustic features, eliminating the need for dynamic features. The proposed model was proved valid through experiments.


---

## Body

## 1. Introduction

With the rapid development of modern computer science, many researchers have shifted their focus to computer-based algorithm composition or automatic music melody generation system. The research results on music melody synthesis and music modeling methods are being applied to various fields. The research of computer music creation aims to quantify and combine the emotional tendencies of music, with the aid of computer and mathematical algorithms. The specific tasks include aided composition, sound simulation and storage, and music analysis and creation [1, 2]. Computer music creation generally relies on artificial intelligence (AI) and machine learning (ML) to generate the music score that matches the original mono-symbol score model or memorize/recognize the rhythms and beats of the music. Despite its broad application prospects, the AI-based composition without needing lots of music knowledge rules is in the theoretical stage [3, 4].Speech processing has been widely applied in composition and songwriting, record production, and entertainment. Unlike simple speech synthesis, music melody synthesis has two additional processing steps: tone detection and transformation [5, 6]. Wenner et al. [7] preprocessed the musical melody synthesis corpus through automatic note segmentation and voiced/unvoiced sound recognition, constructed a high-quality music melody synthesis system, and proposed a music melody adjustment algorithm, which functions as an adaptive filter capable of detecting musical note cycles.AI has already been adopted to realize algorithm composition or automatic music generation [8–12]. Bilbao et al. [13] introduced bidirectional long short-term memory (LSTM) neural network to the mixed music generation system and thus realized the training of multi-voice music datasets. Their approach provides effective chord progressions while ensuring melody time and transposition invariance.Electronic synthetic tones bring rich new sound experience to music of various styles and themes. Electronic musical instruments differ from traditional acoustic instruments in sound rendering principle and acoustic features [14–19]. Miranda et al. [20] expounded the computer-aided means to realize the acoustic features, voice editing, and modulation of electronic sound melodies and provided a valuable reference for applying electronic sound melodies in modern music creation. However, computer-based accompaniment has a rigid chord structure, which cannot easily adapt to diverse music styles. To solve the problem, Taigman et al. [21] put forward an adaptive automatic accompaniment algorithm, including chord series extraction and automatic accompaniment figure acquisition, and created a suitable accompaniment figure database based on chord sequences in the light of the features of music melodies and emotions.The existing studies at home and abroad mostly concentrate on the methods, melodic forms, and tone synergy of computer music creation [22–26]. However, there are very few music melody synthesis models based on artificial neural networks (ANNs). Some ANN-based models cannot adapt to the transposition invariance of original rhythm training set. To overcome the defect, this paper attempts to develop an automatic synthesis technology of music teaching melodies based on recurrent neural network (RNN).The remainder of this paper is organized as follows. Section2 extracts the acoustic features from music melody. Section 3 applies the sequence-sequence model to synthetize general music melodies. Section 4 establishes an RNN to synthetize music melody with singing melody, aiming to find the suitable singing segments for the music melody in teaching scenario. Finally, experiments were carried out to prove the effectiveness of our model.


## 2. Acoustic Feature Extraction

The automatic synthesis of music melody aims to obtain a melody that is beautiful and pleasant to human ears. To describe the differences in the auditory sensitivity of human ears to music melodies of different frequencies, the linear frequencyμ of each music melody was transformed based on mel scale frequency μMR:(1)μMR=2595×log101+μ700.Under the scale of mel scale frequencyμMR, the multiples of the μMR difference between two music melodies are roughly equal to those of the tone difference perceived by human ears.For the above reason, mel scale was adopted to extract acoustic features in our music melody synthesis system. Since the music melody signal in the high-frequency band is relatively weak, such a signal was compensated through preemphasis. Letβ be the preemphasis factor. Then, the preemphasis of music melody a(τ) can be described by(2)bτ=aτ−β⋅aτ−1.To prevent spectrum leakage and enhance the continuity of the left end and right ends of the signal frame, it is necessary to perform framing and windowing of the music melody signal with short-time stationarity features. LetCH(m), m = 0, 1, …, M − 1, be the framed music melody signal and CK(m) be the function of the Hamming window. Then, the windowed signal CH∗(m) can be described by(3)CH∗m=CHm×CKm,where CK(m) can be described by(4)CKm=0.54−0.46cos2πmM−1,0≤m≤M−1,1,otherwise.It is difficult to extract the features from time-domain music melody signal. The general practice is to convert the signal to the frequency domain through short-time Fourier transform (STFT) before further analysis. LetCH∗(m) be the input signal of STFT and M be the number of Fourier points. Then, the fast Fourier transform of the M points of the windowed framed time-domain music melody signal CH∗(m) can be expressed as(5)CHl=∑m=0M−1CH∗mexp−j2πlmM,0≤l≤M.In the frequency domain, the absolute value of the spectrum of the music melody signal can be described as(6)PTl=CHl.The human ears can only detect the frequency components in a certain range. Therefore, the human auditory system could be treated as a filter bank that only allows some frequency signals to pass through. This paper simulates the human auditory system with a mel filter bank. Letμn, μmax, and μmin be the central, upper, and lower frequencies of the filter bank, respectively. Then, the transfer function of the filter bank can be described by(7)QFnl=0,k<μmin,l−μminμn−μmin,μmin≤l≤μn,μmax−lμmax−μn,μn≤l≤μmax,0,l>μmax.LetN be the number of triangular filters. Then, the signal on passing through the mel filter bank QFn(l) can be expressed as(8)on=∑l=μminμmaxPTlQFnln=0,1,2,…,N−1.The mel spectrum can be extracted by taking the logarithm ofon.


## 3. Sequence-Sequence Model-Based Music Melody Synthesis

Both speech synthesis based on statistical parameters and music melody synthesis based on neural networks face the following defects: the complexity of model construction and the dependence of front-end text processing on texts with strong linguistics knowledge. Unlike these approaches, sequence-sequence model-based speech synthesis can directly transform phonetic notations to waveforms and significantly simplify the front-end module. Since a standard neural network cannot directly process input and variable sequences with variable length, a sequence-sequence model is needed to handle the scenario that input sequence is not equal to the output sequence.This paper constructs a sequence-sequence model of music melody to realize the automatic synthesis of melody sequences. Figure1 shows the structure of the music melody synthesis system based on the sequence-sequence model. In the proposed model, the music melody recognition module receives a music melody sequence and outputs a singing melody sequence. The synthesis module receives the target melody sequence and outputs an audio sequence.Figure 1
Music melody synthesis system based on the sequence-sequence model.The RNN model consists of an encoder and a decoder, using an activation function Γ. The hidden stategτ at the current moment τ depends on the input gτ at the current moment and the hidden state gτ−1 at the previous moment τ − 1:(9)gτ=Γgτ−1,aτ.Letgψa be the hidden layer state of the neural network at moment ψa and s(∗) be the nonlinear transform. Then, the middle vector p of the encoder can be obtained through the nonlinear transform of each hidden layer states:(10)p=sg1,g2,g3,g4…gψa.The middle vectorp is equivalent to the final encoded state of the hidden layer:(11)p=sg1,g2,g3,g4…gψa=gψa.The next outputbi of the encoder can be generated based on the middle vector p and historical outputs b1, b2, …, bi-1. The encoder is often adopted to predict the next acoustic feature in music melody composition. It is necessary to determine the middle vector p and the existing acoustic features:(12)bi=argmaxObi=∏i=1nobi|b1,b2,…,bi−1,p.Letri−1 be the state of a hidden layer node in the RNN of the decoder; bi-1 be the output of that node at the moment i−1; and s(∗) be the nonlinear transform. Then, formula (12) can be simplified as(13)bi=sbi−1,ri−1,p.The sequence-sequence model is prone to a potential problem: the input sequence has the same influence on the weight of each element in the output sequence. This problem can be solved by the attention mechanism. Figure2 shows the structure of the attention-based model.Figure 2
Attention-based model.The attention mechanism highlights that different nodes belong to different parts of the input sequence. Letgυ be the hidden layer output of the encoder at the moment v; rτ-1 be the hidden layer output of the encoder at the moment τ − 1; and e be the alignment model. Then, the matching degree DOτj between the location of input layer nodes and output layer nodes can be calculated by(14)DOτυ=erτ−1,gυ,where e is a nonlinear function to compare and compute the matching degree between gυ and rτ−1. The greater DOτ is, the more necessary it is to emphasize the input sequence at the current moment during the decoding of music melody signal. Let U, V, and W be weight matrices. Then, the point multiplication, weighting, weight stitching, and sensing of the alignment model e can be, respectively, described by(15)erτ−1,gj=rτ−1ψgj,erτ−1,gj=rt−1ψVgj,erτ−1,gj=Vrτ−1ψ;gj,erτ−1,gj=UψtanhVrτ−1+Wgj.DOτj can be normalized by(16)βτj=expDOτj∑l=1ψaexpDOτl.The middle vectorpτ can be obtained through weighted summation:(17)pτ=∑j=1ψaβτjgj.Lets(∗) be the nonlinear transform. Then, the next hidden layer state can be calculated by(18)rτ=srτ−1,bτ−1,pτ.To realize frame-level feature mapping from the input to the output, the proposed model needs to transform the contextual features on the phoneme level and the frame level. This section puts forward a length prediction model for music melody, which supports the time supervision labeling. Suppose a phoneme sequence is given for a rhythm in a music melody sequence of the length M. Letε be the model parameter of the RNN; w be the rhythm state sequence of the melody; and Lm be the number of rhythm states of the melody. Then, the length prediction of music melody states can be regarded as the forecast of the length of the state allocation sequence. The prediction goal is to maximize the likelihood in the following formula:(19)logOw|ε,ψ=∑m=1M∑l=1Lmlogom,lδm,l.Letom,l(δm, l) be the probability density function of the length model of the music melody; δm,l be the time of the l-th state of the m-th phoneme; Ψ be the total length constraint; nm,l be the length of rhythm state predicted by network model; and φm,l be the variance obtained from the individual length of each phoneme in the music melody database. Solving the maximum likelihood of formula (19), the rhythm state length of each melody can be obtained by(20)δm,l=nm,l+σ⋅ϕm,l2.LetψS be the phoneme length specified in the music score. Then, σ can be calculated by(21)σ=ψ−∑m=1M∑l=1Lmnm,l∑m=1M∑l=1Lmϕm,l2.


## 4. RNN-Based Melody Synthesis

To find the suitable singing segments for the music melody in the teaching scenario, the RNN-based statistical music melody synthesis algorithm needs to realize the following goal in the synthesis phase: identifying the most possible acoustic feature sequenceũ from the signing melody sequence k with given linguistic features and a series of trained context-dependent music melodies Φ. Then, we have(22)u˜=argmaxuou|k,Φ˜=argmaxu∑∀wχu,w|k,Φ˜=argmaxu,wχu,w|k,Φ˜=argmaxu,wχu,w|k,Φ˜Δw|k,Φ˜=argmaxuχu|w^,Φ˜,where ŵ is the rhythm state of a melody:(23)w^=argmaxwΔw|k,Φ˜.Letλŵi be the mean vector under state ŵi; ∑ŵi be the corresponding covariance matrix; and λw^i=λw^1T,…,λw^QT and Σw^=Σw^1T,…,Σw^QT be the mean vector and covariance matrix under the given state w of singing melody sentence, respectively. If the output probability of the neural network obeys single Gaussian distribution, then formula (25) can be rewritten as(24)u˜=argmaxuouτ|w˜τ,Φ=argmaxu∏τ=1QMut;λw^τ,∑w^τ=argmaxuℜuτ;λw^,∑w^=λw^.From the statistical features of output probability (Figure3), it can be learned that λŵi is a jump series because the rhythm states of a melody are discrete and independent. The music melody signal reconstructed from λŵi has a discontinuous boundary of rhythm states. To solve the problem, this paper introduces an observation vector u, which covers the static acoustic feature and its first- and second-order derivatives with respect to time:(25)uτ=θτT,ΔθτTT.Figure 3
Statistical features of output probability.Let SC be the sparse coefficient matrix. Then, the relationship between the observation vector sequenceu=u1T,…,uQT and the acoustic eigenvector sequence θ=θ1Τ,…,θQT can be described by(26)uτ−1uτuτ+1⋮θτ−1dθτ−1dτdθτdτ+1dθτ+1⋮=SC=⋯⋮⋮⋮⋮⋯⋯0I00⋯⋯−1/2I01/2I0⋯⋯00I0⋯⋯0−1/2I01/2I⋯⋯000I⋯⋯00−1/2I0⋯⋯⋮⋮⋮⋮⋯=⋮θτ−2θτ−1θτθτ+1⋮.Combining formulas (24) and (26):(27)u˜=argmaxuℜuτ;λw^,∑w^=SC⋅θ.The maximization of the output probability is equivalent to finding the maximum ofθ:(28)θ˜=argmaxθℜSC⋅θ;λw^,∑w^=argmaxθlogℜSC⋅θ;λw^,∑w^.Find the partial derivative ofθ in formula (28):(29)∂logℜSC⋅θ;λw^,∑w^∂θ∝∂∂θSC⋅θ−λw^T∑w^−1SC⋅θ−λw^=SCT∑w^−1SC⋅θ−SCT∑w^−1λw^.Make formula (29) equal to 0, and a linear equation about θ can be obtained:(30)SCT∑w^−1SC⋅θ=SCT∑w^−1λw^.Unlike other deep neural networks, the RNN combines the output of the input layer and the output of the hidden layer at the previous moment into the input of the hidden layer. Therefore, the network can capture the dynamic law of sequential music melodies from the periodic connections between hidden layer nodes. Letωga, ωbg, and ωgg be the weight matrices of input layer-hidden layer, hidden layer-output layer, and hidden layer-hidden layer, respectively; γg and γb be the bias vectors of hidden layer and output layer, respectively; G(.) be the activation function between hidden layers; and {aτ}ψτ = 1, gττ|=1ψ, and {bτ}ψτ = 1 be the input music melody features, hidden layer sequence, and output features, respectively. Then, gτ can be expressed as(31)gτ=Gωgaτ+ωgggτ−1+γg.Besides,bτ can be given by(32)bτ=ωbggτ+γb.For the traditional RNN, vanishing gradient problem might occur during network training, owing to the use of backpropagation algorithm. To prevent this problem, the LSTM, a time RNN model, was adopted to synthetize the teaching audios including both singing melody and music melody. As shown in Figure4, an LSTM unit contains an input gate IGτ, a forget gate FGτ, and an output gate OGτ, as well as a memory cell MCτ. Let gτ and aτ be the hidden layer output and input signal of the network at the moment τ, respectively; ωI∗ and ωXH∗ be the weight matrices of input layer nodes and hidden layer nodes, respectively; CPH∗ and γ∗ be the weight and bias, respectively; and ○ be the Hadamard product of the elements of a matrix. Then, the operations of the input gate IGτ, the forget gate FGτ, the memory cell MCτ, and the output gate OGτ can be, respectively, expressed as(33)IGτ=sigmoidωIIGaτ+ωXHIGgτ−1+CPHIG∘MCτ−1+γIG,FGτ=sigmoidωIFGaτ+ωXHFGgτ−1+CPHFG∘MCτ−1+γFG,MCτ=FGτ∘MCτ−1+IGτ∘TanhωIMCaτ+ωXHMCgτ−1+γMC,OGτ=sigmoidωIOGaτ+ωXHOGgτ−1+CPHOG∘MCτ−1+γOG.Figure 4
Structure of an LSTM unit.Then,gτ can be calculated by(34)gτ=OGτ∘TanhMCτ.To reduce the delay of singing melody relative to music melody, the LSTM was adopted to build the acoustic model, and recurrent output layers were configured to further smoothen the acoustic features between adjacent frames. Figure5 shows the framework of the low-delay synthesis model of singing melody relative to music melody. Let ωbb be the weight matrix of the recurrent connections of the output layer extended from the traditional RNN. Then, we have(35)bτ=ωbggτ+ωbbbτ−1+γb.Figure 5
Low-delay synthesis framework of singing melody relative to music melody.Dynamic features are needed to smoothen the parameter trajectories. It is possible to obtain smooth parameter trajectories by smoothing the acoustic parameters with recurrent output layers. The LSTM-based recurrent output layers receive the activationgτ of the hidden layer and the output bτ−1 at the moment τ − 1, process them with the activation function and the input gate operation, and save some of the information to the state of the memory cell:(36)IGτ=sigmoidωIIGgτ+ωXHIGbτ−1+CPHIG∘MCτ−1+γIG.The stateMCτ of the memory cell at time τ can be obtained through forget gate operation and scrapping some useless information:(37)FGτ=sigmoidωIFGgτ+ωXHFGbτ−1+CPHFG∘MCτ−1+γFG,MCτ=FGτ∘MCτ−1+IGτ∘ωIMCgτ+ωXHMCbτ−1+γMC.Finally, the network outputbτ at time τ can be obtained through output gate operation of MCτ:(38)OGτ=sigmoidωIOGgτ+ωXHOGbτ−1+CPHOG∘MCτ−1+γOG,bτ=OGτ∘MCτ.


## 5. Experiments and Result Analysis

To compare the convergence of different music melody synthesis models, Figure6 shows the trend of the loss function value on the verification set of four models: DCNN based on static acoustic feature, LSTM, LSTM-RNN, and our model. The value of the loss function is the difference of model output and the actual value. Extended from AlexNet, the DCNN boasts deep layers and numerous parameters and has been widely applied to signal recognition and image processing. As an RNN, the LSTM is suitable for processing and predicting important events with relatively long intervals and delays in the time series. The network has been adopted in many scientific fields, namely, language learning and translation, robotic control, image analysis, document summarization, speech and image recognition, handwriting recognition, chatbot control, disease, click rate and stock prediction, and music synthesis.Figure 6
Convergence of different network models.The loss of each model continued to decrease with the growing number of iterations and eventually converged. After convergence, the DCNN had the greatest loss, the LSTM and LSTM-RNN had similar losses, and the proposed low-delay LSTM-LSTM realized the smallest loss.To compare the music melody generated by our model with the original music melody, forty segments of music melodies were randomly selected from a test set containing 577 segments. The music melodies synthetized by different networks were objectively measured by four metrics: BAP distortion, F0 RMSE, LE, and MCD. The results in Table1 show that MCD had the greatest influence on the synthetic music melodies. Overall, our model, which further smooths the acoustic features between adjacent frames with recurrent outputs, outperformed other networks, as evidenced by the small gaps of the four metrics. Therefore, our model is highly robust in finding the suitable singing segments for teaching.Table 1
Results before and after the addition of singing melodies.
BAP distortionF0 root mean square error (RMSE)Labeling error of voiced/nonvoiced sound (LE)Mel cepstral distance (MCD)DCNNPreaddition0.24812.3755.4595.135Postaddition0.23911.8925.3744.913LSTMPreaddition0.24112.8415.6215.092Postaddition0.24412.1635.5484.836Our modelPreaddition0.24612.7155.5395.051Postaddition0.24812.7345.5684.993Table2 presents the errors of different model configurations in predicting phoneme length. It can be seen that the optimal configuration is our model with four layers, whose RMSE was 5.19 and the sum of RMSE and cross entropy was 4.18. Besides, our model had better phoneme synthesis effect than DCNN and LSTM, a sign of superiority in the modeling of music melody sequence. Based on RMSE + cross entropy, the least mean square (LMS) of the synthetic melody can be predicted, in order to effectively reduce the RMSE of phoneme length prediction. However, the predicted phoneme length might deviate from the actual length of the music melody. It is important to apply a constraint on the phoneme length of the two melodies. Table 3 shows the errors in the phoneme length before and after applying the constraint.Table 2
Errors of different model configurations in predicting phoneme length.
RMSERMSE + cross entropyNumber of layersDCNNLSTMOur modelDCNNLSTMOur model16.355.815.825.314.824.7625.935.465.485.134.514.3435.745.235.244.864.354.2345.895.655.194.924.534.1855.825.675.814.894.594.47Table 3
Errors in the phoneme length before and after applying the constraint.
Relative to the phoneme length of music melodyRelative to the given phoneme lengthPreconstraint4.157.36Postconstraint4.070As shown in Table3, before the constraint was applied, the predicted phoneme length was inconsistent with the given value. After applying the constraint, the mean error between the predicted value and the given value dropped. This proves the reasonability of introducing the constraint on the phoneme level.Table4 presents the prediction errors of acoustic features of different model configurations on the test set. As shown in Tables 3 and 4, our model achieved a lower prediction error of the acoustic features of the test set than LSTM and DNN, during the synthesis of singing and music melodies. This means that our model can establish a good time series dependence and thus achieve an ideal synthesis effect.Table 4
Prediction errors of acoustic features of different model configurations on the test set.
ModelNumber of layersF0 correlationF0 RMSELELCDDCNN10.7341.393.956.2520.8540.722.675.1230.8938.852.534.8740.8439.032.514.8250.8239.732.544.83LSTM10.8639.463.765.1420.8537.122.644.6930.8436.802.684.2340.8936.712.634.5250.8838.623.854.34Our model10.8638.942.675.0920.8535.232.594.7230.8935.892.524.2440.8835.422.634.1250.8735.732.514.35For the proposed low-delay synthesis model of singing melody relative to music melody, it is important to evaluate the influence of the decoding consistency between singing melody and music melody on the modeling accuracy of acoustic parameters in the synthetic melody. For this purpose, a contrastive experiment was designed to compare the melody generated from natural music melody and that generated from score notes, under different lengths of historical access points (HAPs).Table5 shows the F0 values under different HAP lengths. As shown in Table 5, the F0 RMSE and F0 Pearson correlation did not change with the utilization rate of historical frames and remained independent of the type of source melody (natural melody or score notes). In addition, the HAP length had a limited influence on F0 Pearson correlation and LE.Table 5
F0 values under different HAP lengths.
HAP length1234F0 RMSENatural melody22.3421.7622.3221.35Score notes19.3819.3419.2623.48F0 correlationNatural melody0.940.950.930.99Score notes0.970.920.980.92LE2.352.342.312.39Next, the mean length of the sliding window was set to 5, 10, 15, and 20 frames in turn for the end processing module. After verification and optimization, it was found that the window of 15 frames led to the best experimental results. Table6 presents the prediction errors of different models in fundamental frequency and spectrum. Taking the melody generated from score notes for reference, the melody synthetized by our model had a lower F0 RMSE and a higher F0 Pearson correlation than that obtained by DCNN and LSTM, that is, our model can find the singing melody of better tonal consistency with music melody.Table 6
Prediction errors of different models in fundamental frequency and spectrum.
ModelDCNNLSTMOur modelF0 RMSENatural melody35.2121.7920.45Score notes34.6819.368.59F0 correlationNatural melody0.850.940.93Score notes0.860.950.98LE2.342.384.343.593.53


## 6. Conclusions

Based on the RNN algorithm, this paper probes deep into the automatic synthesis of music teaching melodies. After extracting the acoustic features from music melodies, the authors established a sequence-sequence model for synthetizing general music melodies. To find the suitable signing segments for a given music melody in the teaching scenario, an RNN was set up to synthetize music melody with singing melody. After that, the convergence of different network models was compared through experiments, which verifies the feasibility of our model. In addition, the results of different models were compared before and after adding the singing melody, and the difference of the melody generated by our model and the original music melody was quantified accurately. Furthermore, the prediction error of phoneme time of each model configuration and that after applying the time constraint were obtained through experiments. The relevant results confirm the superiority of our model over DCNN and LSTM in modeling music melody sequence.



---

*Source: 1704995-2021-12-09.xml*