# Long Short-Term Memory Projection Recurrent Neural Network Architectures for Piano’s Continuous Note Recognition

Authors: YuKang Jia; Zhicheng Wu; Yanyan Xu; Dengfeng Ke; Kaile Su
Journal: Journal of Robotics
 (2017)

---

## Abstract

Long Short-Term Memory (LSTM) is a kind of Recurrent Neural Networks (RNN) relating to time series, which has achieved good performance in speech recogniton and image recognition. Long Short-Term Memory Projection (LSTMP) is a variant of LSTM to further optimize speed and performance of LSTM by adding a projection layer. As LSTM and LSTMP have performed well in pattern recognition, in this paper, we combine them with Connectionist Temporal Classification (CTC) to study piano’s continuous note recognition for robotics. Based on the Beijing Forestry University music library, we conduct experiments to show recognition rates and numbers of iterations of LSTM with a single layer, LSTMP with a single layer, and Deep LSTM (DLSTM, LSTM with multilayers). As a result, the single layer LSTMP proves performing much better than the single layer LSTM in both time and the recognition rate; that is, LSTMP has fewer parameters and therefore reduces the training time, and, moreover, benefiting from the projection layer, LSTMP has better performance, too. The best recognition rate of LSTMP is99.8%. As for DLSTM, the recognition rate can reach 100% because of the effectiveness of the deep structure, but compared with the single layer LSTMP, DLSTM needs more training time.


---

## Body

## 1. Introduction

Piano’s continuous note recognition is important for a robot, whether it is a bionic robot, a dance robot, or a music robot. There have been companies researching on music robots. For example, Vadi produced by Vatti is able to identify a voiceprint.Most of the existing piano’s note recognition techniques use Hidden Markov Model (HMM) and Radial Basis Function (RBF) to recognize musical notes with one musical note at a time and therefore are not suitable for continuous note recognition. Fortunately, in the field of pattern recognition, Deep Neural Networks (DNNs) have shown great advantages. DNNs are used to recognize features extracted from a large number of hidden nodes [1] and seek reverse partial guidance through the chain rule and at the same time make the neural network weight matrix convergence through training data iteratively and then achieve recognition [2]. RNN adds a time series based on DNN [3], which makes features have time continuity [4, 5]. However, in experiments, we find that RNN’s time characteristics will disappear completely after four iterations [6], and a music note is generally longer than a frame [7], so RNN is not suitable for piano’s continuous note recognition [8]. Fortunately, a variant of RNN, named LSTM, is proposed [9–12], in which an input gate, an output gate, and a forgotten gate are added to memorize a long-term cell state to maintain long-term memory [8, 9, 13–16]. Furthermore, LSTMP adds a projection layer to LSTM to increase its efficiency and effectiveness.This paper studies LSTM and LSTMP for piano’s continuous note recognition, and in order to solve the temporal classification problem, we combine LSTM and LSTMP with a method named CTC [17]. In experiments, we test the performance of a single layer LSTM, Deep LSTM, and a single layer LSTMP with different parameters. Compared with the traditional piano’s note recognition methods, LSTM and LSTMP can recognize continuous notes, that is, some simple piano music. The experimental results show that a single layer LSTMP can attain a recognition rate of 99.8% and Deep LSTM can reach 100%, which proves that our methods are quite effective.The rest of this paper is organised as follows. In Section2, we first introduce the LSTM network architecture, and then Deep LSTM. LSTMP is illustrated in Section 3. In Section 4, we discuss CTC. The experimental results are presented in Section 5, and finally, in Section 6, we draw conclusions and give our future work.


## 2. LSTM

### 2.1. The LSTM Network Architecture

LSTM is a kind of RNN which succeeds to keep memory for a period of time by adding a “memory cell.” The memory cell is mainly controlled by “the input gate,” “the forgetting gate,” and “the output gate.” The input gate activates the input of information to the memory cell, and the forgetting gate selectively obliterates some information in the memory cell and activates the storage to the next input [18]. Finally, the output gate decides what information will be outputted by the memory cell [19].The LSTM network structure is illustrated in Figure1. Each box represents different data, and the lines with arrows mean data flow among these data. From Figure 1, we can understand how LSTM stores memory for a long period of time.Figure 1
The LSTM network structure.The recognition procedure of LSTM begins with a set of input sequencesx=(x1,x2,…,xt) (xi is a vector) and finally outputs a set of y=(y1,y2,…,yt) (yi is also a vector), which is calculated according to the following equations:(1)it=σWixxt+Wimmt-1+bi(2)ot=σWoxxt+Wommt-1+bo(3)ft=1-it(4)tct=gWcxxt+Wcmmt-1+bc(5)ct=ft⊙ct-1+it⊙tct(6)mt=ot⊙hct(7)yt=ϕWymmt+by.In these equations,i means the input gate, and o and f are the output gate and the forget gate, respectively. tc is the information input to the memory cell, and c includes cell activation vectors, and m is the information the memory cell outputs. W represents weight matrices (e.g., Wix represents the weight matrix from input x to the input gate i). b is the bias (bi is the input gate bias vector), and g and h are the activation function of cell inputting and cell outputting, respectively, regarded as tanh and linear in most of the models and also in this paper. ⊙ is the point multiplication in a matrix. ϕ is the activation function of the neural network output, and we use softmax in this paper.After conducting some experiments, we find that, compared with theft standard equation, (3) is more simple and easier to converge. Not only does the training time become less but also the number of iterations becomes smaller. Therefore, in the neural networks in this paper, we use (3) to calculate ft instead of the ft standard equation.


### 2.2. Deep LSTM

In piano’s continuous note recognition, we also build a multilayer neural network to further increase the recognition rate. Deep LSTM adds an LSTM after another and so on [10]. The added LSTMs have the same structure as the original one. Each layer regards the output from the last layer as the input of the next layer. We hope that the neural networks in different LSTM layers will learn different characteristics, so as to learn the various features of musical notes from different aspects and therefore improve the recognition rate.


## 2.1. The LSTM Network Architecture

LSTM is a kind of RNN which succeeds to keep memory for a period of time by adding a “memory cell.” The memory cell is mainly controlled by “the input gate,” “the forgetting gate,” and “the output gate.” The input gate activates the input of information to the memory cell, and the forgetting gate selectively obliterates some information in the memory cell and activates the storage to the next input [18]. Finally, the output gate decides what information will be outputted by the memory cell [19].The LSTM network structure is illustrated in Figure1. Each box represents different data, and the lines with arrows mean data flow among these data. From Figure 1, we can understand how LSTM stores memory for a long period of time.Figure 1
The LSTM network structure.The recognition procedure of LSTM begins with a set of input sequencesx=(x1,x2,…,xt) (xi is a vector) and finally outputs a set of y=(y1,y2,…,yt) (yi is also a vector), which is calculated according to the following equations:(1)it=σWixxt+Wimmt-1+bi(2)ot=σWoxxt+Wommt-1+bo(3)ft=1-it(4)tct=gWcxxt+Wcmmt-1+bc(5)ct=ft⊙ct-1+it⊙tct(6)mt=ot⊙hct(7)yt=ϕWymmt+by.In these equations,i means the input gate, and o and f are the output gate and the forget gate, respectively. tc is the information input to the memory cell, and c includes cell activation vectors, and m is the information the memory cell outputs. W represents weight matrices (e.g., Wix represents the weight matrix from input x to the input gate i). b is the bias (bi is the input gate bias vector), and g and h are the activation function of cell inputting and cell outputting, respectively, regarded as tanh and linear in most of the models and also in this paper. ⊙ is the point multiplication in a matrix. ϕ is the activation function of the neural network output, and we use softmax in this paper.After conducting some experiments, we find that, compared with theft standard equation, (3) is more simple and easier to converge. Not only does the training time become less but also the number of iterations becomes smaller. Therefore, in the neural networks in this paper, we use (3) to calculate ft instead of the ft standard equation.


## 2.2. Deep LSTM

In piano’s continuous note recognition, we also build a multilayer neural network to further increase the recognition rate. Deep LSTM adds an LSTM after another and so on [10]. The added LSTMs have the same structure as the original one. Each layer regards the output from the last layer as the input of the next layer. We hope that the neural networks in different LSTM layers will learn different characteristics, so as to learn the various features of musical notes from different aspects and therefore improve the recognition rate.


## 3. LSTMP-LSTM with a Projection Layer

In LSTM, there are a large number of calculations in the various gates, calculating the number of parametersN in the neural network. The weight matrix dimension input by the input gate, the output gate, and the cell state at this time is ni∗nc, and the weight matrix dimension at the last time is nc∗nc, and the output matrix dimension connected to the output of the neural network is no∗nc, where ni and no are the dimensions of the input and the output, respectively, and nc is the number of memory cells. We can easily get the following formula:(8)NLSTM=3∗ni∗nc+3∗nc∗nc+no∗nc;that is,(9)NLSTM=3∗ni∗nc+nc∗3∗nc+no.As we increasenc, NLSTM grows in a square pattern. Therefore, increasing the number of memory cells to increase the amount of memory costs a lot, but a smaller cell number will bring a lower recognition rate, so we propose an architecture named LSTMP, which can not only improve the accuracy, but also effectively reduce the computations.In the output layer of the neural network, LSTM outputs a matrix ofnc2∗mt. Then, mt is sent into the output matrix to be outputted and also serves as the input to the neural network at the next time. We add a projection layer to the LSTM architecture, and after passing this layer, mt becomes an nc∗nr matrix called rt, which replaces mt as the input of the next neural network. When the memory cell number of the neural network increases, the number of parameters in the neural network is(10)NLSTMP=3∗ni∗nc+3∗nc∗nr+no∗nr+nc∗nr;that is,(11)NLSTMP=3∗ni∗nc+nr∗4∗nc+no.CalculatingNLSTM-NLSTMP, we have(12)NLSTM-NLSTMP=nc∗3∗nc+no-nr∗4∗nc+no.Therefore, in LSTMP, the factor that affects the total number of parameters changes fromnc∗nc to nc∗nr. We can change the value of nc/nr to reduce the computational complexity. When 3∗nc>4∗nr, LSTMP can speed up the training model. Moreover, with the projection layer, LSTMP can converge faster to ensure the convergence of the model. The mathematical formulae of LSTMP are as follows:(13)it=σWixxt+Wirrt-1+biot=σWoxxt+Worrt-1+boft=1-ittct=gWcxxt+Wcrrt-1+bcct=ft⊙ct-1+it⊙tctmt=ot⊙hctrt=Wrmmtyt=ϕWyrrt+by.In these formulae,rt represents the projection layer, and the other equations are the same as LSTM.Figure2 is the structure of LSTMP, and the part marked with red dashed lines is the projection. By comparing Figure 1 with Figure 2, we can see that LSTMP is LSTM with a projection layer.Figure 2
The LSTMP structure.Algorithm1 is the pseudocode of LSTMP. w is the input weight matrix, and u is the weight matrix of the last result. b is bias and r is the projection matrix. We put the extracted musical notes features into the neural network and the algorithm executes until we get an acceptable recognition rate.Algorithm 1:LSTM-projection.
Require: inputResultlastCelllastwubr
Ensure: CellvalueResult
whileWhenhavinginputdo
tempinput=(˙Resultlast∗uinput)+(˙input∗winput)+b
tempoutput=(˙Resultlast∗uoutput)+(˙input∗woutput)+b
tempcell=(˙Resultlast∗ucell)+(˙input∗wcell)+b
inputgate=sigmoid(tempinput)
outputgate=sigmoid(tempoutput)
forgetgate=1-inputgate
Cellvalue=activate(tempcell)∗inputgate+Celllast∗forgetgate
Result=outputgate∗activate(cellvalue)
Cellvalue=(˙Result,r)
end while


## 4. CTC

The output layer of our LSTM and LSTMP is called CTC [20]. We use CTC because it does not need presegmented training data or external postprocessing to extract label sequences from the network outputs.To be the same as many latest neural networks, CTC has forward and backward algorithms. When it comes to the forward algorithm, the key point is to estimate the distribution through probabilities. Given the lengthT, the input sequence x, and the training set S at time t, the activation ytk of the output unit k at time t is interpreted as the probability of observing label k(orblankifk=L+1):(14)pπ∣x,S=∏t=1Tyπtt.We refer to the elementsπ∈L′T as paths, where L′T is the set of the length T sequences over the alphabet L′=L∪blank. Then we define a many-to-one map ϕ to remove first the repeated labels and then the blanks from the paths. With glance at the paths, will find they are mutually exclusive. According to the characteristic, the conditional probability of some labelling l∈L⩽T can be calculated by summing the probabilities of all the paths mapped onto it by ϕ:(15)pl∣x=∑π∈ϕ-1lpπ∣x.After all these procedures, CTC will complete its classification task.


## 5. Experiments

We conduct all our experiments on a server with 4 Intel Xeon E5-2620 CPUs and 512 GB memories. A NVIDIA Tesla M2070-Q graphics card is used to train all the models. The programming language we use is python 3.5.We choose the piano as our instrument. We record 445 note sequences as our dataset and the length of each sequence is around 8 seconds.In the extraction of features, we carry out Hamming window processing and then take Fast Fourier Transform (FFT) for the real part and the imaginary part of each window. Then we let the FFT result to be orthogonal by adding the square of the real part and that of the imaginary part together. Apart from that, we gain the log of the quadratic sum. Finally, the normalization of the input data is performed.In the experiments, the number of kinds of notes is 8, and the number of input nodes is 9. We try different numbers of cell units in our models, from 20 to 320. The initial value of the neural network is set as a random value within[-0.2,0.2], and the learning rate is 0.001. In terms of the structures, all the neural networks are connected to a single layer CTC. As for the dataset, we choose 80% of the samples as the development set and 20% as the test set.


### 5.1. Experimental Results

Table1 shows the recognition rates and how many times LSTM, DLSTM, and LSTMP with different parameters need to iterate until their recognition rates are stable, and the best results are in bold.Table 1
The number of iterations and the recognition rates of different models.
Model
Number of iterations
Recognition rate
LSTM-20
300
77.2%
LSTM-40
300
87.1%
LSTM-80
300
90.7%
LSTM-160
400
82.5%
DLSTM-two layers
61
85.7%
DLSTM-three layers
71
99.5%
DLSTM-four layers
109
100.0%
DLSTM-five layers
76
97.5%
DLSTM-six layers
103
100.0%
LSTMP-10 to 20
43
56%
LSTMP-30 to 20
35
94.2%
LSTMP-40 to 20
45
96.0%
LSTMP-40 to 30
30
94.0%
LSTMP-60 to 30
22
98.0%
LSTMP-80 to 20
58
99.8%
LSTMP-80 to 30
27
97.3%
LSTMP-80 to 40
29
97.3%
LSTMP-160 to 40
39
99.0%
LSTMP-160 to 80
50
95.0%
LSTMP-320 to 160
93
93.4%In Table1, “LSTMP-80 to 20” means the LSTMP model projecting 80 cell states to 20 cell states. From Table 1, we see that DLSTM and LSTMP perform much better than LSTM, and their best recognition rates are almost the same, which are 100% and 99.8%, respectively. As for the numbers of iterations, LSTMP needs much less iterations than LSTM and DLSTM, which makes LSTMP more suitable for piano’s continuous note recognition for robotics considering the efficiency.


### 5.2. LSTMP and DLSTM with Different Parameters

Figure3 illustrates LSTMP with different parameters and DLSTM with different layers. The x axis means the number of iterations and the y axis means the recognition rate. We see that for LSTMP the model projecting 80 cell states to 20 cell states has the best result, but all LSTMP results are very close. As for DLSTM, we see clearly that Deep LSTM is much better than LSTM with only one layer.Figure 3
The recognition rates of LSTMP with different parameters and DLSTM with different layers.


### 5.3. Comparisons of LSTM, LSTMP, and DLSTM

We compare LSTM, LSTMP, and DLSTM in Figure4. Given the same parameters, LSTMP performs much better than LSTM. As for LSTMP and DLSTM, we find that when the number of iterations is small, LSTMP has great advantages, but as the number of iterations increases, DLSTM becomes better.Figure 4
Comparisons of LSTM, LSTMP, and DLSTM.


## 5.1. Experimental Results

Table1 shows the recognition rates and how many times LSTM, DLSTM, and LSTMP with different parameters need to iterate until their recognition rates are stable, and the best results are in bold.Table 1
The number of iterations and the recognition rates of different models.
Model
Number of iterations
Recognition rate
LSTM-20
300
77.2%
LSTM-40
300
87.1%
LSTM-80
300
90.7%
LSTM-160
400
82.5%
DLSTM-two layers
61
85.7%
DLSTM-three layers
71
99.5%
DLSTM-four layers
109
100.0%
DLSTM-five layers
76
97.5%
DLSTM-six layers
103
100.0%
LSTMP-10 to 20
43
56%
LSTMP-30 to 20
35
94.2%
LSTMP-40 to 20
45
96.0%
LSTMP-40 to 30
30
94.0%
LSTMP-60 to 30
22
98.0%
LSTMP-80 to 20
58
99.8%
LSTMP-80 to 30
27
97.3%
LSTMP-80 to 40
29
97.3%
LSTMP-160 to 40
39
99.0%
LSTMP-160 to 80
50
95.0%
LSTMP-320 to 160
93
93.4%In Table1, “LSTMP-80 to 20” means the LSTMP model projecting 80 cell states to 20 cell states. From Table 1, we see that DLSTM and LSTMP perform much better than LSTM, and their best recognition rates are almost the same, which are 100% and 99.8%, respectively. As for the numbers of iterations, LSTMP needs much less iterations than LSTM and DLSTM, which makes LSTMP more suitable for piano’s continuous note recognition for robotics considering the efficiency.


## 5.2. LSTMP and DLSTM with Different Parameters

Figure3 illustrates LSTMP with different parameters and DLSTM with different layers. The x axis means the number of iterations and the y axis means the recognition rate. We see that for LSTMP the model projecting 80 cell states to 20 cell states has the best result, but all LSTMP results are very close. As for DLSTM, we see clearly that Deep LSTM is much better than LSTM with only one layer.Figure 3
The recognition rates of LSTMP with different parameters and DLSTM with different layers.


## 5.3. Comparisons of LSTM, LSTMP, and DLSTM

We compare LSTM, LSTMP, and DLSTM in Figure4. Given the same parameters, LSTMP performs much better than LSTM. As for LSTMP and DLSTM, we find that when the number of iterations is small, LSTMP has great advantages, but as the number of iterations increases, DLSTM becomes better.Figure 4
Comparisons of LSTM, LSTMP, and DLSTM.


## 6. Conclusions and Future Work

In this paper, we have used neural network structures called LSTM with CTC to recognize continuous musical notes. On the basis of LSTM, we have also tried LSTMP and DLSTM. Among them, LSTMP worked best when projecting80 cell states to 20 cell states, which needed much less iterations than LSTM and DLSTM, making it most suitable for piano’s continuous note recognition.In the future, we will use LSTM, LSTMP, and DLSTM to recognize more complex continuous chord music, such as piano music, violin pieces, or even symphony, which will greatly improve the development of music robots.



---

*Source: 2061827-2017-09-12.xml*