# Polyphonic Piano Transcription with a Note-Based Music Language Model
## 3 Proposed Framework
### Inference Algorithm
We also propose an inference algorithm, which uses the note-based MLM to predict notes at the blank onsets in the thresholding transcription results. The experiments are conducted on the MAPS and LabROSA databases. Although the proposed algorithm only achieves an absolute 0.34% F-measure improvement on the synthetic data, it reaches absolute 0.77% and 2.39% improvements on two real piano test sets, respectively. We also observe that the combination of RBM and recurrent structure models the high-dimensional sequences better than a single RNN or LSTM does. Although the LSTM shows no superiority to other MLMs in transcribing the real piano, the LSTM-RBM always helps the system yield the best results regardless of the performance of acoustic models. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**MLM** & **Recall** & **Precision** & **F-Measure** \\ \hline RNN & 0.6991 & 0.7942 & 0.7436 \\ RNN-RBM & 0.6999 & 0.7952 & 0.7445 \\ LSTM & 0.6991 & 0.7940 & 0.7435 \\ LSTM-RBM & 0.7009 & 0.7952 & 0.7451 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Results for MLMs on the real piano “ENSTDkCl”. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**MLM** & **Recall** & **Precision** & **F-Measure** \\ \hline RNN & 0.5368 & 0.7101 & 0.6114 \\ RNN-RBM & 0.5371 & 0.7105 & 0.6117 \\ LSTM & 0.5339 & 0.7063 & 0.6082 \\ LSTM-RBM & 0.5377 & 0.7108 & 0.6123 \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Results for MLMs on the real piano of LabROSA. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**MLM** & **Recall** & **Precision** & **F-Measure** \\ \hline RNN & 0.8946 & 0.9111 & 0.9027 \\ RNN-RBM & 0.8954 & 0.9112 & 0.9032 \\ LSTM & 0.8948 & 0.9114 & 0.9030 \\ LSTM-RBM & 0.8960 & 0.9112 & 0.9035 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Results for MLMs on the software piano “StbgTGd2”. MLM, music language model. Overall, the improvement of the proposed algorithm over the thresholding method is small. One of the possible reasons is the limited training data. The MLMs are trained using only 161 pieces in the MAPS database, and the small amount of data may lead the neural networks to over-fitting. The abundance of musical scores can provide a way to solve the problem. Besides, the note sequences are indexed using the onset in the current system. Actually, the temporal structure of musical sequences should contain how the notes appear and last correlatively. Ignoring the note's offset or duration time, the representation of musical sequences is partial. Therefore, the MLMs in this paper cannot model the temporal structure of note sequences completely. In the future, we will search for a proper way to represent the note-level musical sequences. One possible solution is to add a duration model to the current MLMs, such as an HMM. This work is partially supported by the National Key Research and Development Plan (Nos. 2016 YFB0801203, 2016 YFB0801200), the National Natural Science Foundation of China (Nos. 11590770-4, U1536117, 11504406, 11461141004) and the Key Science and Technology Project of the Xinjiang Uygur Autonomous Region (No. 2016A03007-1). [PERSON] and [PERSON] conceived of and designed the experiments. [PERSON] performed the experiments and analyzed the data. [PERSON] contributed analysis tools. [PERSON] wrote the paper. The authors declare no conflict of interest. ## Appendix A