# Polyphonic Piano Transcription with a Note-Based Music Language Model
## 3 Proposed Framework
### Inference Algorithm
The \(p\)-value of 0.0472 demonstrates that the improvement of the proposed algorithm over the thresholding method is statistically significant. Figure 4 shows the transcriptions of the three post-processing methods along with the corresponding ground truth piano roll. The excerpt is a part of track bach_847 MINp_align in the LabROSA dataset. As shown in the ground truth, the polyphony at each time is two. In the results of thresholding, there are five blank onsets from 31.9 s-33.3 s. From the bottom subfigure, we can see that the proposed algorithm predicts notes at these five onsets. Although there is no blank onset in the results of GBS, some notes could not be predicted. For example, compared with the other two subfigures, there are false negative notes at the first and the last two onsets in the middle subfigure. This example demonstrates that the proposed algorithm can achieve better performance than other two post-processing methods. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Post-Processing** & **Recall** & **Precision** & **F-Measure** \\ \hline Thresholding & 0.4667 & 0.7688 & 0.5884 \\ GBS (RNN) & 0.4507 & 0.7352 & 0.5626 \\ Proposed (RNN) & 0.5368 & 0.7101 & 0.6114 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Transcription results on the real piano of LabROSA. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Post-Processing** & **Recall** & **Precision** & **F-Measure** \\ \hline Thresholding & 0.6765 & 0.8115 & 0.7374 \\ GBS (RNN) & 0.6693 & 0.7919 & 0.7225 \\ Proposed (RNN) & 0.6991 & 0.7942 & 0.7436 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Transcription results on the real piano “ENSTDkCl”. To further evaluate the performance of note-based MLMs, more transcription experiments are conducted using the proposed inference algorithm. Table 5 presents the transcription results of software piano \"StbgTGd2\". As shown in Table 5, the performance is improved slightly when we replace the RNN cells with LSTM cells in the MLMs. This is largely attributed to the fact that the LSTM could model longer term dependencies in note sequences than RNN. The RBM-based joint models outperform the single RNN or LSTM, which indicates that combining the RBM and recurrent structure as the MLM can estimate high-dimensional, temporal distributions better. Figure 4: Binary piano-roll transcription of an example track obtained through the thresholding method (**a**), the GBS algorithm (**b**) and the proposed Local beam search (LBS) algorithm (**c**). Figure 3: F-measure on the three test sets as a function of the threshold. The evaluation results on the real piano \"ENSTDkCl\" are displayed in Table 6 correspondingly. Adding RBM to the RNN or LSTM improves the MLM's performance in all respects. However, the LSTM has no superiority over RNN without the RBM. The main reason is that the acoustic model achieves a poor performance on transcribing the real piano. In this case, there are many errors in the thresholding results or history solutions. Therefore, the LSTM's advantage of longer memory does not work here. The combination of RBM and LSTM can alleviate the problem because the distribution estimator RBM has the attribution of denoising. Table 7 shows the transcription results of the LabROSA dataset. As shown in Table 6, similar results can be seen in Table 7 where the best performance is achieved by the LSTM-RBM. We also observe the differences between the results of LSTM and other MLMs. In the results of thresholding, the error rate is rather high. Therefore, the LSTM accumulates more errors than RNN and leads to the worst performance. ## 5 Conclusions

In this paper, we propose note-based MLMs for modeling note-level music structure. These note-based MLMs are trained to predict notes at the next onset, which is different from the smoothing operation of existing frame-based MLMs. An integrated architecture is used to combine the outputs of the MLM and the note-based acoustic model directly.