# Polyphonic Piano Transcription with a Note-Based Music Language Model
## 3 Proposed Framework
### Inference Algorithm
To evaluate the proposed system, we also use the whole LabROSA piano transcription dataset as another test set [22]. There are 29 pieces of music in this database, along with aligned MIDI files. The MIDI data are collected from Piano-mid.de, and piano recordings are made using a Yamaha Disklavier playback grand piano. ### Experimental Settings

The acoustic model takes the spectrograms of CQT as input. The audio signal is segmented with a frame length of 100 ms and a hop size of 10 ms. A context window of nine frames is applied to the 267 dimensional CQTs so that we could obtain a spectrogram slice. The two CNNs have the same structure, except for the output layer. The model configurations for the CNNs are presented in Table 1. For the spectrogram slices of \(267\times 9\), the first convolutional layer with 10 filters of size \(16\times 2\) computes 10 feature maps of size \(252\times 8\). The next layer performs max-pooling of \(2\times 2\), reducing the size of maps to \(126\times 4\). The second convolutional layer contains 20 filters of size \(11\times 3\), and the max-pooling size of the second pooling layer is also set to \(2\times 2\). The fully-connected layer contains 256 units, and the number of units in the output layer changes with the task. In the CNN for onset detection, the output layer has a single unit. In the CNN for multi-pitch estimation, the output layer has 88 units and employs the sigmoid as the activation function to yield 88 independent pitch probabilities. The CNNs were trained using mini-batch gradient descent with size 256. The [PERSON] algorithm was used in the training [32]. An initial learning rate of 0.01 was decreased to zero over 100 epochs. To prevent over-fitting, a dropout of 0.5 was applied to each network. We also used the method of early stopping, in which training was stopped if the cost (cross entropy) did not decrease for 20 epochs. As mentioned in Section 2, we take the RNN, RNN-RBM and their LSTM variants as MLMs. Both the RNN and LSTM have one single hidden layer, which contains 100 hidden nodes. In the RNN-RBM or LSTM-RBM, the number of recurrent hidden nodes is also 100, and the RBM has 50 hidden units. The training pieces are divided into sub-sequences of length 20. All these MLMs \begin{table}
\begin{tabular}{c c c} \hline \hline
**Type** & **Patch Size/Stride** & **Input Size** \\ \hline Conv 1 & \(16\times 2/1\) & \(267\times 9\) \\ Pool 1 & \(2\times 2/2\) & \(252\times 8\times 10\) \\ Conv 2 & \(11\times 3/1\) & \(126\times 4\times 10\) \\ Pool 2 & \(2\times 2/2\) & \(116\times 2\times 20\) \\ Fully-connected & 256 & \(58\times 1\times 20\) \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Model configuration for the CNNs. were trained using the note sequences by back-propagation through time (BPTT). We used mini-batch of size 100 and the Adam algorithm for gradient updating. The initial learning rate was set to 0.01, which was linearly reduced to zero over 100 iterations. In addition to dropout, we also adopted early stopping to prevent over-fitting. Note-based metrics are employed to assess the performance of the proposed system. A note event is regarded as right if its pitch is correct and its onset is within a \(\pm\)50 ms range of the ground truth onset. These measures are defined as:

\[P =\frac{N_{TP}}{N_{TP}+N_{FP}} \tag{17}\] \[R =\frac{N_{TP}}{N_{TP}+N_{FN}}\] (18) \[F =\frac{2*P*R}{P+R} \tag{19}\]

where \(P\), \(R\), \(F\) correspond to the precision, recall and F-measure, respectively, and \(N_{TP}\), \(N_{FP}\) and \(N_{FN}\) are the numbers of true positives, false positives and false negatives, respectively. ### Results

The transcription experiments are performed with various configurations. The CNN-based acoustic model yields a sequence of probabilities for 88 individual pitches, and various post-processing methods are used to transform the probabilities into binary notations. The first method is simplest, which applies a threshold to the acoustic outputs.