## 2 Proposed Framework
### Note Recognition  
Recently, convolutional learning has achieved great success in music signal processing, such as genre classification [34], artist classification [35], and chord detection [36]. In the task of AMT, CNNs have also been evaluated for onset detection and frame-based transcription, respectively. In the experiments of onset detection, [PERSON] used CNNs of different architectures [37]. The results shows that a CNN with linear rectifier outperformed the state-of-the-art while requiring less manual preprocessing. [PERSON] utilized a CNN to transcribe polyphonic piano music frame-by-frame, and the output was estimated pitches at each frame [21]. Although CNN yields the best performance on the frame-based metrics, an NMF method outperforms CNN on note-based metrics. So, it is promising for CNN to make use of the note onset and generate a note-based transcription. Here we train a CNN to detect onset and another CNN to estimate pitches at each detected onset. CNNs are neural networks characterized by a convolutional structure. The convolutional layers are designed to preserve the spatial structure of the input. In each convolutional layer, a set of weights act on a fixed-size local region of the input. These weights are then repeatedly applied to the entire input to produce a feature map. After the convolution of input with shared weights, the output of the convolutional layer is obtained by adding a bias term and then applying a non-linear function. Each unit of out feature map in the convolutional layer can be computed as:  
\[q_{j,m}=f(\sum_{i}^{I}\sum_{n}^{N}o_{i,n+m-1}w_{i,j,n}+b_{j}) \tag{1}\]  
where \(o_{i,m}\) is the \(m\)th unit of the \(i\)th input feature map, \(q_{j,m}\) is the \(m\)th unit of the \(j\)th output feature map, \(w_{i,j,n}\) is the \(n\)th element of the weight vector, \(b_{j}\) is the bias term added to the \(j\)th feature map, \(f(\cdot)\) is the activation function. \(I\) is the number of input feature maps, and \(N\) is the size of weight filter. A convolutional layer is often followed by a pooling layer, which subsamples each feature map. For example, the most common max pooling only retains the maximum value in non-overlapping cells. When the max pooling function is used, the pooling layer is defined as:  
\[p_{j,m}=\max_{k=1}^{K}q_{j,(m-1)\times s+k} \tag{2}\]  
where \(K\) is the pooling size and \(s\) is the shift size of pooling windows. Here, \(p_{j,m}\) is the \(m\)th unit of the \(j\)th output feature map. \(q_{j,m}\) is the \(m\)th unit of the \(j\)th input feature map in this pooling layer, and it is also the corresponding unit of the output feature map in the last convolutional layer. Finally, the CNN ends in fully-connected layers that integrate the information of layers below. In audio signal processing, the input to the CNN is a window of feature frames centering around time \(t\), whereas the output contains posterior probabilities of different categories at time \(t\). There are several motivations for applying CNNs to music transcription. Firstly, aggregating over several frames achieves better performance than processing a single frame. For example, the attack stage of notes can be modeled by applying a context window around the onset so that the onset will be  
Figure 2: Diagram for the proposed framework. CNN: convolutional neural network; NMF: non-negative matrix factorization. detected more accurately. Secondly, the architecture of the CNN can learn features along both the time and frequency axes. CNN is proper for processing the harmonic structure in a spectrogram because of its shift invariance. Compared with deep neural network (DNN) and RNN, the weight sharing and pooling architecture leads to a reduction of parameters. In the proposed note recognition stage, two CNNs are trained using a constant Q transform (CQT) of the music signal. The spectrogram of CQT is suited as time-frequency representation for music since its frequency bins are evenly spaced on a logarithmic axis.