Skipping the Frame-Level: Event-Based Piano Transcription With Neural Semi-CRFs

1 Introduction . The task of Automatic Music Transcription ( AMT ) aims to transcribe a music recording into some form of music notation [ Benetos et al. , 2018 ] . Examples of notation include MIDI event sequences , e.g , Hawthorne et al . [ 2018 ] , Kong et al . [ 2020 ] , Kim and Bello [ 2019 ] , Kwon et al . [ 2020 ] , and staff notation , e.g. , Nakamura et al . [ 2018 ] , Román et al. , 2019 ] . In this work , we address the problem of transcribing piano music into a MIDI event sequence . MIDI transcription involves constructing a sequence of events , each specified by its onset and offset positions , with the constraint that two events of the same event type ( e.g. , certain pitches and pedals ) can not overlap . In addition to onsets and offsets , the velocity ( i.e. , a value that represents the intensity of a key strike , which informs the loudness ) associated with each event is often estimated . In recent years , neural network based approaches have reached the state of the art for the problem of piano transcription , e.g. , Hawthorne et al . [ 2018 ] , Kong et al . [ 2020 ] , Kwon et al . [ 2020 ] . They operate at the frame-level and make predictions for different stages of a note event , i.e. , the onset , offset , and pitch activation , separately . In order to extract note-level predictions , they use manually designed procedures to combine the disjoint frame-level predictions . These include thresholding , ∗Code is available at https : //github.com/Yujia-Yan/Skipping-The-Frame-Level 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . peak picking [ Hawthorne et al. , 2018 , Kong et al. , 2020 ] or low-order hidden Markov models [ Kwon et al. , 2020 ] . This two-stage approach requires manually crafted procedures and a manually designed state structure modeling the temporal evolution of notes . In this work , we propose a direct approach to note-level transcription . Instead of scoring and aggregating note activations across individual time frames , our approach directly scores a time interval ’ s likelihood of covering the entire process of a certain event ( e.g. , a note , pedal usage ) . This greatly simplifies the formulation of piano transcription . Specifically , the proposed method takes the logmel spectrogram of an audio segment ( e.g. , 10s ) as input , and uses a contextual model to produce contextual embeddings for each frame . It then uses a score model to score each possible time interval within the segment to assess whether it covers the entire span of a musical event ( e.g. , notes , pedals ) as specified by a specialized zeroth-order semi-Markov conditional random fields ( semi-CRF ) [ Sarawagi and Cohen , 2004 ] . These scores are then decoded using Viterbi algorithm into a set of non-overlapping intervals for each event type , i.e. , note-level transcription . Finally , other attributes of these intervals ( i.e. , velocity and refined boundary positions ) are also estimated . Previous works using semi-CRF were concerned with shorter sequence lengths and smaller interval lengths [ Kong et al. , 2016 , Liu et al. , 2016 , Kemos et al. , 2019 , Lu et al. , 2016 ] , and usually only attempted to estimate a single channel ( track ) of non-overlapping events . In contrast , our problem deals with sequences of 400-1600 frames ( approximately 10s-40s ) and requires a separate semiCRF for around 90 different labelling channels ( 88 pitches + 1-3 pedals ) , each corresponding to a specific event type . Another challenge is the wide range of event duration that can be encountered in piano music , which further increases the computational expenses of the problem . These challenges prevented us from using existing general purpose semi-CRF methods due to computational expenses . Instead , we introduce a zeroth-order Semi-CRF formulation that is specifically adapted to the problem and show that it can be implemented efficiently ( see Table 3 ) for the problem size of interest . Our experiments on the MAESTRO dataset show that the proposed system achieves Note w/ Offset F1 of 88.72 % and Note w/ Offset & Velocity F1 of 87.75 % , exceeding the current state of the art while being smaller and faster . We believe that this simple , fast and well-performing approach is also extensible to other similar tasks with intervals as the prediction target , such as polyphonic sound event detection or speaker diarization . 2 Related Works . Piano Transcription There have been many proposed approaches to piano transcription in the last several decades . Early works consist of simple signal processing methods such as spectral peakpicking [ Klapuri et al. , 2000 , Bello et al. , 2006 ] , which estimates fundamental frequencies directly from the spectrum , or spectral decomposition [ Smaragdis and Brown , 2003 , O ’ Hanlon and Plumbley , 2014 ] , where a short-time spectrum is factorized into spectral components with corresponding activations . Several parametric models have also been proposed to perform spectro-temporal decomposition [ Vincent et al. , 2009 , Emiya et al. , 2009 , Cheng et al. , 2016 ] . Time-domain decomposition has been also investigated in Cogliati et al . [ 2017 , 2016 ] . Most of these methods aim to model notes explicitly as parametric templates , and tend to suffer from the lack of generalization . Other methods include machine learning techniques such as Hidden Markov Models [ Raphael , 2002 , Böck and Schedl , 2012 ] or Support Vector Machines [ Poliner and Ellis , 2006 , Weninger et al. , 2013 ] . Recently , neural network based approaches have made significant advances [ Sigtia et al. , 2016 , Hawthorne et al. , 2018 , 2019 , Kim and Bello , 2019 ] by taking advantage of large Disklavier piano databases . Many methods separate piano transcription into frame-wise polyphonic pitch estimation and note tracking , thereby optimizing for frame-level predictions which are used to estimate quantized note intervals . While intuitive , this practice has a major drawback : the quality of the note-level predictions is fundamentally limited by the quality of the frame-level activations , which can be sporadic , discontinuous , and non-uniform in terms of their strength . Kameoka et al . [ 2007 ] attempted to circumvent this problem by performing harmonic temporal template structured clustering to explain spectrogram observations as originating from distinct sources , directly estimating notes as discrete events . This reflects the idea of [ Bregman , 1990 ] , i.e. , that meaningful auditory events are results of simultaneous and sequential groupings . However , the underlying mental process of low-level groupings are more complex than clustering according to simple parametric harmonic temporal templates . Some more recent works [ Kelz et al. , 2019 , Kwon et al. , 2020 ] attempt to incorporate manuallydefined state structure for modeling the temporal evolution of notes . However , the state structure ( usually of low order ) may not reflect the actual temporal evolution and thus limits the representative power of these models . Cogliati et al . [ 2017 ] directly estimates the activation of note templates in the time domain , but these templates are preset and are piano- and context-dependent , lacking the generalization to unseen pianos or recording environments . The proposed method learns to estimate note events holistically , rather than adopting the frame-tonote two-stage approach or performing and aggregating multiple subtasks ( i.e. , onset , offset , and pitch estimation ) . It can be viewed as performing groupings directly in the event space , with lowlevel details being learned by the neural networks . Semi-Markov CRFs A semi-Markov conditional random field ( semi-CRF , Sarawagi and Cohen [ 2004 ] ) defines a conditional probability distribution over sets of non-overlapping labeled intervals within an input sequence . Semi-CRFs have been used in Chinese word segmentation [ Liu et al. , 2016 , Kong et al. , 2016 ] , named entity recognition [ Zhuo et al. , 2016 , Ye and Ling , 2018 , Arora et al. , 2019 ] , character-level part-of-speech tagging [ Kemos et al. , 2019 ] , phone recognition [ Lu et al. , 2016 ] , chord recognition [ Masada and Bunescu , 2017 ] , etc . The time complexity for computing the partition function and inferring the most likely configuration is quadratic with respect to the length of the sequence . Therefore , most works set an upper bound on the length of a single interval , e.g. , Kemos et al . [ 2019 ] , to make the computation tractable . In this work , a semi-CRF is defined for each event type ( i.e. , a note of a specific pitch , a pedal ) . Our semi-CRF formulation differs from the standard formulation in the following ways : 1 ) Different events in a semi-CRF are allowed to overlap on their endpoints ( boundary frames ) , since audio frames represent a time period and can contain both the end and beginning of an event of the same type . This treatment allows us to use a larger hop size without adjustment due to overlap in the same frame , so that the sequence length for the same audio segment can be reduced ; 2 ) A frame is not required to belong to any event . Despite the loosened definition , we use the name semi-CRF to refer to this type of structured prediction module . Compared to the tasks mentioned above , the task of piano transcription has a longer input sequence and a larger range of possible event duration . The former makes the time complexity issue more prominent , while the latter makes it difficult to specify an upper bound on event duration for speedup purposes . We show that our formulation is efficient for piano transcription and hence does not require an upper bound on event duration , thanks to a set of implementation optimizations . 3 Proposed Semi-CRF Approach to Piano Transcription . The proposed system transcribes the input audio into a list of musical events , i.e. , notes and pedals . Here the term events refers to acoustical events that span certain time intervals ( not to be confused with raw MIDI events , i.e. , note on , note off , cc , etc. , used for serializing the performance ) . Taking an audio segment ( e.g. , 10s ) , the transcription process is illustrated in Figure 1 . First , a log-mel spectrogram is computed after taking the short-time Fourier transform ( STFT ) of the audio . This is used as input to a contextual model to produce contextual features across time frames . Such features are then fed to a score model to calculate two kinds of scores . The first kind of score ( Score ) assigns a score to indicate whether an interval is an event of a certain event type . The second kind of score ( Scoreϵ ) indicates whether an interval that spans two frames is not part of any event for a certain event type . Finally , a Viterbi algorithm is used to decode the most likely sequence of events for each event type , using the aforementioned scores . Attributes , i.e. , velocity and refined onset/offset positions , are then estimated for each extracted event . 3.1 CRF Formulation . Let X = < x0 , x1 , . . . , xN−1 > be an audio segment containing a sequence of N time frames . Let Y = { ( i , j , eventType ) , i ≤ j } be the set of musical events entirely contained within this segment , with time quantized to audio frames . Here i and j are respectively the beginning and ending frame indices for each event , and eventType is the type of the event , e.g. , a specific key ( pitch ) of the the 88 keys of a piano or the sustain pedal . Events that extend outside the audio segment are not considered in this formulation , but will be handled in the inference process ( See Section 3.3 ) . We assume that for the same event type , two events A and B are non-overlapping , i.e. , either jA ≤ iB or jB ≤ iA . In this formulation , it is allowed to have single-frame events where i = j ( note how this would make the formulation differ from the standard semi-CRFs ) . We associate each event with a set of attributes : 1 ) the relative non-quantized position of onsets and offsets at the sub-frame level , represented by a value from −0.5 to 0.5 indicating the position relative to the quantized index of the frame , and 2 ) the MIDI velocity of the event , represented by a discrete value from 0 to 127 . We use YeventType to denote the subset of events that contains only a specific event type . For each event type , we model the following conditional probability : p ( YeventType|X ) = 1 Z ( eventType ) exp [ ∑ ( i , j , eventType ) ∈YeventType score ( i , j , eventType ) + ∑ [ i−1 , i ] not covered in YeventType scoreϵ ( i− 1 , i , eventType ) ] , ( 1 ) where score ( i , j , eventType ) assigns a score to the interval [ i , j ] to indicate whether it is an event of eventType , scoreϵ ( i − 1 , i , eventType ) assigns a score to the interval [ i − 1 , i ] that spans two frames to indicate whether it is not covered by any event of eventType , which inversely represents the frame-level activation for an event , and Z ( eventType ) is the normalization factor . The non-event score ( scoreϵ ) for two consecutive frames comes from skipping one position in a feasible solution ( see example in the next paragraph ) , and it also allows the model to degenerate to frame-level event prediction . Here , for notational convenience , we omit X for every term . For numerical stability , Eqn . ( 1 ) is computed in the log-domain . The exponent , i.e. , the summation of all score ( i , j , eventType ) and scoreϵ ( i− 1 , i , eventType ) corresponding to YeventType , is the unnormalized log-likelihood . As an example , for a single eventType , assuming there are 7 audio frames < 0,1,2,3,4,5,6 > in total , for the interval set candidate { [ 0,0 ] , [ 2,4 ] , [ 4,5 ] } , the corresponding unnormalized log-likelihood is computed as score ( 0 , 0 ) + scoreϵ ( 0 , 1 ) + scoreϵ ( 1 , 2 ) + score ( 2 , 4 ) + score ( 4 , 5 ) + scoreϵ ( 5 , 6 ) . The computation of logZ ( eventType ) and its gradient w.r.t . to score/scoreϵ , ∇ logZ ( eventType ) , is critical in training . The detailed procedure is shown in Algorithm 1 ( forward-backward algorithm ) . In practice , we compute logZ and ∇ logZ for all eventType ( s ) in parallel . The forward stage and the backward stage in the forward-backward algorithm are batched together , but with all positions of the input flipped , as to compute them in a single pass without the need for two separate update equations , since their calculations are essentially the same . We also find that a substantial speedup ( > 5x ) can be achieved by using a custom gradient computation via the backward pass of the forwardbackward algorithm for logZ , as opposed to using automatic differentiation w.r.t . logZ in PyTorch . This is likely due to a more contiguous memory access pattern and the elimination of unnecessary operations . In order to make memory access at each step more contiguous , the event scores Score and Scoreϵ are organized as Tend× Tstart×N and Tend×N tensors , respectively , with Tend being the ending position of an interval , Tstart being the beginning position of an interval , and N being the number of eventType ( s ) . Here , each slice along the N dimension represents a separate semi-CRF channel . A runtime benchmark of the semi-CRF layer presented is provided in Table 3 . Algorithm 1 Forward-backward algorithm for logZ and ∇ logZ for a specific event type . Input : function score ( i , j ) , function scoreϵ ( i− 1 , i ) Output : logZ and ∇ logZ Forward stage : Initialize the forward variable : v ( 0 ) ← log ( exp ( score ( 0 , 0 ) ) + 1 ) for all j = 1 , . . . , N − 1 do v ( j ) ← log { exp [ v ( j − 1 ) + scoreϵ ( j − 1 , j ) ] + ∑ k < j exp [ v ( k ) + score ( k , j ) ] } v ( j ) ←v ( j ) + log { 1 + exp [ score ( j , j ) ] } end for Readout the log partition function : logZ ← v ( N − 1 ) Backward stage : Initialize the backward variable : q ( N − 1 ) ← log { exp [ score ( N − 1 , N − 1 ) ] + 1 } for all j = N − 2 , . . . , 0 do q ( j ) ← log { exp [ q ( j + 1 ) + scoreϵ ( j , j + 1 ) ] + ∑ k > j exp [ q ( k ) + score ( j , k ) ] } q ( j ) ←q ( j ) + log { 1 + exp [ score ( j , j ) ] } end for Read out the posterior marginals as derivatives : for all i=0 , . . . , N-1 do p ( i , i ) ← exp { v ( i ) + q ( i ) + score ( i , i ) − 2 log [ exp ( score ( i , i ) + 1 ) ] − logZ } end for for all i < j do p ( i , j ) ← exp [ v ( i ) + q ( j ) + score ( i , j ) − logZ ] end for for all i=1 , . . . , N-1 do pϵ ( i− 1 , i ) ← exp [ v ( i− 1 ) + q ( i ) + scoreϵ ( i− 1 , i ) − logZ ] end for ∂ logZ ∂score ( i , j ) = p ( i , j ) , ∂ logZ ∂scoreϵ ( i−1 , i ) = pϵ ( i− 1 , i )

Summary: The authors take a well studied network architecture and task (MAESTRO piano transcription) and propose a straightforward extension of previous work to incorporate a CRF output layer that can directly predict temporal dependencies among the output labels. This contrasts with previous SOTA techniques that use multiple output heads (for separate parts of notes) and heuristics to infer the notes from network outputs. The CRF loss function is used to train a network similar to previous work, and shows improvements in classification accuracy, most notably in Onset+Offset F1 score.