Skipping the Frame-Level: Event-Based Piano Transcription With Neural Semi-CRFs

1 Introduction . The task of Automatic Music Transcription ( AMT ) aims to transcribe a music recording into some form of music notation [ Benetos et al. , 2018 ] . Examples of notation include MIDI event sequences , e.g , Hawthorne et al . [ 2018 ] , Kong et al . [ 2020 ] , Kim and Bello [ 2019 ] , Kwon et al . [ 2020 ] , and staff notation , e.g. , Nakamura et al . [ 2018 ] , Román et al. , 2019 ] . In this work , we address the problem of transcribing piano music into a MIDI event sequence . MIDI transcription involves constructing a sequence of events , each specified by its onset and offset positions , with the constraint that two events of the same event type ( e.g. , certain pitches and pedals ) can not overlap . In addition to onsets and offsets , the velocity ( i.e. , a value that represents the intensity of a key strike , which informs the loudness ) associated with each event is often estimated . In recent years , neural network based approaches have reached the state of the art for the problem of piano transcription , e.g. , Hawthorne et al . [ 2018 ] , Kong et al . [ 2020 ] , Kwon et al . [ 2020 ] . They operate at the frame-level and make predictions for different stages of a note event , i.e. , the onset , offset , and pitch activation , separately . In order to extract note-level predictions , they use manually designed procedures to combine the disjoint frame-level predictions . These include thresholding , ∗Code is available at https : //github.com/Yujia-Yan/Skipping-The-Frame-Level 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . peak picking [ Hawthorne et al. , 2018 , Kong et al. , 2020 ] or low-order hidden Markov models [ Kwon et al. , 2020 ] . This two-stage approach requires manually crafted procedures and a manually designed state structure modeling the temporal evolution of notes . In this work , we propose a direct approach to note-level transcription . Instead of scoring and aggregating note activations across individual time frames , our approach directly scores a time interval ’ s likelihood of covering the entire process of a certain event ( e.g. , a note , pedal usage ) . This greatly simplifies the formulation of piano transcription . Specifically , the proposed method takes the logmel spectrogram of an audio segment ( e.g. , 10s ) as input , and uses a contextual model to produce contextual embeddings for each frame . It then uses a score model to score each possible time interval within the segment to assess whether it covers the entire span of a musical event ( e.g. , notes , pedals ) as specified by a specialized zeroth-order semi-Markov conditional random fields ( semi-CRF ) [ Sarawagi and Cohen , 2004 ] . These scores are then decoded using Viterbi algorithm into a set of non-overlapping intervals for each event type , i.e. , note-level transcription . Finally , other attributes of these intervals ( i.e. , velocity and refined boundary positions ) are also estimated . Previous works using semi-CRF were concerned with shorter sequence lengths and smaller interval lengths [ Kong et al. , 2016 , Liu et al. , 2016 , Kemos et al. , 2019 , Lu et al. , 2016 ] , and usually only attempted to estimate a single channel ( track ) of non-overlapping events . In contrast , our problem deals with sequences of 400-1600 frames ( approximately 10s-40s ) and requires a separate semiCRF for around 90 different labelling channels ( 88 pitches + 1-3 pedals ) , each corresponding to a specific event type . Another challenge is the wide range of event duration that can be encountered in piano music , which further increases the computational expenses of the problem . These challenges prevented us from using existing general purpose semi-CRF methods due to computational expenses . Instead , we introduce a zeroth-order Semi-CRF formulation that is specifically adapted to the problem and show that it can be implemented efficiently ( see Table 3 ) for the problem size of interest . Our experiments on the MAESTRO dataset show that the proposed system achieves Note w/ Offset F1 of 88.72 % and Note w/ Offset & Velocity F1 of 87.75 % , exceeding the current state of the art while being smaller and faster . We believe that this simple , fast and well-performing approach is also extensible to other similar tasks with intervals as the prediction target , such as polyphonic sound event detection or speaker diarization . 2 Related Works . Piano Transcription There have been many proposed approaches to piano transcription in the last several decades . Early works consist of simple signal processing methods such as spectral peakpicking [ Klapuri et al. , 2000 , Bello et al. , 2006 ] , which estimates fundamental frequencies directly from the spectrum , or spectral decomposition [ Smaragdis and Brown , 2003 , O ’ Hanlon and Plumbley , 2014 ] , where a short-time spectrum is factorized into spectral components with corresponding activations . Several parametric models have also been proposed to perform spectro-temporal decomposition [ Vincent et al. , 2009 , Emiya et al. , 2009 , Cheng et al. , 2016 ] . Time-domain decomposition has been also investigated in Cogliati et al . [ 2017 , 2016 ] . Most of these methods aim to model notes explicitly as parametric templates , and tend to suffer from the lack of generalization . Other methods include machine learning techniques such as Hidden Markov Models [ Raphael , 2002 , Böck and Schedl , 2012 ] or Support Vector Machines [ Poliner and Ellis , 2006 , Weninger et al. , 2013 ] . Recently , neural network based approaches have made significant advances [ Sigtia et al. , 2016 , Hawthorne et al. , 2018 , 2019 , Kim and Bello , 2019 ] by taking advantage of large Disklavier piano databases . Many methods separate piano transcription into frame-wise polyphonic pitch estimation and note tracking , thereby optimizing for frame-level predictions which are used to estimate quantized note intervals . While intuitive , this practice has a major drawback : the quality of the note-level predictions is fundamentally limited by the quality of the frame-level activations , which can be sporadic , discontinuous , and non-uniform in terms of their strength . Kameoka et al . [ 2007 ] attempted to circumvent this problem by performing harmonic temporal template structured clustering to explain spectrogram observations as originating from distinct sources , directly estimating notes as discrete events . This reflects the idea of [ Bregman , 1990 ] , i.e. , that meaningful auditory events are results of simultaneous and sequential groupings . However , the underlying mental process of low-level groupings are more complex than clustering according to simple parametric harmonic temporal templates . Some more recent works [ Kelz et al. , 2019 , Kwon et al. , 2020 ] attempt to incorporate manuallydefined state structure for modeling the temporal evolution of notes . However , the state structure ( usually of low order ) may not reflect the actual temporal evolution and thus limits the representative power of these models . Cogliati et al . [ 2017 ] directly estimates the activation of note templates in the time domain , but these templates are preset and are piano- and context-dependent , lacking the generalization to unseen pianos or recording environments . The proposed method learns to estimate note events holistically , rather than adopting the frame-tonote two-stage approach or performing and aggregating multiple subtasks ( i.e. , onset , offset , and pitch estimation ) . It can be viewed as performing groupings directly in the event space , with lowlevel details being learned by the neural networks . Semi-Markov CRFs A semi-Markov conditional random field ( semi-CRF , Sarawagi and Cohen [ 2004 ] ) defines a conditional probability distribution over sets of non-overlapping labeled intervals within an input sequence . Semi-CRFs have been used in Chinese word segmentation [ Liu et al. , 2016 , Kong et al. , 2016 ] , named entity recognition [ Zhuo et al. , 2016 , Ye and Ling , 2018 , Arora et al. , 2019 ] , character-level part-of-speech tagging [ Kemos et al. , 2019 ] , phone recognition [ Lu et al. , 2016 ] , chord recognition [ Masada and Bunescu , 2017 ] , etc . The time complexity for computing the partition function and inferring the most likely configuration is quadratic with respect to the length of the sequence . Therefore , most works set an upper bound on the length of a single interval , e.g. , Kemos et al . [ 2019 ] , to make the computation tractable . In this work , a semi-CRF is defined for each event type ( i.e. , a note of a specific pitch , a pedal ) . Our semi-CRF formulation differs from the standard formulation in the following ways : 1 ) Different events in a semi-CRF are allowed to overlap on their endpoints ( boundary frames ) , since audio frames represent a time period and can contain both the end and beginning of an event of the same type . This treatment allows us to use a larger hop size without adjustment due to overlap in the same frame , so that the sequence length for the same audio segment can be reduced ; 2 ) A frame is not required to belong to any event . Despite the loosened definition , we use the name semi-CRF to refer to this type of structured prediction module . Compared to the tasks mentioned above , the task of piano transcription has a longer input sequence and a larger range of possible event duration . The former makes the time complexity issue more prominent , while the latter makes it difficult to specify an upper bound on event duration for speedup purposes . We show that our formulation is efficient for piano transcription and hence does not require an upper bound on event duration , thanks to a set of implementation optimizations . 3 Proposed Semi-CRF Approach to Piano Transcription . The proposed system transcribes the input audio into a list of musical events , i.e. , notes and pedals . Here the term events refers to acoustical events that span certain time intervals ( not to be confused with raw MIDI events , i.e. , note on , note off , cc , etc. , used for serializing the performance ) . Taking an audio segment ( e.g. , 10s ) , the transcription process is illustrated in Figure 1 . First , a log-mel spectrogram is computed after taking the short-time Fourier transform ( STFT ) of the audio . This is used as input to a contextual model to produce contextual features across time frames . Such features are then fed to a score model to calculate two kinds of scores . The first kind of score ( Score ) assigns a score to indicate whether an interval is an event of a certain event type . The second kind of score ( Scoreϵ ) indicates whether an interval that spans two frames is not part of any event for a certain event type . Finally , a Viterbi algorithm is used to decode the most likely sequence of events for each event type , using the aforementioned scores . Attributes , i.e. , velocity and refined onset/offset positions , are then estimated for each extracted event . 3.1 CRF Formulation . Let X = < x0 , x1 , . . . , xN−1 > be an audio segment containing a sequence of N time frames . Let Y = { ( i , j , eventType ) , i ≤ j } be the set of musical events entirely contained within this segment , with time quantized to audio frames . Here i and j are respectively the beginning and ending frame indices for each event , and eventType is the type of the event , e.g. , a specific key ( pitch ) of the the 88 keys of a piano or the sustain pedal . Events that extend outside the audio segment are not considered in this formulation , but will be handled in the inference process ( See Section 3.3 ) . We assume that for the same event type , two events A and B are non-overlapping , i.e. , either jA ≤ iB or jB ≤ iA . In this formulation , it is allowed to have single-frame events where i = j ( note how this would make the formulation differ from the standard semi-CRFs ) . We associate each event with a set of attributes : 1 ) the relative non-quantized position of onsets and offsets at the sub-frame level , represented by a value from −0.5 to 0.5 indicating the position relative to the quantized index of the frame , and 2 ) the MIDI velocity of the event , represented by a discrete value from 0 to 127 . We use YeventType to denote the subset of events that contains only a specific event type . For each event type , we model the following conditional probability : p ( YeventType|X ) = 1 Z ( eventType ) exp [ ∑ ( i , j , eventType ) ∈YeventType score ( i , j , eventType ) + ∑ [ i−1 , i ] not covered in YeventType scoreϵ ( i− 1 , i , eventType ) ] , ( 1 ) where score ( i , j , eventType ) assigns a score to the interval [ i , j ] to indicate whether it is an event of eventType , scoreϵ ( i − 1 , i , eventType ) assigns a score to the interval [ i − 1 , i ] that spans two frames to indicate whether it is not covered by any event of eventType , which inversely represents the frame-level activation for an event , and Z ( eventType ) is the normalization factor . The non-event score ( scoreϵ ) for two consecutive frames comes from skipping one position in a feasible solution ( see example in the next paragraph ) , and it also allows the model to degenerate to frame-level event prediction . Here , for notational convenience , we omit X for every term . For numerical stability , Eqn . ( 1 ) is computed in the log-domain . The exponent , i.e. , the summation of all score ( i , j , eventType ) and scoreϵ ( i− 1 , i , eventType ) corresponding to YeventType , is the unnormalized log-likelihood . As an example , for a single eventType , assuming there are 7 audio frames < 0,1,2,3,4,5,6 > in total , for the interval set candidate { [ 0,0 ] , [ 2,4 ] , [ 4,5 ] } , the corresponding unnormalized log-likelihood is computed as score ( 0 , 0 ) + scoreϵ ( 0 , 1 ) + scoreϵ ( 1 , 2 ) + score ( 2 , 4 ) + score ( 4 , 5 ) + scoreϵ ( 5 , 6 ) . The computation of logZ ( eventType ) and its gradient w.r.t . to score/scoreϵ , ∇ logZ ( eventType ) , is critical in training . The detailed procedure is shown in Algorithm 1 ( forward-backward algorithm ) . In practice , we compute logZ and ∇ logZ for all eventType ( s ) in parallel . The forward stage and the backward stage in the forward-backward algorithm are batched together , but with all positions of the input flipped , as to compute them in a single pass without the need for two separate update equations , since their calculations are essentially the same . We also find that a substantial speedup ( > 5x ) can be achieved by using a custom gradient computation via the backward pass of the forwardbackward algorithm for logZ , as opposed to using automatic differentiation w.r.t . logZ in PyTorch . This is likely due to a more contiguous memory access pattern and the elimination of unnecessary operations . In order to make memory access at each step more contiguous , the event scores Score and Scoreϵ are organized as Tend× Tstart×N and Tend×N tensors , respectively , with Tend being the ending position of an interval , Tstart being the beginning position of an interval , and N being the number of eventType ( s ) . Here , each slice along the N dimension represents a separate semi-CRF channel . A runtime benchmark of the semi-CRF layer presented is provided in Table 3 . Algorithm 1 Forward-backward algorithm for logZ and ∇ logZ for a specific event type . Input : function score ( i , j ) , function scoreϵ ( i− 1 , i ) Output : logZ and ∇ logZ Forward stage : Initialize the forward variable : v ( 0 ) ← log ( exp ( score ( 0 , 0 ) ) + 1 ) for all j = 1 , . . . , N − 1 do v ( j ) ← log { exp [ v ( j − 1 ) + scoreϵ ( j − 1 , j ) ] + ∑ k < j exp [ v ( k ) + score ( k , j ) ] } v ( j ) ←v ( j ) + log { 1 + exp [ score ( j , j ) ] } end for Readout the log partition function : logZ ← v ( N − 1 ) Backward stage : Initialize the backward variable : q ( N − 1 ) ← log { exp [ score ( N − 1 , N − 1 ) ] + 1 } for all j = N − 2 , . . . , 0 do q ( j ) ← log { exp [ q ( j + 1 ) + scoreϵ ( j , j + 1 ) ] + ∑ k > j exp [ q ( k ) + score ( j , k ) ] } q ( j ) ←q ( j ) + log { 1 + exp [ score ( j , j ) ] } end for Read out the posterior marginals as derivatives : for all i=0 , . . . , N-1 do p ( i , i ) ← exp { v ( i ) + q ( i ) + score ( i , i ) − 2 log [ exp ( score ( i , i ) + 1 ) ] − logZ } end for for all i < j do p ( i , j ) ← exp [ v ( i ) + q ( j ) + score ( i , j ) − logZ ] end for for all i=1 , . . . , N-1 do pϵ ( i− 1 , i ) ← exp [ v ( i− 1 ) + q ( i ) + scoreϵ ( i− 1 , i ) − logZ ] end for ∂ logZ ∂score ( i , j ) = p ( i , j ) , ∂ logZ ∂scoreϵ ( i−1 , i ) = pϵ ( i− 1 , i )

Summary: This paper presents an event-based polyphonic piano transcription model using neural semi-CRFs. Unlike the state-of-the-art "onset and frame" model and its variations, which rely on frame-level note state predictions, this work is based on segmenting a long sequence into a set of intervals which correspond to notes and silences. They also propose a forward-backward learning algorithm for computing the score functions of the intervals and, as a result, it enables to compute the partition function. This new approach achieved comparable performance to previous state-of-the-arts and it has a great potential to audio event detection tasks.