Skipping the Frame-Level: Event-Based Piano Transcription With Neural Semi-CRFs
Authors: Yujia Yan
Frank Cwitkowitz
Zhiyao Duan
Venue: NeurIPS 2021 Poster (2021)
Keywords: Music
Audio
Piano Transcrition
Music Transcription
Semi-Markov
CRFs
Sound Event Detection
Music Information Retrieval

Abstract: Piano transcription systems are typically optimized to estimate pitch activity at each frame of audio. They are often followed by carefully designed heuristics and post-processing algorithms to estimate note events from the frame-level predictions. Recent methods have also framed piano transcription as a multi-task learning problem, where the activation of different stages of a note event are estimated independently. These practices are not well aligned with the desired outcome of the task, which is the specification of note intervals as holistic events, rather than the aggregation of disjoint observations. In this work, we propose a novel formulation of piano transcription, which is optimized to directly predict note events. Our method is based on Semi-Markov Conditional Random Fields (semi-CRF), which produce scores for intervals rather than individual frames. When formulating piano transcription in this way, we eliminate the need to rely on disjoint frame-level estimates for different stages of a note event. We conduct experiments on the MAESTRO dataset and demonstrate that the proposed model surpasses the current state-of-the-art for piano transcription. Our results suggest that the semi-CRF output layer, while still quadratic in complexity, is a simple, fast and well-performing solution for event-based prediction, and may lead to similar success in other areas which currently rely on frame-level estimates.

Skipping the Frame-Level: Event-Based Piano Transcription With Neural Semi-CRFs
Yujia Yan', Frank Cwitkowitz, Zhiyao Duan
Department of Electrical and Computer Engineering
University of Rochester
Rochester, NY, 14627 USA
yujia. yriia.ya@@ochssterr. edu, fcwi koowurr www.hester.eeeee zhiyao www.oodcunnnnteteeeeee.. edu
Abstract
Piano transcription systems are typically optimized to estimate pitch activity at each frame of audio. They often followed by carefully designed heuristics and post-processing algorithms to estimate note events from the frame-level predic- tions. Recent methods have also framed piano transcription ing problem, where the activation of different stages of a note event are estimated independently. These practices are not well aligned with the desired outcome of the task, which is the specification of note intervals as holistic events, rather than the aggregation of disjoint observations. lation of piano transcription, which is optimized to directly predict note events. Our method is based on Semi-Markov Conditional Random Fields (semi-CRF), which produce scores for intervals rather than individual frames. When formu- lating piano transcription in this way, we eliminate the need to rely on disjoint frame-level estimates for different stages of a note event. We conduct experiments on the MAESTRO dataset and demonstrate that the proposed model surpasses the current state-of-the-are for piano transcription. Our results suggest that the semi- CRF output layer, while still quadratic in complexity, is a simple, fast and well- performing solution for event-based prediction, and may lead to similar success in other areas which currently rely on frame-level estimates.
multi-task learn-
this work, we propose a novel formu-
1 Introduction
The task of Automatic Music Transcription (AMT) aims to transcribe music recording into some form of music notation Benetos et all 2018). Examples of notation include MIDI event sequences, e.g, /Hawthorne all [2018), Kong et all [200], Kim and Bello [201%) Kwon alll [2020], and staff notation, e.g., Nakamura et all [008), Román etl 2019). In this work, we address the problem of transeribing piano music into MIDI event sequence. MIDI transcription involves constructing sequence of events, each specified by its onset and offset positions, with the constraint that two events of the same event type (e.g., certain pitches and pedals) cannot overlap. addition to onsets and offsets, the velocity (i.e., value that represents the intensity of a key strike, which informs the loudness) associated with each event often estimated.
In recent years, neural network based approaches have reached the state of the art for the problem of piano transcription, e.g., Hawthoree et all [201]] Kong all [2020], Kwon et [022]]. They operate at the frame-level and make predictions for different stages of note event, i.e., the onset, offset, and pitch activation, separately. order to extract note-level predictions, they use manually designed procedures to combine the disjoint frame-level predictions. These include thresholding,
"Code available at
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

peak picking [Hawthorne at 2018, Kong et all 2021 or low-order hidden Markov models [K stan 2020]. This two-stage approach requires manually crafted procedures and a manually designed state structure modeling the temporal evolution of notes.
In this work, we propose a direct approach note-level transcription. Instead of scoring and aggre- gating note activations across individual time frames, our approach directly scores a time interval's likelihood of covering the entire process of a certain event (c.g., note, pedal usage). This greatly simplifies the formulation of piano transcription. Specifically, the proposed method takes the log- mel spectrogram of an audio segment (e.g., 10s) as input, and uses contextual model to produce contextual embeddings for each frame. It then uses score model to score each possible time interval within the segment assess whether it covers the entire span of musical event (e.g., notes, ped- als) as specified by a specialized zeroth-order semi- Mani-M conditional random fields (semi-CRF) [Sarawagi and Cohen, 200))). These scores are then decoded using Viterbi algorithm into set of non-overlapping intervals for each event type, i.e., note-level transcription. Finally, other attributes of these intervals (i.e., velocity and refined boundary positions) are also estimated.
Previous works using semi-CRF were concerned with shorter sequence lengths and smaller interval lengths [Kong et all 2016, emmeral all 2016, Kemos etall 2019, etll 2016), and usually only attempted to estimate a single channel (track) of non-overlapping events. In contrast, our problem deals with sequences of 400-1600 frames (approximately 10s-40s) and requires a separate semi- CRF for around 90 different labelling channels (88 pitches + 1-3 pedals), each corresponding to a specific event type. Another challenge the wide range of event duration that can be encountered in piano music, which further increases the computational expenses of the problem. These chal- lenges prevented us from using existing general purpose semi-CRF methods due to computational expenses. Instead, we introduce a zeroth-order Semi-CRF formulation that is specifically adapted to the problem and show that can be implemented efficiently (see Table L) for the problem size of interest.
Our experiments on the MAESTRO dataset show that the proposed system achieves Note w/ Offiet Fi of 78% and Note w/ Offset & Velocity F1 of 87.75%, exceeding the current state of the art while being smaller and faster. We believe that this simple, fast and well-peefornning approach is also extensible to other similar tasks with intervals as the prediction target, such as polyphonic sound event detection or speaker diarization.
2 Related Works
Piano Transcription There have been many proposed approaches to piano transcription in the last several decades. Early works consist of simple signal processing methods such as spectral peak- picking [Klapuri et all, 2000, Bello et all, 2006]. which estimates fundamental frequencies directly from the spectrum, or spectral decomposition [Smaragdds and Brown, 2003, HaHaan and Plumbz Jey, 2014]] where a short-time spectrum is factorized into spectral components with corresponding activations. Several parametric models have also been proposed to perform spectro-temporal decom- position [|mmcett et all 200%. Emiya et all, 2009 Cheng et all 2016]. Time-domain decomposition has been also investigated in Cogliati et all explicitly as parametric templates, and tend to suffer from the lack of generalization. Other meth- ods include machine learning techniques such as Hidden Markov Models [Raphael, 2002, Böck and
2017 201G). Most of these methods aim to model notes
Schedl, 2012) or Support Vector Machines [Poliner and BiTrs 2006, WWninger et all 2013). Recently, neural network based approaches have made significant advances [Sigtia et all, 2016, Hawthore all 2018, 2019, Kim and Bella, 201%) by taking advantage of large Disklavier piano databases.
Many methods separate piano transcription into frame-wise polyphonic pitch estimation and note tracking, thereby optimizing for frame-level predictions which are used to estimate quantized note intervals. While intuitive, this practice has a major drawback: the quality of the note-level predic- tions is fundamentally limited by the quality of the frame-level activations, which can be sporadic, discontinuous, and non-uniform in terms of their strength. Kameoka all [[(X0Z]] attempted to cir- cumvent this problem by performing harmonic temporal template structured clustering to explain spectrogram observations as originating from distinct sources, directly estimating notes as discrete events. This reflects the idea of [Brrggaan [990], i.e., that meaningful auditory events are results of simultaneous and sequential groupings. However, the underlying mental process of low-level group- ings more complex than clustering according to simple parametric harmonic temporal templates.

---

Toui
EventType (notes/ppdals)
log-mel spectrum
N
Score,
Decode
Contextual Model
Score Model
Refined
Altributes:
Velocity
OnseV/OlTsel position
N
Tesi
Thegin
Extracted
Interval:
Time
Score
Proposed system overview. In the middle part of the figure. Thegnn and Tend are the number of beginning positions and ending positions, respectively, and N is the number of eventType(s).
Figure I:
Some more recent works [Kelz et all 2019 Kwon et all 2020| attempt to incorporate manually- defined state structure for modeling the temporal evolution of notes. However, the state structure (usually of low order) may reflect the actual temporal evolution and thus limits the representa- tive power of these models. Coglati el al. [207]] directly estimates the activation note templates in the time domain, but these templates are preset and are piano- and context-dependent, lacking the generalization to unseen pianos or recording environments.
The proposed method learns estimate note events holistically, rather than adopting the frame-to- note two-stage approach or performing and aggregating multiple subtasks (i.c., onset, offset, and pitch estimation). It can be viewed performing groupings directly in the event space, with low- level details being learned by the neural networks.
Semi-Markov CRFs semi-Markov conditional random field (semi-CRF, Sarawagr and Cohen
defines conditional probability distribution sets of non-overlapping labeled intervals
within an imput sequence. Semi-CRFs have been used in Chinese word segmentation [Lw ef all
Kong et al, (0166, named entity recognition [Zhun 2016. Ye and Ling, 2018. Arora et
2019
character-level part-of-speech tagging |emos et all 2019), phone recognition [Lu et all
2016 chord recognition [Masada and Bunescu. 2017)), etc. The time complexity for computing the
partition function and inferring the most likely configuration is quadratic with respect to the length of the sequence. Therefore, most works set an upper bound on the length of single interval, e.g..
Kemos et all [201%], to make the computation tractable.
In this work, semi-CRF is defined for each semi-CRF formulation differs from the standard formulation in the following ways: 1) Different events in semi-CRF are allowed overlap on their endpoints (boundary frames), since audio frames represent time period and can contain both the end and beginning of an event of the same type. This treatment allows to use larger hop size without adjustment due overlap in the same frame, so that the sequence length for the same audio segment can be reduced; 2) A frame is not required to belong to any event. Despite the loosened definition, we use the name semi-CRF refer to this type of structured prediction module.
type (i.e., note of specific pitch, a pedal). Our
Compared to the tasks mentioned above, the task of piano transcription has longer input sequence and a larger range of possible event duration. The former makes the time complexity issue more prominent, while the latter makes it difficult specify upper bound event duration for speedup purposes. We show that our formulation is efficient for piano transcription and hence does not require an upper bound on event duration, thanks set of implementation optimizations.
3 Proposed Semi-CRF Approach to Piano Transcription
The proposed system transcribes the input audio into list of musical events, i.e., notes and pedals. Here the term events refers to acoustical events that span certain time intervals (not to be confused with raw MIDI events, i.c., note on, note off, cc, etc., used for serializing the performance). Taking
3

---

an audio segment (e.g,, 10s), the transcription process is illustrated in Figure M1 First, log-mel spectrogram is computed after taking the short-time Fourier transform (STFT) of the audio. This is used as input to a contextual model to produce contextual features across time frames. Such features are then fed a score model to calculate two kinds of scores. The first kind of score (Score) assigns score to indicate whether an interval is an event of a certain event type. The second kind of score (Score,) indicates whether an interval that spans two frames is not part of any event for a certain event type. Finally, Viterbi algorithm is used to decode the most likely sequence of events for each event type, using the aforementioned scores. Autributes, i.e. velocity and refined onset/offset positions, are then estimated for each extracted event.
3.1 CRF Formulation
Let X' =< Io, 21. IN-1 > be an audio segment containing a sequence of N time frames. Let )' = ( even(Type), i S j} be the set of musical events entirely contained within this segment, with time quantized to audio frames. Here i and are respectively the beginning and ending frame indices for each event, and eventType is the type of the event, e.g., a specific key (pitch) of the the 88 keys of a piano or the sustain pedal. Events that extend outside the audio segment are not considered in this formulation, but will be handled the inference process (See Section (.3) We assume that for the same event type, two events A and B are non-overlapping, i.e., either jA S iB or jB S iA- In this formulation, it is allowed to have single-frame events where i j (note how this would make the formulation differ from the standard semi-CRFs).
We associate each event with a set of attributes: 1) the relative non-quantized position of onsets and offsets at the sub-frame level, represented by a value from -0.5 to 0.5 indicating the position relative to the quantized index of the frame, and 2) the MIDI velocity of the event, represented by a discrete value from 0 to 127.
We use Deventlype to denote the subset of events that contains only specific event type. For each event type, we model the following conditional probability:
p(Veventlypll (2)
exp
score(i. eventT
Z( ((veenTyype)
(v.jevenrfyyee) ED eootttyp
score, (i - 1, even/Type) i,eee//yyyT ].
Σ
+
[i=1,t] not covered in Veuuiioe
where score(i, eecei...eeettttppp. ype) assigns a score to the interval [ to indicate whether it is an event of eveníType, score, 1, i, eveníType) assigns a score to the interval [ 1, that spans two frames to indicate whether it is not covered by any event of even/Type, which inversely represents the frame-level activation for an event, and Z(even/Type) is the normalization factor. The non-event score (score,) for two consecutive frames comes from skipping one position in feasible solution (see example in the next paragraph), and it also allows the model to degenerate to frame-level event prediction. Here, for notational convenience, we omit X' for every term.
For numerical stability, Eqn. (II is computed in the log-domain. The exponent, i.e., the summation of all score(t. j, event/ype) and score, 1, evenrType) corresponding to Veventyyp, is the unnor- malized log- ligeeeilioood. As an example, for single eventType, assuming there are 7 audio frames 1,2.3,23, 3,5,6,66> in total, for the interval set candidate [[0.0], [2,4], [4,5]], the corresponding unnor- malized log-likelihooo is computed as score(0, 0) + score, (0, 1) + score, (1, + score(2, 4) score(4, 5) + score, 6).
The computation of log (event ype) and its gradient w.r.t. to score/score, log Z(eventT ype). is critical in training. The detailed procedure is shown in Algorithm II woorward-bbckwaard algorithm). In practice, we compute log 2 and V log Z for all eventType(s) in parallel. The forward stage and the backward stage in the forward- backaard algorithm are batched together, but with all positions of the input flipped, as to compute them in a single pass without the need for two separate update equations, since their calculations are essentially the same. We also find that substantial speedup (>5x) can be achieved by using custom gradient computation via the backward pass of the forward- backward algorithm for log Z, as opposed to using automatic differentiation w.r.t. log Z in PyTorch. This is likely due to a more contiguous memory access pattern and the elimination of unnecessary operations.

---

In order to make memory access at each step more contiguous, the event scores Score and Score, are organized as T Tnd x T start& N and Tend N tensors, respectively, with Tend being the ending position of an interval, starr being the beginning position of an interval, and N being the number of even(Type(s). Here, each slice along the N dimension represents separate semi-CRF channel.
A runtime benchmark of the semi-CRF layer presented is provided in Table B
Algorithm 1 Norward-backward algorithm for log Z and V log Z for specific event type.
Input: function score(i, function score, 1,
Output:
log Z and V log Z
Forward stage:
Initialize the forward variable: u(0) <- log(exp(score(0.0) + 1) for all = -
do
v(i) + log { explo(j - 1) (j 1,5)] + Σ
explu(k) + score(k..)]] bマj
v(i) te( + log(1 exp[score
end for
Readout the log partition function: log Z + v(N 1)
Backward stage:
Initialize the backward variable: g(N 1) + -bog/oxplscor(N--1,, 1,N 1)] + for all j = N 2..... do
((1) + log { exp (j expia(i
+ 1) score 1)] Σ
expla(k) + score(j. kzj
<(i) ( + log(1 + exp(score(j..i))
end for
Read out the posterior marginals as derivatives:
for all i=0...., N-1 do
p(i, i) + ((() + q(i) + score(i, i) - loglexp(score((,) i) + 1)] log } end for
for all i<j do
p(i,j) t exp (u()) {(j) score(i. j) log
end for for all i=l...., N-1 do
(i - 1, t explo(i - 1) + q(i) + score, (i - 1, - log Z end for
8 log Z [ecore(f]] p(i,j); Dscore, 1,3 pe(i - 1,
log
3.2 Training Objectives
For training, we use maximum likelihood estimation (MLE), where the conditional log- likeliodd defined to consolidate the conditional probability in Eqn. (II) over all event types, assuming their conditional independence given X::
logp()"(2'= =
Σ log ((Neoou7spp
evenTTpe
One may question the validity of this conditional independence assumption on different event- Type(s); here, we keep things simple, as previous approaches often make this assumption. We leave this for future investigation.
5

---

In addition to the log- log-likelihood for the presence of events defined in Eqn. (2), we also learn to predict three attributes for each event:
log p(attributes]e) (e) = log p(velocityle) + log p(refined onsetle) + log p(refined offset /e).
(3)
Here we use to denote an event. We parameterize these terms with the following distributions, i.e., the Softmax/Multinomial distribution, and the Continuous Bernoulli distribution Loaiza-GGanem and Cunnngham, 2019:
velocityle Softmax(p(ee),,
(4)
0.5 + refined onset/offser ~ ContinuousBermouli(((c...
where (((e) and A(e) parameters produced by neural networks that take features of the interval as the input. The interval features used in this work are described in Section 14. The final objective defined as
L = [log p())))) + Σ log p(atribbbut.)ss))..... eEy
(5)
More discriminative and cost-sensitive losses such as max- mar-in and softmax margin can be used as drop-in replacements, but we leave this investigation future work.
3.3 Inference
We use dynamic programming (Viterbi) to infer the most likely interval sequence for every eventType independently. The procedure is described in Algorithm A
When processing longer audio recordings, our system transcribes audio segment by segment. Be- cause the system is designed to ignore all events that extend outside a segment, these audio segments need overlap with each other. In this work, segments are 10s long and the overlap is 5s. In order to properly handle notes near the boundaries of a segment, we modify the algorithm such that it forced to take the result from the overlapping portions into account as follows. Within an audio segment, decoding is performed in reverse order: backtracking starts from the position immediately after the last event decoded in the last segment of the same event type, if any of those overlap with the current segment. We found that, for each segment, it works slightly better to discard all events of which the onsets fall into the overlapping region with the next segment, which is procedure similar to the overlap-discart method for computing discrete convolutions for long signal.
Algorithm 2 Viterbi (MAP) decoding of specific eventType within audio segment.
Input: function score(i, function score, (i 1) backtracking starting frame position Output: a set of intervals y vNN 1) f maxx(scoreAA 1, 1). for all; j E N - 2............................... do
vij + 1) + score, + skip if inactive (o(k) + score k)} if an interval v(j) + v(j) + max((core((,))))m 0) single frame case end for
v(j) + max
Perform backtracking starting from position / to get y
After events are extracted, feature vectors of the events, described in Section KZL, are used predict attributes, namely, velocity and refined onset/offset positions. These attributes are then assembled with the events to form event tuple (onset, offset, eveníType, veloctty for the final output.
3.4 Model Architectures
In this section, we present details of the neural architectures for the three components used in the proposed approach: 1) contextual model, 2) score models, and 3) attribute predictors. To recap, we first apply contextual model transform the input audio frames into sequence of contextual embeddings. These contextual embeddings are then used by the score models to compute score(-) for possible events, in a fashion similar to Cross and Huang (201), Kilaey and Klein [2018]] Lm at [20]A], and (.). After events are extracted, contextual embeddings are also used by the attribute predictors to predict attributes (i.e., velocity and refined onset/offset times) associated with each event.

---

ConvBlock
Contextual embeddings:
Ray Rx-1
(++)
X
Contextual Model
Tayin
MLPLaayer
PLayer
Sonne, (1
1,4,))
by., bv Bys Sreeathee.
Dhe Rg, he hy. Rovertpe
Input frames 2gy 4--1
MLyyeyer
MLPLayer
Nh by @
Bu.1.hu. h.]
Tirst Free moments of hij:
(a) Contextual model
(b) Score models
(c) Attribute predictors
Figure 2: Model architectures. The contextual model transforms the input frame sequence into sequence of context embeddings, based on which scores and attributes are computed by the score models and attribute predictors, respectively.
3.4.1 Contextual Model
As input to the contextual model, we compute a log-mel spectrogram with frame size of 4096 samples, a hop size of 1024 samples, and the Hann window function. Following Hawthorne et all [2U18] Kong all [202]] use 229-bands with frequency range from 30 Hz to 8000 Hz. In ad- dition to a single Hann window applied to each frame, we also experimented with applying multiple adjustable Gaussian windows (5 windows, initialized with equidistant centers and constant variance) to the same frame and concatenating the resulting log-mel spectra together for improving the tem- poral resolution of the spectra input. By doing this, the input becomes log-mel spectrogram with multiple channels for each fimeuecyy bin, with each channel obtained from a different window function. We feed the log-mel spectrogram through four convolutional blocks with kernel size of 3 and 48, 64. 92, 128 filters, respectively. Each convolutional block contains two 2-d convolution lay- ers, followed by batch normalization and a Gaussian Error Linear Unit (GELU) activation functioon Hendrycks and Gimpel, 2016). At the end of each block, the output is pooled along the frequency dimension using average pooling with a kernel size and stride of 2. The channel and frequency dimensions are flattened together and fed into two-layer bidirectional GRU with hidden size 256. This contextual model largely the same as the one used in Kong et all [2020] however, we only utilize one instance of the block rather than stacking it for different frame-level prediction targets.
3.4.2 Score(i, and Score, - 1,,
As shown in Fig. 2R the features used for scoring an interval [ include contextual embeddings at its two endpoints, i.e., h; hy. their elementwise multiplication h, hj, and the first three moments for the contextual embeddings within the interval [. These features are chosen because they provide information for events whole while being cheap compute, and are often seen in the literature when extracting holistic/pairwise features. These features are concatenated and fed into a three-layer feed-forward neural network (MLPLayer) obtain raw intervalic score tensor with shape Tend X T beginn x N, where the three dimensions are the ending position of an interval, the beginning position of interval, and eventType channels, respectively. The output size of this network is equal to the number of event types and the hidden size is equal to 4x the output size. On top of the raw score tensor, as shown the figure, we experimented with applying a simple convolutional block with two convolutional layers of kernel size 3 and filter size being 3x the output size, with the hope that the block can aggregate neighboring information in the space of intervals in order to increase the expressivity. We also experimented with scaling the intervallic score by the length of the interval, motivated by the following observation: when the score function for an interval is chosen to be the variance alone with this length scaling added, the SemiCRF layer in this
7

---

work is equivalent to optimal 1-d k- k-means clustering. This scaling adjusts the score for event to be roughly at the same scale of the aggregated non-event score at the same time interval at imitialization.
For the inactivity score, score, (i 1,4), only contextual embeddings two consecutive frames are used. These similarly fed into three-layer feed-forward network with output size equal to the number of event types.
3.4.3 Attribute Predictor
As shown in Fig. 2d, for predicting attributes associated with each event, we use features of end- points concatenated with an trainable evenTTye embedding vector SeventType with size 256 for fully specifyyng the event. These features are fed into separate multi-layer feed-forwar networks. The velocity prediction network has hidden sizes of 512/512, and produces a 128-dimensional logits to infer the MIDI velocity value. The refined onset and offset time prediction network has hidden sizes of 512/128, and produces a 2-dimensional real vector, which is used as logits for the Continuous Bernoulli Distribution for onset/offset refinement, respectively, defined in Eqn. (9))
4 Experiments
4.1
Dataset
We conduct experiments using the MAESTRO v2 dataset [Hawthome all 2019, which con- tains around 200 hours of MIDII-syncronnzeed (3ms precision) virtuoso piano performance record- ings. The recordings were collected across several years of the International Piano-e- Piano-e-Comper Competition, and were recorded on Yamaha Disklavier pianos. All recordings are sampled at 44.1 kHz, except for files from the 2017 and 2018 competitions, which are sampled at 48 kHz; we thus downsample them to 44.1 kHz for consistency. For comparing with other works, we follow the convention to extend the offset of notes to the offset of any simultancous sustain pedal event [HHathorne: all 2018, Kong et al. 2020].
4.2 Training
We use batch size of 12 and Adabelief /hhuang al. 2020] optimizer with weight decay of le-4. We use oneCycle [Smith and Topin, 2019 learning rate scheduler with maximum learning rate set to 6e-4 for 180k iterations and cosine annealing. The learning rate is increased gradually for 20% of iterations and then gradually annealed 1.5e-5. We automatically determine the value for gradient clipping by using the 0.8 quantile of the gradient norm during the last 10k iterations, which a strategy similar to Seetharaman et all [021 We apply dropout with rate 0.1 on the attribute predictors and the score model.
4.3 Evaluation Metrics
We follow the standard piano transcription evaluation procedure, validating note predictions with multiple levels of criteria. The most basic metric considers a note prediction correct if the esti- mated onset is within 50 ms of the corresponding ground-truth onset. The next incremental metric additionally requires that the offset prediction is within 50 ms or 20% of the note duration from the ground-truth offset position. The final incremental metric additionally requires that the velocity estimate is within a tolerance of 0.1 ([0,1) normalized velocity) from the ground-truth velocity, as defined in Hawthorne et [2018]. We directly use the implementation contained in the mir_eval library [attel all 2014 to compute these three note- metrics.
We also introduce activation-level metries which serve hopsize- aagiizze-ggssccccccc replacement for the commonly reported frame-level metrics for evaluating how predicted time spans events overlap with the ground truth (see the supplementary material for details).
Following the convention, each result is averaged across all pieces within the test set. We compute similar metrics, minus the velocity variation, for the sustain pedal activity, as in Kong all [02)].

---

4.4
Main Results
We compare the proposed system the state-of-the-ar methods for piano transcription using the MAESTRO v2 test split. We recompute these metrics for other systems directly from the transcribed MIDI files generated by their pretrained models. We also report our results for our model trained and evaluated on the MAESTRO v3 splits for future reference. The note transcription and pedal transcription results are listed in Tables and Q. respectively.
Activation
11
Note Onset
Note w/ Offset
Note w/ Offset & Vel.
Method
P
R
P
P
1
R
A
P
R
Fi
P
R
Fi
MAESTRO v2
Hawthorne et all [2019]
86,84
89.24
87.82
97.88
92.26
94.93
82,09
77.44
79.65
78.37
73,94
76.05
Kong all [2020]
90.09
90.42
90.15
98. 16
95.46
96.77
85.65
83.32
84.45
84.18
81.92
83.02
Proposed w/o Extra Win
93.84
88.07
90.75
98.85
93.97
96,31
90.67
86.24
88.37
89.60
85,25
87.34
Proposed w/o Post Conv
93,90
88.18
90.85
98.86
94.06
96.36
90.52
86.18
88.26
89.49
85.23
87.27
Proposed w/o Len Scaling
93.85
88.25
90.87
98,80
94.15
96.39
90.70
86.48
88.51
89.66
85.51
87.50
Proposed Proposed (batchsize=20)
93.84
88.48
90.98
98.78
94.18
96.39
90.79
86.62
88.
89.78
85,68
87.65
93.85
88.72
91.11
98.66
94.50
96.51
90.68
86.89
88.72
89.68
85.96
87.75
MAESTRO
Proposed (batchsize= 20)
93.79
88.36
90.75
98.69
93.96
96.11
90.79
86.46
88.42
89.78
85.51
87.44
Table 1: Piano transcription note results for the proposed methods and various related works.
Activation R
Onset
Onset & Offset
Method
Fi
Fi
Fi
MAESTRO v2
Kong et al. [2020]] Proposed w/o Extra Win
94.14
94.29
94.11
77.43
78.19
77.71
73.56
74.21
73.81
95.29
86.36
90.02
81.56
73.47
76.99
78.14
70.50
73.83
Proposed w/o Post Conv
95.19
87.05
90.36
81.72
73.29
76.99
77.99
70.05
73.54
Proposed w/o Len Scaling
94.73
87.32
90.34
80.79
73.51
76.71
77.36
70.51
73.53
Proposed Proposed (split) Proposed (batchsize=20)
95.13
87.71
90.73
82.14
74.91
78.10
78.48
71.72
74.71
95.20
89.60
91.84
83.01
77.55
79.98
79.72
74.55
76.85
95.35
87.63
90.78
82.27
75.61
78.55
78.72
72.42
75.20
MAESTRO v3
Proposed (batchsize= 20)
95.17
88.33
90.98
82.18
75.81
78.52
78.75
72.74
75.30
Table 2: Sustain pedal detection results for the proposed methods and various related works.
On MAESTRO v2, the proposed system achieves a Note w/ Offset F1 of 58%72% and a Note w/ Offiser & Velocity F, of 87.75% significantly outperforming previous methods in predicting holistic note events. Regarding the Note Onset Fl score, Kong et all [212]] has the highest performance. Our proposed method slightly underperforms Kong et all [2020], but still outperforms Hawthorne all [2019] by about 1.5%. Regarding the Activation F1 score, our proposed method achieves the highest performance, outperforming Kong et al, [202]] by 1% and outperforming Hawthorne et all [201]] by 3%. This is the case even though the proposed method is not optimized to output frame-level predictions, while the compared methods both have frame-level prediction branch.
The Proposed w/o Extra Wins entry refers to the variation which does not apply extra adjustible windows for creating multiple version of log-mel spectrogram the input. The results indicate that this oversampled log-mel spectrogram input helped improve the system slightly. The Proposed w/o Post Conv entry refers to the variation which does not include the final convolutional block when computing the score matrix. The results indicate that this convolutional block also helped improve the system slightly. The Proposed w/o Len Scaling entry refers to the variation which scales the scores in the intervallic score matrix by the length of the interval. The results indicate that this scaling of intervallic scores also helped improve the system slightly. We also found that increasing the batch size from 12 to 20 improves the result.
Regarding discrepancies with respect to numbers reported in [Kong et al 2020J, we inspected their code and found a bug that affects a small portion of note offsets when handling pedal extension. In our evaluation, we use the correct offset labels, which improves the results of their model slightly. For pedals, they use a onset tolerance of 200ms.

---

In terms of sustain pedal transcription results (Table 2D, our method achieves an Onset F1 of 78.55% and an Onset w/ Offiee F of 75.2%. outperforming Kong et al, [2020] in predicting holistic pedal events. Because Kong et all [202]] uses separate model replicating their note transcription model to predict pedal activity separately, we also experimented with splitting the original architecture into two branches with only the convolutional blocks shared, still training these two tasks (notes and pedals) together. The result is shown in the table as Proposed (split). This variation with two branches for notes and pedals further improves all Fl metrics for pedal transcription. It suggests that there may be some negative transfer between these two tasks and further improvements may be achieved by tuning branches for notes and pedals separately. Kong et all [2020] yields Fi of 94.1 1% for predicting the pedal activation, that is 2% higher than the proposed model, due to its higher recall. This difference may be due to the fact that our system is only trained to make predictions on the holistic events instead of individual frames.
4.5 Runtime Analysis
The proposed model has O(L²N) time complexity, where L is the length of the input sequence, iee. the number of frames, and is the number of eventType(s). The semi-CRF layers involve sequential computation, and traditionally were often considered too slow for practical use with re- spect to long sequences if no restrictions were applicd. Here we benchmark the computation time of several components of the proposed algorithm that have quadratic time complexity, using input sequence of a reasonable length for music transcription. The results are shown in Table E. With a hop size of 1024 samples and sampling rate of 44.1 KHz, 400, 800, and 1600 frames correspond to audio segments of 9.29 s, 18.58 s, and 37.15 s. respectively, which are common lengths used for audio processing. For these lengths, the ratio of the audio length to the time taken by each compo- nent are within the acceptable range. The algorithms were implemented in PyTorch, and we believe that further speedup can be achieved with native C+CCUUA implementation.
400
Frames
800 F Frames
1600
Frames CPU
Components
GPU
CPU
GPU
CPU
GPU
Forward Backward (Py autodiff)
0.39
5.95
4.40
44.69
17.68
354.40
Forward Only
0.05
0.20
0.09
0.80
0.23
3.08
Backwrd-Backward
0.08
0.58
0.12
2.32
0.27
7.12
Viterbi
0.23
0.16
0.27
0.36
0.54
0.88
Pairwise Scores
0.06
0.84
0.25
3.14
0.99
11.76
Table Running time (seconds) of algorithm components that have quadratic time complexity w.r.t.
the input length on Intel(R) Core(TM) i7-7800X CPU @ 3.50 GHz and Nvidia GTX 1080TI. Events for N = 90 eventTypes and for a single audio segment are predicted.
To get a clearer idea on the overall speed of the system, we then compare the entire running time with the previous ofate-of-the-an system transcribing the same audio file, Carl Czerny Grand Sonata 0p.145 145 No.9, which is 33.3 minutes long. For a fair comparison with Kong et al, [2121]] we do not batchify our computations across audio segments for transcribing a single audio file. Our system takes 95s while Kong et al. [2020]] takes 353s.
5 Conclusions
In this work, we proposed a piano transcription system designed to directly predict note events. uses a specialized formulation of a semi-Markov conditional random fields, where the prediction targets are set of non-overlapping events for each piano key and the sustain pedal. Using this for- mulation, we eliminate the need to build post-pros post-processing algorithms or heuristics atop frame-level estimates, as well as the need to aggregate disjoint predictions from multiple tasks, e.g., onset, offset and pitch estimation. The results of our experiments on MAESTRO, popular piano transcription dataset, show that our model significantly outperforms two state-of-the the piano transcription meth- ods on note-level transcription. We show that the proposed method can be implemented efficiently enough for the task domain and that the final system transcribes faster than the existing state of the art system. We believe that this simple, fast, and well-performing approach is extensible to other similar tasks which currently rely on frame-level estimates.
10

---

Funding Disclosure
This work has been partially funded by the National Science Foundation grants No. 184614 and DGE-1122591.. Additional revenues related to this work: Zhiyao Duan's sabbatical leave at Kwai Inc. and part-time activities at Mango Future Group Limited.
References
Ravneet Arora, Chen-Tse Tsai, Ketevan Tsereteli, Prabhanjan Kambadur, and Yi Yang. semi- Markov structured support vector machine model for high-precision named entity recognition. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5862-5866, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19- 1587. URL https: h/pp:://wwww http://wwwwwwwww...///.. aclweb. wtt/ http//ttbologg//9111110
Juan Pablo Bello, Laurent Daudet, and Mark B Sandler. Automatic piano transcription using fre- quency and time-domain information. IEEE Transactions on Audio. Speech, and Language Pro- cessing, ((6):2242-2251. 2006.
Emmanouil Benetos, Simon Dixon, Zhiyao Duan, and Sebastian Ewert. Automatic music transcrip- tion: An overview. IEEE Signal Processing Magazine, 36(1):20-33) 30, 2018.
Sebastian Böck and Markus Schedl. Polyphonic piano note transcription with recurrent neural networks. In 2012 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 121-124. IEEE, 2012.
Albert Bregman. Auditory Scene Analysis: The Perceptual Organization of Sound, volume 95. 01 1990. doi: 10. 10.1124//.404444
Tian Cheng, Matthias Mauch, Emmanouil Benetos, Simon Dixon, al. An attack/decay model for piano transcription. In Proceedings of the 17th International Society for Music Information Retrieval Conference .IMIR 2016,, New York City, United States, 2016.
Andrea Cogliati, Zhiyao Duan, and Brendt Wohlberg. Context-dependent piano music transcription with convolutional sparse coding. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 24(12):2218 2230, 2016.
Andrea Cogliati, Zhiyao Duan, and Brendt Wohlberg. Piano transcription with convolutional sparse lateral inhibition. IEEE Signal Processing Letters, 24(4):392-396 396, 2017.
James Cross and Liang Huang. Span-based constituency parsing with a structure-label system and provably optimal dynamic oracles. In Proceedings of EMNLP. 2016.
Valentin Emiya, Roland Badeau, and Bertrand David. Multipitch estimation of piano sounds using a new probabilistic spectral smoothness principle. IEEE Transactions on Audio, Speech, and Language Processing, 1663): 2009.
Curtis Hawthorne, Erich Elsen, Jialin Song, Adam Roberts, Ian Simon, Colin Raffel, Jesse Engel, Sageev Oore, and Douglas Eck. Onsets and frames: Dual-objective piano transcription. In Pro- ceedings of the 19th International Society for Music Information Retrieval Conference, ISMIR 2018, Paris, France, 2018, 2018. URL https: /ttp:///axxvv..o org/ abs/1710
/ttt///xw.....///////////.... 11153
Curtis Hawthorne, Andriy Stasyuk, Adam Roberts, Ian Simon, Cheng-Zhi Anna Huang, Sander Dicleman, Erich Elsen, Jesse Engel, and Douglas Eck. Enabling factorized piano music modeling and generation with the MAESTRO dataset. International Conference on Learning Represen- tations, 2019. URL https://openreviee.. wttp://ooooommddrr11/....
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv: 1606.08150. 2016.
Hirokazu Kameoka, Takuya Nishimoto, and Shigeki Sagayama. A multipitch analyzer based on harmonic temporal structured clustering. IEEE Transactions on Audio, Speech, and Language Processing, 15(3):882-99) 2007.

---

Rainer Kelz, Sebastian Böck, and Gerhard Widmer. Deep polyphonic adsr piano note transcription. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Process- ing (ICASSP), pages 246-250. IEEE, 2019.
Apostolos Kemos, Heike Adel, and Hinrich Schütze. Neural semi-markov conditional random fields for robust character-basee based part-of-speech tagging. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume I (Long and Short Papers), pages 2736-2743. 2019.
Jong Wook Kim and Juan Pablo Bello. Adversarial learning for improved onsets and frames music transcription. In Proceedings of the 20th International Society for Music Information Retrieval Conference, ISMIR 2019, pages 670-677, 2019.
Nikita Kitaev and Dan Klein. Constituency parsing with self-attentive encoder. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages
2018.
Anssi Klapuri, Tuomas Virtanen, and Jan-Markus Holm. Robust multipitch estimation for the anal- ysis and manipulation of polyphonic musical signals. In Proc. COST-G6 Conference on Digital Audio Effects, pages 233-236, 2000.
Lingpeng Kong, Chris Dyer, and Noah A Smith. Segmental recurrent neural networks. 2016
Qiuqiang Kong, Bochen Li, Xuchen Song, Yuan Wan, and Yuxuan Wang. High-resolution piano transcription with pedals by regressing onsets and offsets times. arXiv preprint arXiv:20/0.01815, 2020.
Taegyun Kwon, Dasaem Jeong, and Juhan Nam. Polyphonic piano transcription using autoregressive multi-state note model. In Proceedings of the 19th International Society for Music Information Retrieval Conference, ISMIR 2018, Paris, France, 2018, 2020. URL patts::/arrtie org/ /bs/
1/10.1155 11133
Yijia Liu, Wanxiang Che, Jiang Guo, Bing Qin, and Ting Liu. Exploring segment representations for neural segmentation models. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, pages 2880-2886, 2016.
Gabriel Loaiza-Ganem and J. Cunningham. The continuous bernoulli: fixing a pervasive error in variational autoencoders. In NeurlPS, 2019.
L. Lu, Lingpeng Kong, Chris Dyer, Noah A. Smith, and Renals. Segmental recurrent neural networks for end-to-end speech recognition. In INTERSPEECH, 2016.
Kristen Masada and Razvan C Bunescu. Chord recognition in symbolic music using semi-markov conditional random fields. In Proceedings of the 18th International Society for Music Information Retrieval Conference, ISMIR 2017, Suzhou, China, pages 272-278, 2017.
Eita Nakamura, Emmanouil Benetos, Kazuyoshi Yoshii, and Simon Dixon. Towards complete poly- phonic music transcription: Integrating multi-pitch detection and rhythm quantization. 20/8 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 101-105. IEEE, 2018.
Ken O'Hanlon and Mark D Plumbley. Polyphonic piano transcription using non-negative matrix factorisation with group sparsity. 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3112-3116. IEEE, 2014.
Graham Poliner and Daniel PW Ellis. A discriminative model for polyphonic piano transcription. EURASIP Journal on Advances in Signal Processing, 2007:1-9, 2006.
Colin Raffel, Brian McFee, Eric J. Humphrey, Justin Salamon, Oriol Nieto, Dawen Liang, and Daniel P. W Ellis. Mir_eval A transparent implementation of common mir metrics. In Proceed- ings of the 15th International Society for Music Information Retrieval Conference, pages 367-372, 2014.
12

---

Christopher Raphael. Automatic transcription of piano music. In 3rd International Conference on. Music Information Retrieval, ISMIR 2002, Paris, France, 2002.
Miguel A Román, Antonio Pertusa, and Jorge Calvo-Zaragoza. An end-to-end framework for audio- (o-score music transcription monophonic excerpts. In Proceedings of the 19th International Society for Music Information Retrieval Conference, ISMIR 2018, Paris, France, 2018.
Miguel A Román, Antonio Pertusa, and Jorge Calvo-Zaragoza. A holistic approach to polyphonic music transcription with neural networks. In ISMIR, 2019.
Sunita Sarawagi and William W Cohen. Semi-markov conditional random fields for information extraction. In Proceedings of the 17th International Conference on Neural Information Processing Systems, pages 1185-1192. 2004.
Prem Seetharaman, Gordon Wichern, Bryan Pardo, and Jonathan Le Roux. Autoclip: Adaptive gradient clipping for source separation networks. In 2020 IEEE 30th International Workshop on Machine Learning for Signal Processing (MISP), pages 1-6. IEEE, 2020.
Siddharth Sigtia, Emmanouil Benetos, and Simon Dixon. An end-to-end neural network for poly- phonic piano music transcription. IEEE/ACM Transactions Audio, Speech, and Language Processing, 24(5):927-939, 2016.
Paris Smaragdis and Judith C Brown. Non-negative matrix factorization for polyphonic music tran- scription. In 2003 IEEE Workshop on Applications of Signal Processing Audio and Acoustics (IEEE Cat. No. 03TH8684), pages 177-180. IEFE, 2003.
Leslie N Smith and Nicholay Topin. Super-convergent Very fast training of neural networks using large learning rates. In Artificial Intelligence and Machine Learning for Multi- -Daiiinn Operations Applications, volume 11006, page 1 1100612. International Society for Optics and Photonics, 2019.
Emmanuel Vincent, Nancy Bertin, and Roland Badeau. Adaptive harmonic spectral decomposition for multiple pitch estimation. IEEE Transactions on Audio, Speech, and Language Processing, 18(3):528 537,53377500 2009.
Felix Weninger, Christian Kirst, Björn Schuller, and Hans-Joachim Bungartz. discriminative ap- proach to polyphonic piano note transcription using supervised non-negative matrix factorization. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 6-10. IEEE, 2013.
Zhixiu Ye and Zhen-Hua Ling, Hybrid semi-markov erf for neural sequence labeling. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguisties (Volume 2: Short Papers), pages 235-240, 2018
Juntang Zhuang. Tommy Tang, Yifan Ding, Sekhar Tatikonda, Nicha Dvornek, Xenophon Pa- pademetris, and James Duncan. Adabelief optimizer: Adapting stepsizes by the belicf in observed gradients. Conference Neural Information Processing Systems, 2020.
Jingwei Zhuo, Yong Cao, Jun Zhu, Bo Zhang, and Zaiqing Nic. Segment-level sequence modeling using gated recursive semi-markov conditional random fields. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1413- 1423, 2016.
13


Reviews:
[{"review_id": "eQqKD74IjoR", "reviewer": "Reviewer_ygBT", "summary": "The paper proposes to use semi-CRF for piano transcription. Since piano transcripts include asynchronous events and vanilla semi-CRF is not able to cope with these, the paper assumes that the events are independent from each other. Additional features for predicting the velocity, onset, offset refinements are included. Experiments show that the semi-CRF approach can be on par with the state of the art.", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I am giving a score of 5, because the novelty is somewhat lacking, the approach assumes a strong assumption, the experiments do not demonstrate the strengths, weaknesses, properties of the proposed model.\n\n# Novelty\n\nThe novelty of this paper lies in the application of semi-CRF to the task of piano transcription, so it is somewhat lacking. Piano transcription is actually the perfect task to extend semi-CRF to handle asynchronous events. The novelty could have been a lot more significant, if the paper extended semi-CRF for that.\n\n# Approach\n\nThe major assumption in the approach is that the events are assumed to be independent of each other. This is clearly not the right assumption, but it is somewhat alleviated by the fact that the scores are generated by a one network. In other words, it can be seen as 88 + 3 independent semi-CRF models with a shared feature encoder.\n\nThe paper also talks about the implementation in depth. It reads like an ad-hoc implementation, but in fact, the forward-backward can be implemented as matrix multiplications. A general implementation, such as torch-struct (link below), should suffice.\n\nhttps://github.com/harvardnlp/pytorch-struct\n\nAnother approach is to implement forward-backward on CPU, while keeping the rest of the computation for neural networks on GPU. Comparing to the FLOPS in neural networks, the forward-backward in semi-CRF should not be the bottleneck.\n\nMaximum duration is not imposed in this word, but I suppose if efficiency is of importance, maximum duration should be imposed. It might also help modeling, because it is much less likely to hold a pitch or a pedal for a really long time. It would be even better to have explicit duration modeling, and that the one of the strengths of using semi-CRF instead of other frame-based models.\n\n# Experiments\n\nThe numbers for different systems are extremely close. I wonder what amount of improvement is considered significant. It seems that (Kong et al., 2020) is the better ones in most category. The paper does not provide any experiments to study the internals of the proposed model. If the goal is to get the best numbers, taking features from (Kong et al., 2020) and integrate them into semi-CRF might be worth a try.\n\nThe computation time table does not have a baseline. It is probably better to report real-time factors. Since one of the strengths of choosing the particular score function is its efficiency over (Kong et al., 2020), it is probably better to put the runtime of both models in the table.\n\n# Presentation\n\nThe paper overall reads well.\n\nFigure 1 is certainly helpful. However, it is probably better to write out the explicit forward computation with equations.\n\nAlgorithm 1 and 2 are presented in many other papers, for example (Kong et al., 2015). It might be better to save the space for more experiments or more analysis.\n\nI'm a little surprised that the application of semi-CRF on speech is not mentioned in the related work. Speech input is actually more similar to the input for piano transcription, than to say named-entity recognition. See for example the paper below and the citations therein.\n\nEnd-to-end neural segmental models for speech recognition\nHao Tang, Liang Lu, Lingpeng Kong, Kevin Gimpel, Karen Livescu, Chris Dyer, Noah A. Smith, Steve Renals\nIEEE Journal of Selected Topics in Signal Processing, 2017\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Perhaps the system can be used to judge ones piano performance, and that could have a negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "XpzTm-vQBsK", "reviewer": "Reviewer_nVTb", "summary": "This paper discusses a novel approach to piano transcription which involves predicting notes as events rather than frame-level activity. This approach aligns well with the eventual goal of music transcription, i.e., transcribing musical audio to some form of music notation, e.g.: MIDI or staff which are both based on note events. The method proposed utilizes semi-Markov CRFs which have been utilized to directly predict note/pedal events as non-overlapping continuous intervals in the audio. Utilizing standard metrics for piano transcription, the authors show that their method outperforms current state-of-the-art methods which perform frame-wise transcription.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The approach presented in this paper steps away from recent method for piano transcription which are often improvements of the Onsets and Frames paper by Hawthorne et al. This approach paves a way for modern event-based transcription and may inspire more researchers to approach similar problems using the framework shown here.\n- The results of this approach are better than current SOTA which are for frame-based transcription.\n- The method is well explained and evaluated using the standard dataset (MAESTRO) and metrics. \n\nWeaknesses:\n- I do not see many issues with this paper in terms of methodology. One downside to this paper can possibly be w.r.t broader interest in the ML community. The authors do a good job of framing an important problem in the music information retrieval space in the framework of semi-Markov CRFs which is novel, but in terms of algorithmic contribution, I am not sure if there is much novelty. This paper might better suit the audience of conferences such as ICASSP or ISMIR since the scope is pretty narrow.\n\nOverall Impression:\n- As mentioned above, this paper presents a novel approach to solving the piano transcription problem. The method is well explained and evaluated thoroughly leading to improvements over current SOTA. My only criticism towards this paper in the context of this conference is that perhaps the contribution is too narrow in scope. Therefore I rate this paper marginally above acceptance.\n\nComments:\n- Line 114: Section XXX -> Section 3.3\n- The writing can be organized a little better mainly in Section 3. There is some mixup of algorithm details and implementation details which might be better left separate. This is simply a stylistic suggestion. Feel free to ignore.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors do not discuss any potential negative societal impacts of their work. I am unable to think of negative impacts of algorithms for piano transcription. Perhaps the authors can discuss the impact of automatic music transcription (AMT) algorithms broadly. For example, musicians oftentimes teach lessons or sell transcriptions of their works (e.g. via Sheet Happens Publishing) and AMT systems can potentially cut into their earnings if used maliciously.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "ULl0iLHCo8r", "reviewer": "Reviewer_LiUp", "summary": "They propose an \"event-based prediction\" (instead of relying on frame-level estimates) for piano transcription. They formalize this idea via a semi-markov conditional random fields (Semi-CRF) – since, in the author's words, \"the Semi-CRF layer promotes a direct formulation for event (interval) prediction, eliminating the need for estimating activations at the frame-level followed by post processing\".", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Their method obtains comparable results to the current state-of-the-art, arguing that their approach is faster than state-of-the-art. However they do not report run times for the state-of-the-art models for a fair comparison.\n\nI also don't couldn't identify substantial machine learning contribution, because the method they employ (semi-CRF) were already proposed in 2004. Further, their empirical results are not significantly improving current state-of-the-art performance. However, the idea of using semi-CRF for this tasks (relying on events instead of frame-level estimates) is novel and interesting. For this reason, I'd recommend to submit into a music (e.g., ISMIR) or audio (e.g., ICASSP) research venues.\n\nMinor comments:\n- Line 16: \"while still quadratic in complexity\" -> this implies a (not presented nor mentioned) baseline that is quadratic in complexity.\n- Line 114: \"(See Section XXX)\".\n- It was unclear when you were referring to \"non-overlapping\". Overlap in time or in frequency or pitches overlap?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations: the model it's not very novel from the machine learning perspective, and the results do not clearly outperform the state-of-the-art.\n\nSocietal Impact: music transcription, if solved, could be a very useful intermediate tool for music information retrieval. Together with music source separation, are considered the \"holy grail\" since solving these two tasks could help many relevant tasks and applications.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "3mXaUNjpq86", "reviewer": "Reviewer_1vpu", "summary": "This paper applies a semi-CRF to the music transcription task, with experimental results on the MAESTRO piano dataset.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I found this paper difficult to read, to an extent that challenged my ability understand all the technical details. I will do my best to review the content below, with an understanding that I may have misunderstood certain aspects of this paper. I have a strong background in music transcription, and some familiarity with the mathematics of CRF's, so I believe this paper is objectively quite confusing. For that reason alone, I recommend against its publication in this current form.\n\nTo the best of my understanding, this paper does not contain a significant contribution to the Semi-CRF literature. Therefore, it should be evaluated on the strength of its contribution to music transcription. The contribution to transcription has two components: (1) a reframing of the transcription task that \"skips the frame-level\" and (2) reasonably strong empirical transcription results on the MAESTRO dataset. I discuss the empirical results at the end of the review.\n\n\n\n### The Semi-CRF Implementation\n\nI like the framing of music transcription as a sparse interval labeling problem. It less clear to me why this Semi-CRF implementation is a useful way to accomplish this. The Semi-CRF described in this paper is crippled in two significant ways that seem to undermine the advantages of using such a model. First, an independent Semi-CRF is constructed for each event type (pitch class) so by design this model does not model note correlations. Second, this implementation is a zeroth order Semi-CRF so it also does not model temporal correlations. I understand that these design restrictions may be necessary to adapt the problem to a Semi-CRF's structure, but I would have liked to see some motivating discussions for the decision to use a Semi-CRF, versus e.g. the seq2seq models that have replaced CRF's in many problem domains.\n\n\nI don't understand what the parameters \"theta\" are in Equation 1. A traditional CRF is a log-linear model, so I initially guessed that \"theta\" was a set of weights on pre-trained feature vectors generated by the score and score_eps models. But on closer reading, I think these are actually the parameters of the score models themselves? In that case, I guess you are training the CRF end-to-end with the neural score and contextual models? That suggests to me that you are computing gradients via some hybrid approach, wherein you compute partial derivatives w.r.t. the neural model outputs via analytical expressions (Algorithm 1) and then chain these calculations together with derivatives of the neural model outputs w.r.t. their parameters. Whether or not this guess is correct, the current description is very confusing!\n\nRelated to these questions about the parameters and calculation of gradients: I would also be interested if the authors could provide any insight about the claim (lines 137-139) that Algorithm 1 is faster than autodiff. Autodiff should be a small constant factor times the cost of a forward pass, so unless there really is just a big win on small constants here (e.g. 0-2x analytical versus 4-5x autodiff) this claim is somewhat surprising. Is this a well-established fact in the CRF literature that analytical gradients are faster than autodiff?\n\n\nI don't entirely understand the asymmetry in handling event (score) versus no-event (score_eps) labels. Rather than having these \"empty\" no-event positions, couldn't we use a dense labeling of event and no-event intervals? This seems like it would be more efficient too: rather than evaluating score_eps at every unlabeled frame, we would make just one calculation for each contiguous chunk of time without an event. If this isn't possible, then there should be a clearer discussion of why this modeling choice was necessary.\n\n\nIt looks like there is a single contextual model shared by all the upstream score and attribute models. And there are distinct score and attribute models for each event type that do not share weights with each other. Is that correct?\n\n\n### Empirical Results\n\n\nI find the empirical state-of-the-art claims slightly overstated and underanalyzed. Results are significantly better than the Kong et al. baseline for Note w/ Offset and Note w/ Offset & Velocity metrics, but only marginally better for the frame wise metric (90.81 vs 90.17) and marginally worse under note onset measurement (96.31 vs 96.77). Sustain pedal detection is significantly worse than the Kong et al. baseline. It makes some intuitive sense to me that the note-based metrics might favor the Semi-CRF, but the paper doesn't make an argument about why we should objectively prefer these metrics. I would have liked to see an analysis about the source of disagreement among these metrics (which a priori, we might expect to be very correlated). This might provide evidence for why we should prefer a certain type of metric, and potentially support the SOTA claims that only really hold up for the note-based metrics.\n\n\n### Post-Rebuttal and Discussion\n\n\nThanks for your detailed response. I appreciate the quantitative analysis of claims about analytical vs. autodiff implementations. There is some lingering confusion among the reviewers (myself included) about *why* this would be the case, and why the DP implementation is the bottleneck in these systems. Further analysis of these questions (in addition to the empirical statistics) could bolster the machine learning contribution of this work and complement the music contribution. This might help with selling this work to the ML community. \n\nYour response about temporal and harmonic correlations is reasonable, and I think it would really help to add this as motivating discussion to the paper (possibly even in the introduction). This would help to clarify what you're doing, to justify why you're doing it this way, and to avoid surprising reviewers/readers with independence assumptions later on.\n\nI have increased my score to a 5, in light of (1) the stronger empirical transcription results, (2) the empirical support for claims about the DP implementation, and (3) promises of improvements to the paper's clarity & exposition. Because I can't evaluate the extent of the writing improvements (3), I am reluctant to give a stronger endorsement to the current version. But I hope you don't take my review too negatively: I think this is good work, and with some more polish I think this will become a very good paper.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates piano transcription based on semi-CRF models.  This approach views music note labeling as an event prediction problem.  The semi-CRF is used to score the non-overlapping continuous intervals to avoid conventionally used frame-level local decision.   Experiments are conducted on the MAESTRO  dataset and competing results are reported by the authors.   All reviewers agree that the idea of applying semi-CRF to music transcription is novel to the application domain but raise concerns on the implementation of semi-CRF.  For instance, temporal correlation of events is not considered in the current semi-CRF model.  Also, it is not clear why autodiff loses advantages over the authors' implementation in speed.  The rebuttal is good which answers the majority of the questions.  Overall, the idea seems to be interesting to the music transcription but the revision to be made in responses to the concerns seems to be significant. Given its current form and some lingering concerns, I would recommend the authors revise and resubmit the work to a future conference. 
Decision: Accept (Poster)