Skipping the Frame-Level: Event-Based Piano Transcription With Neural Semi-CRFs
Authors: Yujia Yan
Frank Cwitkowitz
Zhiyao Duan
Venue: NeurIPS 2021 Poster (2021)
Keywords: Music
Audio
Piano Transcrition
Music Transcription
Semi-Markov
CRFs
Sound Event Detection
Music Information Retrieval

Abstract: Piano transcription systems are typically optimized to estimate pitch activity at each frame of audio. They are often followed by carefully designed heuristics and post-processing algorithms to estimate note events from the frame-level predictions. Recent methods have also framed piano transcription as a multi-task learning problem, where the activation of different stages of a note event are estimated independently. These practices are not well aligned with the desired outcome of the task, which is the specification of note intervals as holistic events, rather than the aggregation of disjoint observations. In this work, we propose a novel formulation of piano transcription, which is optimized to directly predict note events. Our method is based on Semi-Markov Conditional Random Fields (semi-CRF), which produce scores for intervals rather than individual frames. When formulating piano transcription in this way, we eliminate the need to rely on disjoint frame-level estimates for different stages of a note event. We conduct experiments on the MAESTRO dataset and demonstrate that the proposed model surpasses the current state-of-the-art for piano transcription. Our results suggest that the semi-CRF output layer, while still quadratic in complexity, is a simple, fast and well-performing solution for event-based prediction, and may lead to similar success in other areas which currently rely on frame-level estimates.

Skipping the Frame-Level: Event-Based Piano Transcription With Neural Semi-CRFs
Yujia Yan', Frank Cwitkowitz, Zhiyao Duan
Department of Electrical and Computer Engineering
University of Rochester
Rochester, NY, 14627 USA
yujia. yriia.ya@@ochssterr. edu, fcwi koowurr www.hester.eeeee zhiyao www.oodcunnnnteteeeeee.. edu
Abstract
Piano transcription systems are typically optimized to estimate pitch activity at each frame of audio. They often followed by carefully designed heuristics and post-processing algorithms to estimate note events from the frame-level predic- tions. Recent methods have also framed piano transcription ing problem, where the activation of different stages of a note event are estimated independently. These practices are not well aligned with the desired outcome of the task, which is the specification of note intervals as holistic events, rather than the aggregation of disjoint observations. lation of piano transcription, which is optimized to directly predict note events. Our method is based on Semi-Markov Conditional Random Fields (semi-CRF), which produce scores for intervals rather than individual frames. When formu- lating piano transcription in this way, we eliminate the need to rely on disjoint frame-level estimates for different stages of a note event. We conduct experiments on the MAESTRO dataset and demonstrate that the proposed model surpasses the current state-of-the-are for piano transcription. Our results suggest that the semi- CRF output layer, while still quadratic in complexity, is a simple, fast and well- performing solution for event-based prediction, and may lead to similar success in other areas which currently rely on frame-level estimates.
multi-task learn-
this work, we propose a novel formu-
1 Introduction
The task of Automatic Music Transcription (AMT) aims to transcribe music recording into some form of music notation Benetos et all 2018). Examples of notation include MIDI event sequences, e.g, /Hawthorne all [2018), Kong et all [200], Kim and Bello [201%) Kwon alll [2020], and staff notation, e.g., Nakamura et all [008), Román etl 2019). In this work, we address the problem of transeribing piano music into MIDI event sequence. MIDI transcription involves constructing sequence of events, each specified by its onset and offset positions, with the constraint that two events of the same event type (e.g., certain pitches and pedals) cannot overlap. addition to onsets and offsets, the velocity (i.e., value that represents the intensity of a key strike, which informs the loudness) associated with each event often estimated.
In recent years, neural network based approaches have reached the state of the art for the problem of piano transcription, e.g., Hawthoree et all [201]] Kong all [2020], Kwon et [022]]. They operate at the frame-level and make predictions for different stages of note event, i.e., the onset, offset, and pitch activation, separately. order to extract note-level predictions, they use manually designed procedures to combine the disjoint frame-level predictions. These include thresholding,
"Code available at
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

peak picking [Hawthorne at 2018, Kong et all 2021 or low-order hidden Markov models [K stan 2020]. This two-stage approach requires manually crafted procedures and a manually designed state structure modeling the temporal evolution of notes.
In this work, we propose a direct approach note-level transcription. Instead of scoring and aggre- gating note activations across individual time frames, our approach directly scores a time interval's likelihood of covering the entire process of a certain event (c.g., note, pedal usage). This greatly simplifies the formulation of piano transcription. Specifically, the proposed method takes the log- mel spectrogram of an audio segment (e.g., 10s) as input, and uses contextual model to produce contextual embeddings for each frame. It then uses score model to score each possible time interval within the segment assess whether it covers the entire span of musical event (e.g., notes, ped- als) as specified by a specialized zeroth-order semi- Mani-M conditional random fields (semi-CRF) [Sarawagi and Cohen, 200))). These scores are then decoded using Viterbi algorithm into set of non-overlapping intervals for each event type, i.e., note-level transcription. Finally, other attributes of these intervals (i.e., velocity and refined boundary positions) are also estimated.
Previous works using semi-CRF were concerned with shorter sequence lengths and smaller interval lengths [Kong et all 2016, emmeral all 2016, Kemos etall 2019, etll 2016), and usually only attempted to estimate a single channel (track) of non-overlapping events. In contrast, our problem deals with sequences of 400-1600 frames (approximately 10s-40s) and requires a separate semi- CRF for around 90 different labelling channels (88 pitches + 1-3 pedals), each corresponding to a specific event type. Another challenge the wide range of event duration that can be encountered in piano music, which further increases the computational expenses of the problem. These chal- lenges prevented us from using existing general purpose semi-CRF methods due to computational expenses. Instead, we introduce a zeroth-order Semi-CRF formulation that is specifically adapted to the problem and show that can be implemented efficiently (see Table L) for the problem size of interest.
Our experiments on the MAESTRO dataset show that the proposed system achieves Note w/ Offiet Fi of 78% and Note w/ Offset & Velocity F1 of 87.75%, exceeding the current state of the art while being smaller and faster. We believe that this simple, fast and well-peefornning approach is also extensible to other similar tasks with intervals as the prediction target, such as polyphonic sound event detection or speaker diarization.
2 Related Works
Piano Transcription There have been many proposed approaches to piano transcription in the last several decades. Early works consist of simple signal processing methods such as spectral peak- picking [Klapuri et all, 2000, Bello et all, 2006]. which estimates fundamental frequencies directly from the spectrum, or spectral decomposition [Smaragdds and Brown, 2003, HaHaan and Plumbz Jey, 2014]] where a short-time spectrum is factorized into spectral components with corresponding activations. Several parametric models have also been proposed to perform spectro-temporal decom- position [|mmcett et all 200%. Emiya et all, 2009 Cheng et all 2016]. Time-domain decomposition has been also investigated in Cogliati et all explicitly as parametric templates, and tend to suffer from the lack of generalization. Other meth- ods include machine learning techniques such as Hidden Markov Models [Raphael, 2002, Böck and
2017 201G). Most of these methods aim to model notes
Schedl, 2012) or Support Vector Machines [Poliner and BiTrs 2006, WWninger et all 2013). Recently, neural network based approaches have made significant advances [Sigtia et all, 2016, Hawthore all 2018, 2019, Kim and Bella, 201%) by taking advantage of large Disklavier piano databases.
Many methods separate piano transcription into frame-wise polyphonic pitch estimation and note tracking, thereby optimizing for frame-level predictions which are used to estimate quantized note intervals. While intuitive, this practice has a major drawback: the quality of the note-level predic- tions is fundamentally limited by the quality of the frame-level activations, which can be sporadic, discontinuous, and non-uniform in terms of their strength. Kameoka all [[(X0Z]] attempted to cir- cumvent this problem by performing harmonic temporal template structured clustering to explain spectrogram observations as originating from distinct sources, directly estimating notes as discrete events. This reflects the idea of [Brrggaan [990], i.e., that meaningful auditory events are results of simultaneous and sequential groupings. However, the underlying mental process of low-level group- ings more complex than clustering according to simple parametric harmonic temporal templates.

---

Toui
EventType (notes/ppdals)
log-mel spectrum
N
Score,
Decode
Contextual Model
Score Model
Refined
Altributes:
Velocity
OnseV/OlTsel position
N
Tesi
Thegin
Extracted
Interval:
Time
Score
Proposed system overview. In the middle part of the figure. Thegnn and Tend are the number of beginning positions and ending positions, respectively, and N is the number of eventType(s).
Figure I:
Some more recent works [Kelz et all 2019 Kwon et all 2020| attempt to incorporate manually- defined state structure for modeling the temporal evolution of notes. However, the state structure (usually of low order) may reflect the actual temporal evolution and thus limits the representa- tive power of these models. Coglati el al. [207]] directly estimates the activation note templates in the time domain, but these templates are preset and are piano- and context-dependent, lacking the generalization to unseen pianos or recording environments.
The proposed method learns estimate note events holistically, rather than adopting the frame-to- note two-stage approach or performing and aggregating multiple subtasks (i.c., onset, offset, and pitch estimation). It can be viewed performing groupings directly in the event space, with low- level details being learned by the neural networks.
Semi-Markov CRFs semi-Markov conditional random field (semi-CRF, Sarawagr and Cohen
defines conditional probability distribution sets of non-overlapping labeled intervals
within an imput sequence. Semi-CRFs have been used in Chinese word segmentation [Lw ef all
Kong et al, (0166, named entity recognition [Zhun 2016. Ye and Ling, 2018. Arora et
2019
character-level part-of-speech tagging |emos et all 2019), phone recognition [Lu et all
2016 chord recognition [Masada and Bunescu. 2017)), etc. The time complexity for computing the
partition function and inferring the most likely configuration is quadratic with respect to the length of the sequence. Therefore, most works set an upper bound on the length of single interval, e.g..
Kemos et all [201%], to make the computation tractable.
In this work, semi-CRF is defined for each semi-CRF formulation differs from the standard formulation in the following ways: 1) Different events in semi-CRF are allowed overlap on their endpoints (boundary frames), since audio frames represent time period and can contain both the end and beginning of an event of the same type. This treatment allows to use larger hop size without adjustment due overlap in the same frame, so that the sequence length for the same audio segment can be reduced; 2) A frame is not required to belong to any event. Despite the loosened definition, we use the name semi-CRF refer to this type of structured prediction module.
type (i.e., note of specific pitch, a pedal). Our
Compared to the tasks mentioned above, the task of piano transcription has longer input sequence and a larger range of possible event duration. The former makes the time complexity issue more prominent, while the latter makes it difficult specify upper bound event duration for speedup purposes. We show that our formulation is efficient for piano transcription and hence does not require an upper bound on event duration, thanks set of implementation optimizations.
3 Proposed Semi-CRF Approach to Piano Transcription
The proposed system transcribes the input audio into list of musical events, i.e., notes and pedals. Here the term events refers to acoustical events that span certain time intervals (not to be confused with raw MIDI events, i.c., note on, note off, cc, etc., used for serializing the performance). Taking
3

---

an audio segment (e.g,, 10s), the transcription process is illustrated in Figure M1 First, log-mel spectrogram is computed after taking the short-time Fourier transform (STFT) of the audio. This is used as input to a contextual model to produce contextual features across time frames. Such features are then fed a score model to calculate two kinds of scores. The first kind of score (Score) assigns score to indicate whether an interval is an event of a certain event type. The second kind of score (Score,) indicates whether an interval that spans two frames is not part of any event for a certain event type. Finally, Viterbi algorithm is used to decode the most likely sequence of events for each event type, using the aforementioned scores. Autributes, i.e. velocity and refined onset/offset positions, are then estimated for each extracted event.
3.1 CRF Formulation
Let X' =< Io, 21. IN-1 > be an audio segment containing a sequence of N time frames. Let )' = ( even(Type), i S j} be the set of musical events entirely contained within this segment, with time quantized to audio frames. Here i and are respectively the beginning and ending frame indices for each event, and eventType is the type of the event, e.g., a specific key (pitch) of the the 88 keys of a piano or the sustain pedal. Events that extend outside the audio segment are not considered in this formulation, but will be handled the inference process (See Section (.3) We assume that for the same event type, two events A and B are non-overlapping, i.e., either jA S iB or jB S iA- In this formulation, it is allowed to have single-frame events where i j (note how this would make the formulation differ from the standard semi-CRFs).
We associate each event with a set of attributes: 1) the relative non-quantized position of onsets and offsets at the sub-frame level, represented by a value from -0.5 to 0.5 indicating the position relative to the quantized index of the frame, and 2) the MIDI velocity of the event, represented by a discrete value from 0 to 127.
We use Deventlype to denote the subset of events that contains only specific event type. For each event type, we model the following conditional probability:
p(Veventlypll (2)
exp
score(i. eventT
Z( ((veenTyype)
(v.jevenrfyyee) ED eootttyp
score, (i - 1, even/Type) i,eee//yyyT ].
Σ
+
[i=1,t] not covered in Veuuiioe
where score(i, eecei...eeettttppp. ype) assigns a score to the interval [ to indicate whether it is an event of eveníType, score, 1, i, eveníType) assigns a score to the interval [ 1, that spans two frames to indicate whether it is not covered by any event of even/Type, which inversely represents the frame-level activation for an event, and Z(even/Type) is the normalization factor. The non-event score (score,) for two consecutive frames comes from skipping one position in feasible solution (see example in the next paragraph), and it also allows the model to degenerate to frame-level event prediction. Here, for notational convenience, we omit X' for every term.
For numerical stability, Eqn. (II is computed in the log-domain. The exponent, i.e., the summation of all score(t. j, event/ype) and score, 1, evenrType) corresponding to Veventyyp, is the unnor- malized log- ligeeeilioood. As an example, for single eventType, assuming there are 7 audio frames 1,2.3,23, 3,5,6,66> in total, for the interval set candidate [[0.0], [2,4], [4,5]], the corresponding unnor- malized log-likelihooo is computed as score(0, 0) + score, (0, 1) + score, (1, + score(2, 4) score(4, 5) + score, 6).
The computation of log (event ype) and its gradient w.r.t. to score/score, log Z(eventT ype). is critical in training. The detailed procedure is shown in Algorithm II woorward-bbckwaard algorithm). In practice, we compute log 2 and V log Z for all eventType(s) in parallel. The forward stage and the backward stage in the forward- backaard algorithm are batched together, but with all positions of the input flipped, as to compute them in a single pass without the need for two separate update equations, since their calculations are essentially the same. We also find that substantial speedup (>5x) can be achieved by using custom gradient computation via the backward pass of the forward- backward algorithm for log Z, as opposed to using automatic differentiation w.r.t. log Z in PyTorch. This is likely due to a more contiguous memory access pattern and the elimination of unnecessary operations.

---

In order to make memory access at each step more contiguous, the event scores Score and Score, are organized as T Tnd x T start& N and Tend N tensors, respectively, with Tend being the ending position of an interval, starr being the beginning position of an interval, and N being the number of even(Type(s). Here, each slice along the N dimension represents separate semi-CRF channel.
A runtime benchmark of the semi-CRF layer presented is provided in Table B
Algorithm 1 Norward-backward algorithm for log Z and V log Z for specific event type.
Input: function score(i, function score, 1,
Output:
log Z and V log Z
Forward stage:
Initialize the forward variable: u(0) <- log(exp(score(0.0) + 1) for all = -
do
v(i) + log { explo(j - 1) (j 1,5)] + Σ
explu(k) + score(k..)]] bマj
v(i) te( + log(1 exp[score
end for
Readout the log partition function: log Z + v(N 1)
Backward stage:
Initialize the backward variable: g(N 1) + -bog/oxplscor(N--1,, 1,N 1)] + for all j = N 2..... do
((1) + log { exp (j expia(i
+ 1) score 1)] Σ
expla(k) + score(j. kzj
<(i) ( + log(1 + exp(score(j..i))
end for
Read out the posterior marginals as derivatives:
for all i=0...., N-1 do
p(i, i) + ((() + q(i) + score(i, i) - loglexp(score((,) i) + 1)] log } end for
for all i<j do
p(i,j) t exp (u()) {(j) score(i. j) log
end for for all i=l...., N-1 do
(i - 1, t explo(i - 1) + q(i) + score, (i - 1, - log Z end for
8 log Z [ecore(f]] p(i,j); Dscore, 1,3 pe(i - 1,
log
3.2 Training Objectives
For training, we use maximum likelihood estimation (MLE), where the conditional log- likeliodd defined to consolidate the conditional probability in Eqn. (II) over all event types, assuming their conditional independence given X::
logp()"(2'= =
Σ log ((Neoou7spp
evenTTpe
One may question the validity of this conditional independence assumption on different event- Type(s); here, we keep things simple, as previous approaches often make this assumption. We leave this for future investigation.
5

---

In addition to the log- log-likelihood for the presence of events defined in Eqn. (2), we also learn to predict three attributes for each event:
log p(attributes]e) (e) = log p(velocityle) + log p(refined onsetle) + log p(refined offset /e).
(3)
Here we use to denote an event. We parameterize these terms with the following distributions, i.e., the Softmax/Multinomial distribution, and the Continuous Bernoulli distribution Loaiza-GGanem and Cunnngham, 2019:
velocityle Softmax(p(ee),,
(4)
0.5 + refined onset/offser ~ ContinuousBermouli(((c...
where (((e) and A(e) parameters produced by neural networks that take features of the interval as the input. The interval features used in this work are described in Section 14. The final objective defined as
L = [log p())))) + Σ log p(atribbbut.)ss))..... eEy
(5)
More discriminative and cost-sensitive losses such as max- mar-in and softmax margin can be used as drop-in replacements, but we leave this investigation future work.
3.3 Inference
We use dynamic programming (Viterbi) to infer the most likely interval sequence for every eventType independently. The procedure is described in Algorithm A
When processing longer audio recordings, our system transcribes audio segment by segment. Be- cause the system is designed to ignore all events that extend outside a segment, these audio segments need overlap with each other. In this work, segments are 10s long and the overlap is 5s. In order to properly handle notes near the boundaries of a segment, we modify the algorithm such that it forced to take the result from the overlapping portions into account as follows. Within an audio segment, decoding is performed in reverse order: backtracking starts from the position immediately after the last event decoded in the last segment of the same event type, if any of those overlap with the current segment. We found that, for each segment, it works slightly better to discard all events of which the onsets fall into the overlapping region with the next segment, which is procedure similar to the overlap-discart method for computing discrete convolutions for long signal.
Algorithm 2 Viterbi (MAP) decoding of specific eventType within audio segment.
Input: function score(i, function score, (i 1) backtracking starting frame position Output: a set of intervals y vNN 1) f maxx(scoreAA 1, 1). for all; j E N - 2............................... do
vij + 1) + score, + skip if inactive (o(k) + score k)} if an interval v(j) + v(j) + max((core((,))))m 0) single frame case end for
v(j) + max
Perform backtracking starting from position / to get y
After events are extracted, feature vectors of the events, described in Section KZL, are used predict attributes, namely, velocity and refined onset/offset positions. These attributes are then assembled with the events to form event tuple (onset, offset, eveníType, veloctty for the final output.
3.4 Model Architectures
In this section, we present details of the neural architectures for the three components used in the proposed approach: 1) contextual model, 2) score models, and 3) attribute predictors. To recap, we first apply contextual model transform the input audio frames into sequence of contextual embeddings. These contextual embeddings are then used by the score models to compute score(-) for possible events, in a fashion similar to Cross and Huang (201), Kilaey and Klein [2018]] Lm at [20]A], and (.). After events are extracted, contextual embeddings are also used by the attribute predictors to predict attributes (i.e., velocity and refined onset/offset times) associated with each event.

---

ConvBlock
Contextual embeddings:
Ray Rx-1
(++)
X
Contextual Model
Tayin
MLPLaayer
PLayer
Sonne, (1
1,4,))
by., bv Bys Sreeathee.
Dhe Rg, he hy. Rovertpe
Input frames 2gy 4--1
MLyyeyer
MLPLayer
Nh by @
Bu.1.hu. h.]
Tirst Free moments of hij:
(a) Contextual model
(b) Score models
(c) Attribute predictors
Figure 2: Model architectures. The contextual model transforms the input frame sequence into sequence of context embeddings, based on which scores and attributes are computed by the score models and attribute predictors, respectively.
3.4.1 Contextual Model
As input to the contextual model, we compute a log-mel spectrogram with frame size of 4096 samples, a hop size of 1024 samples, and the Hann window function. Following Hawthorne et all [2U18] Kong all [202]] use 229-bands with frequency range from 30 Hz to 8000 Hz. In ad- dition to a single Hann window applied to each frame, we also experimented with applying multiple adjustable Gaussian windows (5 windows, initialized with equidistant centers and constant variance) to the same frame and concatenating the resulting log-mel spectra together for improving the tem- poral resolution of the spectra input. By doing this, the input becomes log-mel spectrogram with multiple channels for each fimeuecyy bin, with each channel obtained from a different window function. We feed the log-mel spectrogram through four convolutional blocks with kernel size of 3 and 48, 64. 92, 128 filters, respectively. Each convolutional block contains two 2-d convolution lay- ers, followed by batch normalization and a Gaussian Error Linear Unit (GELU) activation functioon Hendrycks and Gimpel, 2016). At the end of each block, the output is pooled along the frequency dimension using average pooling with a kernel size and stride of 2. The channel and frequency dimensions are flattened together and fed into two-layer bidirectional GRU with hidden size 256. This contextual model largely the same as the one used in Kong et all [2020] however, we only utilize one instance of the block rather than stacking it for different frame-level prediction targets.
3.4.2 Score(i, and Score, - 1,,
As shown in Fig. 2R the features used for scoring an interval [ include contextual embeddings at its two endpoints, i.e., h; hy. their elementwise multiplication h, hj, and the first three moments for the contextual embeddings within the interval [. These features are chosen because they provide information for events whole while being cheap compute, and are often seen in the literature when extracting holistic/pairwise features. These features are concatenated and fed into a three-layer feed-forward neural network (MLPLayer) obtain raw intervalic score tensor with shape Tend X T beginn x N, where the three dimensions are the ending position of an interval, the beginning position of interval, and eventType channels, respectively. The output size of this network is equal to the number of event types and the hidden size is equal to 4x the output size. On top of the raw score tensor, as shown the figure, we experimented with applying a simple convolutional block with two convolutional layers of kernel size 3 and filter size being 3x the output size, with the hope that the block can aggregate neighboring information in the space of intervals in order to increase the expressivity. We also experimented with scaling the intervallic score by the length of the interval, motivated by the following observation: when the score function for an interval is chosen to be the variance alone with this length scaling added, the SemiCRF layer in this
7

---

work is equivalent to optimal 1-d k- k-means clustering. This scaling adjusts the score for event to be roughly at the same scale of the aggregated non-event score at the same time interval at imitialization.
For the inactivity score, score, (i 1,4), only contextual embeddings two consecutive frames are used. These similarly fed into three-layer feed-forward network with output size equal to the number of event types.
3.4.3 Attribute Predictor
As shown in Fig. 2d, for predicting attributes associated with each event, we use features of end- points concatenated with an trainable evenTTye embedding vector SeventType with size 256 for fully specifyyng the event. These features are fed into separate multi-layer feed-forwar networks. The velocity prediction network has hidden sizes of 512/512, and produces a 128-dimensional logits to infer the MIDI velocity value. The refined onset and offset time prediction network has hidden sizes of 512/128, and produces a 2-dimensional real vector, which is used as logits for the Continuous Bernoulli Distribution for onset/offset refinement, respectively, defined in Eqn. (9))
4 Experiments
4.1
Dataset
We conduct experiments using the MAESTRO v2 dataset [Hawthome all 2019, which con- tains around 200 hours of MIDII-syncronnzeed (3ms precision) virtuoso piano performance record- ings. The recordings were collected across several years of the International Piano-e- Piano-e-Comper Competition, and were recorded on Yamaha Disklavier pianos. All recordings are sampled at 44.1 kHz, except for files from the 2017 and 2018 competitions, which are sampled at 48 kHz; we thus downsample them to 44.1 kHz for consistency. For comparing with other works, we follow the convention to extend the offset of notes to the offset of any simultancous sustain pedal event [HHathorne: all 2018, Kong et al. 2020].
4.2 Training
We use batch size of 12 and Adabelief /hhuang al. 2020] optimizer with weight decay of le-4. We use oneCycle [Smith and Topin, 2019 learning rate scheduler with maximum learning rate set to 6e-4 for 180k iterations and cosine annealing. The learning rate is increased gradually for 20% of iterations and then gradually annealed 1.5e-5. We automatically determine the value for gradient clipping by using the 0.8 quantile of the gradient norm during the last 10k iterations, which a strategy similar to Seetharaman et all [021 We apply dropout with rate 0.1 on the attribute predictors and the score model.
4.3 Evaluation Metrics
We follow the standard piano transcription evaluation procedure, validating note predictions with multiple levels of criteria. The most basic metric considers a note prediction correct if the esti- mated onset is within 50 ms of the corresponding ground-truth onset. The next incremental metric additionally requires that the offset prediction is within 50 ms or 20% of the note duration from the ground-truth offset position. The final incremental metric additionally requires that the velocity estimate is within a tolerance of 0.1 ([0,1) normalized velocity) from the ground-truth velocity, as defined in Hawthorne et [2018]. We directly use the implementation contained in the mir_eval library [attel all 2014 to compute these three note- metrics.
We also introduce activation-level metries which serve hopsize- aagiizze-ggssccccccc replacement for the commonly reported frame-level metrics for evaluating how predicted time spans events overlap with the ground truth (see the supplementary material for details).
Following the convention, each result is averaged across all pieces within the test set. We compute similar metrics, minus the velocity variation, for the sustain pedal activity, as in Kong all [02)].

---

4.4
Main Results
We compare the proposed system the state-of-the-ar methods for piano transcription using the MAESTRO v2 test split. We recompute these metrics for other systems directly from the transcribed MIDI files generated by their pretrained models. We also report our results for our model trained and evaluated on the MAESTRO v3 splits for future reference. The note transcription and pedal transcription results are listed in Tables and Q. respectively.
Activation
11
Note Onset
Note w/ Offset
Note w/ Offset & Vel.
Method
P
R
P
P
1
R
A
P
R
Fi
P
R
Fi
MAESTRO v2
Hawthorne et all [2019]
86,84
89.24
87.82
97.88
92.26
94.93
82,09
77.44
79.65
78.37
73,94
76.05
Kong all [2020]
90.09
90.42
90.15
98. 16
95.46
96.77
85.65
83.32
84.45
84.18
81.92
83.02
Proposed w/o Extra Win
93.84
88.07
90.75
98.85
93.97
96,31
90.67
86.24
88.37
89.60
85,25
87.34
Proposed w/o Post Conv
93,90
88.18
90.85
98.86
94.06
96.36
90.52
86.18
88.26
89.49
85.23
87.27
Proposed w/o Len Scaling
93.85
88.25
90.87
98,80
94.15
96.39
90.70
86.48
88.51
89.66
85.51
87.50
Proposed Proposed (batchsize=20)
93.84
88.48
90.98
98.78
94.18
96.39
90.79
86.62
88.
89.78
85,68
87.65
93.85
88.72
91.11
98.66
94.50
96.51
90.68
86.89
88.72
89.68
85.96
87.75
MAESTRO
Proposed (batchsize= 20)
93.79
88.36
90.75
98.69
93.96
96.11
90.79
86.46
88.42
89.78
85.51
87.44
Table 1: Piano transcription note results for the proposed methods and various related works.
Activation R
Onset
Onset & Offset
Method
Fi
Fi
Fi
MAESTRO v2
Kong et al. [2020]] Proposed w/o Extra Win
94.14
94.29
94.11
77.43
78.19
77.71
73.56
74.21
73.81
95.29
86.36
90.02
81.56
73.47
76.99
78.14
70.50
73.83
Proposed w/o Post Conv
95.19
87.05
90.36
81.72
73.29
76.99
77.99
70.05
73.54
Proposed w/o Len Scaling
94.73
87.32
90.34
80.79
73.51
76.71
77.36
70.51
73.53
Proposed Proposed (split) Proposed (batchsize=20)
95.13
87.71
90.73
82.14
74.91
78.10
78.48
71.72
74.71
95.20
89.60
91.84
83.01
77.55
79.98
79.72
74.55
76.85
95.35
87.63
90.78
82.27
75.61
78.55
78.72
72.42
75.20
MAESTRO v3
Proposed (batchsize= 20)
95.17
88.33
90.98
82.18
75.81
78.52
78.75
72.74
75.30
Table 2: Sustain pedal detection results for the proposed methods and various related works.
On MAESTRO v2, the proposed system achieves a Note w/ Offset F1 of 58%72% and a Note w/ Offiser & Velocity F, of 87.75% significantly outperforming previous methods in predicting holistic note events. Regarding the Note Onset Fl score, Kong et all [212]] has the highest performance. Our proposed method slightly underperforms Kong et all [2020], but still outperforms Hawthorne all [2019] by about 1.5%. Regarding the Activation F1 score, our proposed method achieves the highest performance, outperforming Kong et al, [202]] by 1% and outperforming Hawthorne et all [201]] by 3%. This is the case even though the proposed method is not optimized to output frame-level predictions, while the compared methods both have frame-level prediction branch.
The Proposed w/o Extra Wins entry refers to the variation which does not apply extra adjustible windows for creating multiple version of log-mel spectrogram the input. The results indicate that this oversampled log-mel spectrogram input helped improve the system slightly. The Proposed w/o Post Conv entry refers to the variation which does not include the final convolutional block when computing the score matrix. The results indicate that this convolutional block also helped improve the system slightly. The Proposed w/o Len Scaling entry refers to the variation which scales the scores in the intervallic score matrix by the length of the interval. The results indicate that this scaling of intervallic scores also helped improve the system slightly. We also found that increasing the batch size from 12 to 20 improves the result.
Regarding discrepancies with respect to numbers reported in [Kong et al 2020J, we inspected their code and found a bug that affects a small portion of note offsets when handling pedal extension. In our evaluation, we use the correct offset labels, which improves the results of their model slightly. For pedals, they use a onset tolerance of 200ms.

---

In terms of sustain pedal transcription results (Table 2D, our method achieves an Onset F1 of 78.55% and an Onset w/ Offiee F of 75.2%. outperforming Kong et al, [2020] in predicting holistic pedal events. Because Kong et all [202]] uses separate model replicating their note transcription model to predict pedal activity separately, we also experimented with splitting the original architecture into two branches with only the convolutional blocks shared, still training these two tasks (notes and pedals) together. The result is shown in the table as Proposed (split). This variation with two branches for notes and pedals further improves all Fl metrics for pedal transcription. It suggests that there may be some negative transfer between these two tasks and further improvements may be achieved by tuning branches for notes and pedals separately. Kong et all [2020] yields Fi of 94.1 1% for predicting the pedal activation, that is 2% higher than the proposed model, due to its higher recall. This difference may be due to the fact that our system is only trained to make predictions on the holistic events instead of individual frames.
4.5 Runtime Analysis
The proposed model has O(L²N) time complexity, where L is the length of the input sequence, iee. the number of frames, and is the number of eventType(s). The semi-CRF layers involve sequential computation, and traditionally were often considered too slow for practical use with re- spect to long sequences if no restrictions were applicd. Here we benchmark the computation time of several components of the proposed algorithm that have quadratic time complexity, using input sequence of a reasonable length for music transcription. The results are shown in Table E. With a hop size of 1024 samples and sampling rate of 44.1 KHz, 400, 800, and 1600 frames correspond to audio segments of 9.29 s, 18.58 s, and 37.15 s. respectively, which are common lengths used for audio processing. For these lengths, the ratio of the audio length to the time taken by each compo- nent are within the acceptable range. The algorithms were implemented in PyTorch, and we believe that further speedup can be achieved with native C+CCUUA implementation.
400
Frames
800 F Frames
1600
Frames CPU
Components
GPU
CPU
GPU
CPU
GPU
Forward Backward (Py autodiff)
0.39
5.95
4.40
44.69
17.68
354.40
Forward Only
0.05
0.20
0.09
0.80
0.23
3.08
Backwrd-Backward
0.08
0.58
0.12
2.32
0.27
7.12
Viterbi
0.23
0.16
0.27
0.36
0.54
0.88
Pairwise Scores
0.06
0.84
0.25
3.14
0.99
11.76
Table Running time (seconds) of algorithm components that have quadratic time complexity w.r.t.
the input length on Intel(R) Core(TM) i7-7800X CPU @ 3.50 GHz and Nvidia GTX 1080TI. Events for N = 90 eventTypes and for a single audio segment are predicted.
To get a clearer idea on the overall speed of the system, we then compare the entire running time with the previous ofate-of-the-an system transcribing the same audio file, Carl Czerny Grand Sonata 0p.145 145 No.9, which is 33.3 minutes long. For a fair comparison with Kong et al, [2121]] we do not batchify our computations across audio segments for transcribing a single audio file. Our system takes 95s while Kong et al. [2020]] takes 353s.
5 Conclusions
In this work, we proposed a piano transcription system designed to directly predict note events. uses a specialized formulation of a semi-Markov conditional random fields, where the prediction targets are set of non-overlapping events for each piano key and the sustain pedal. Using this for- mulation, we eliminate the need to build post-pros post-processing algorithms or heuristics atop frame-level estimates, as well as the need to aggregate disjoint predictions from multiple tasks, e.g., onset, offset and pitch estimation. The results of our experiments on MAESTRO, popular piano transcription dataset, show that our model significantly outperforms two state-of-the the piano transcription meth- ods on note-level transcription. We show that the proposed method can be implemented efficiently enough for the task domain and that the final system transcribes faster than the existing state of the art system. We believe that this simple, fast, and well-performing approach is extensible to other similar tasks which currently rely on frame-level estimates.
10

---

Funding Disclosure
This work has been partially funded by the National Science Foundation grants No. 184614 and DGE-1122591.. Additional revenues related to this work: Zhiyao Duan's sabbatical leave at Kwai Inc. and part-time activities at Mango Future Group Limited.
References
Ravneet Arora, Chen-Tse Tsai, Ketevan Tsereteli, Prabhanjan Kambadur, and Yi Yang. semi- Markov structured support vector machine model for high-precision named entity recognition. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5862-5866, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19- 1587. URL https: h/pp:://wwww http://wwwwwwwww...///.. aclweb. wtt/ http//ttbologg//9111110
Juan Pablo Bello, Laurent Daudet, and Mark B Sandler. Automatic piano transcription using fre- quency and time-domain information. IEEE Transactions on Audio. Speech, and Language Pro- cessing, ((6):2242-2251. 2006.
Emmanouil Benetos, Simon Dixon, Zhiyao Duan, and Sebastian Ewert. Automatic music transcrip- tion: An overview. IEEE Signal Processing Magazine, 36(1):20-33) 30, 2018.
Sebastian Böck and Markus Schedl. Polyphonic piano note transcription with recurrent neural networks. In 2012 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 121-124. IEEE, 2012.
Albert Bregman. Auditory Scene Analysis: The Perceptual Organization of Sound, volume 95. 01 1990. doi: 10. 10.1124//.404444
Tian Cheng, Matthias Mauch, Emmanouil Benetos, Simon Dixon, al. An attack/decay model for piano transcription. In Proceedings of the 17th International Society for Music Information Retrieval Conference .IMIR 2016,, New York City, United States, 2016.
Andrea Cogliati, Zhiyao Duan, and Brendt Wohlberg. Context-dependent piano music transcription with convolutional sparse coding. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 24(12):2218 2230, 2016.
Andrea Cogliati, Zhiyao Duan, and Brendt Wohlberg. Piano transcription with convolutional sparse lateral inhibition. IEEE Signal Processing Letters, 24(4):392-396 396, 2017.
James Cross and Liang Huang. Span-based constituency parsing with a structure-label system and provably optimal dynamic oracles. In Proceedings of EMNLP. 2016.
Valentin Emiya, Roland Badeau, and Bertrand David. Multipitch estimation of piano sounds using a new probabilistic spectral smoothness principle. IEEE Transactions on Audio, Speech, and Language Processing, 1663): 2009.
Curtis Hawthorne, Erich Elsen, Jialin Song, Adam Roberts, Ian Simon, Colin Raffel, Jesse Engel, Sageev Oore, and Douglas Eck. Onsets and frames: Dual-objective piano transcription. In Pro- ceedings of the 19th International Society for Music Information Retrieval Conference, ISMIR 2018, Paris, France, 2018, 2018. URL https: /ttp:///axxvv..o org/ abs/1710
/ttt///xw.....///////////.... 11153
Curtis Hawthorne, Andriy Stasyuk, Adam Roberts, Ian Simon, Cheng-Zhi Anna Huang, Sander Dicleman, Erich Elsen, Jesse Engel, and Douglas Eck. Enabling factorized piano music modeling and generation with the MAESTRO dataset. International Conference on Learning Represen- tations, 2019. URL https://openreviee.. wttp://ooooommddrr11/....
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv: 1606.08150. 2016.
Hirokazu Kameoka, Takuya Nishimoto, and Shigeki Sagayama. A multipitch analyzer based on harmonic temporal structured clustering. IEEE Transactions on Audio, Speech, and Language Processing, 15(3):882-99) 2007.

---

Rainer Kelz, Sebastian Böck, and Gerhard Widmer. Deep polyphonic adsr piano note transcription. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Process- ing (ICASSP), pages 246-250. IEEE, 2019.
Apostolos Kemos, Heike Adel, and Hinrich Schütze. Neural semi-markov conditional random fields for robust character-basee based part-of-speech tagging. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume I (Long and Short Papers), pages 2736-2743. 2019.
Jong Wook Kim and Juan Pablo Bello. Adversarial learning for improved onsets and frames music transcription. In Proceedings of the 20th International Society for Music Information Retrieval Conference, ISMIR 2019, pages 670-677, 2019.
Nikita Kitaev and Dan Klein. Constituency parsing with self-attentive encoder. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages
2018.
Anssi Klapuri, Tuomas Virtanen, and Jan-Markus Holm. Robust multipitch estimation for the anal- ysis and manipulation of polyphonic musical signals. In Proc. COST-G6 Conference on Digital Audio Effects, pages 233-236, 2000.
Lingpeng Kong, Chris Dyer, and Noah A Smith. Segmental recurrent neural networks. 2016
Qiuqiang Kong, Bochen Li, Xuchen Song, Yuan Wan, and Yuxuan Wang. High-resolution piano transcription with pedals by regressing onsets and offsets times. arXiv preprint arXiv:20/0.01815, 2020.
Taegyun Kwon, Dasaem Jeong, and Juhan Nam. Polyphonic piano transcription using autoregressive multi-state note model. In Proceedings of the 19th International Society for Music Information Retrieval Conference, ISMIR 2018, Paris, France, 2018, 2020. URL patts::/arrtie org/ /bs/
1/10.1155 11133
Yijia Liu, Wanxiang Che, Jiang Guo, Bing Qin, and Ting Liu. Exploring segment representations for neural segmentation models. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, pages 2880-2886, 2016.
Gabriel Loaiza-Ganem and J. Cunningham. The continuous bernoulli: fixing a pervasive error in variational autoencoders. In NeurlPS, 2019.
L. Lu, Lingpeng Kong, Chris Dyer, Noah A. Smith, and Renals. Segmental recurrent neural networks for end-to-end speech recognition. In INTERSPEECH, 2016.
Kristen Masada and Razvan C Bunescu. Chord recognition in symbolic music using semi-markov conditional random fields. In Proceedings of the 18th International Society for Music Information Retrieval Conference, ISMIR 2017, Suzhou, China, pages 272-278, 2017.
Eita Nakamura, Emmanouil Benetos, Kazuyoshi Yoshii, and Simon Dixon. Towards complete poly- phonic music transcription: Integrating multi-pitch detection and rhythm quantization. 20/8 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 101-105. IEEE, 2018.
Ken O'Hanlon and Mark D Plumbley. Polyphonic piano transcription using non-negative matrix factorisation with group sparsity. 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3112-3116. IEEE, 2014.
Graham Poliner and Daniel PW Ellis. A discriminative model for polyphonic piano transcription. EURASIP Journal on Advances in Signal Processing, 2007:1-9, 2006.
Colin Raffel, Brian McFee, Eric J. Humphrey, Justin Salamon, Oriol Nieto, Dawen Liang, and Daniel P. W Ellis. Mir_eval A transparent implementation of common mir metrics. In Proceed- ings of the 15th International Society for Music Information Retrieval Conference, pages 367-372, 2014.
12

---

Christopher Raphael. Automatic transcription of piano music. In 3rd International Conference on. Music Information Retrieval, ISMIR 2002, Paris, France, 2002.
Miguel A Román, Antonio Pertusa, and Jorge Calvo-Zaragoza. An end-to-end framework for audio- (o-score music transcription monophonic excerpts. In Proceedings of the 19th International Society for Music Information Retrieval Conference, ISMIR 2018, Paris, France, 2018.
Miguel A Román, Antonio Pertusa, and Jorge Calvo-Zaragoza. A holistic approach to polyphonic music transcription with neural networks. In ISMIR, 2019.
Sunita Sarawagi and William W Cohen. Semi-markov conditional random fields for information extraction. In Proceedings of the 17th International Conference on Neural Information Processing Systems, pages 1185-1192. 2004.
Prem Seetharaman, Gordon Wichern, Bryan Pardo, and Jonathan Le Roux. Autoclip: Adaptive gradient clipping for source separation networks. In 2020 IEEE 30th International Workshop on Machine Learning for Signal Processing (MISP), pages 1-6. IEEE, 2020.
Siddharth Sigtia, Emmanouil Benetos, and Simon Dixon. An end-to-end neural network for poly- phonic piano music transcription. IEEE/ACM Transactions Audio, Speech, and Language Processing, 24(5):927-939, 2016.
Paris Smaragdis and Judith C Brown. Non-negative matrix factorization for polyphonic music tran- scription. In 2003 IEEE Workshop on Applications of Signal Processing Audio and Acoustics (IEEE Cat. No. 03TH8684), pages 177-180. IEFE, 2003.
Leslie N Smith and Nicholay Topin. Super-convergent Very fast training of neural networks using large learning rates. In Artificial Intelligence and Machine Learning for Multi- -Daiiinn Operations Applications, volume 11006, page 1 1100612. International Society for Optics and Photonics, 2019.
Emmanuel Vincent, Nancy Bertin, and Roland Badeau. Adaptive harmonic spectral decomposition for multiple pitch estimation. IEEE Transactions on Audio, Speech, and Language Processing, 18(3):528 537,53377500 2009.
Felix Weninger, Christian Kirst, Björn Schuller, and Hans-Joachim Bungartz. discriminative ap- proach to polyphonic piano note transcription using supervised non-negative matrix factorization. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 6-10. IEEE, 2013.
Zhixiu Ye and Zhen-Hua Ling, Hybrid semi-markov erf for neural sequence labeling. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguisties (Volume 2: Short Papers), pages 235-240, 2018
Juntang Zhuang. Tommy Tang, Yifan Ding, Sekhar Tatikonda, Nicha Dvornek, Xenophon Pa- pademetris, and James Duncan. Adabelief optimizer: Adapting stepsizes by the belicf in observed gradients. Conference Neural Information Processing Systems, 2020.
Jingwei Zhuo, Yong Cao, Jun Zhu, Bo Zhang, and Zaiqing Nic. Segment-level sequence modeling using gated recursive semi-markov conditional random fields. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1413- 1423, 2016.
13


Reviews:
[{"review_id": "vpQ56z-VzAW", "reviewer": "Reviewer_jUoX", "summary": "The authors take a well studied network architecture and task (MAESTRO piano transcription) and propose a straightforward extension of previous work to incorporate a CRF output layer that can directly predict temporal dependencies among the output labels. This contrasts with previous SOTA techniques that use multiple output heads (for separate parts of notes) and heuristics to infer the notes from network outputs. The CRF loss function is used to train a network similar to previous work, and shows improvements in classification accuracy, most notably in Onset+Offset F1 score.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper presents a straightforward approach, but impressive results of value to the music research community. The significance is limited by the domain-specific nature of the work, but the contribution is high-quality. Most significantly, the clarity of the writing and description of the CRF layer / forward-backward algorithm could be significantly improved to make the paper more useful to readers.\n\n\nOriginality/Significance: This work is an incremental combination of existing techniques for AMT (ConvRNN architectures, CRF) on existing benchmarks. However, the implementation of the differentiable CRF layer is not trivial, and demonstrating that it actually improves the note offset predictions as well as it does is a valuable contribution to the field. The work is not directly applicable to domains beyond piano transcription, which limits the relevance for the broader ML community, but the techniques could possibly be useful for other domains with further domain-specific adaptations. \n\nQuality: The authors clearly demonstrate an improvement of the SOTA on the MAESTRO dataset, which is a good target dataset for their model/task. While the numbers are clearly superior for the proposed system, it is unclear why there are some inconsistencies with the evaluation numbers for the other papers (https://arxiv.org/pdf/2010.01815.pdf, https://arxiv.org/pdf/1710.11153.pdf). For instance, Kong et al. show an F1 note + offset of 82.47, but are reported at 84.45. It is unclear from the paper the reasons for these discrepancies, and whether the public models are used to recompute metrics for this paper, which casts a little confusion over the numbers for the reported system. \n\nIf that is the case, it's worth noting that the MAESTRO v2 dataset has several known erroneous tracks in it, and it would be more beneficial to the field to start using the MAESTRO v3 dataset for training and comparison.\n\nFor the benchmarking comparisons, it could be helpful to add a row for the time ratio of Viterbi decoding / nn model forward prediction, to help support the claim that it is indeed a small fraction.\n\n\nClarity: For the most part the ideas of the paper are presented clearly, but this is clearly the area for the most improvement. Specifically, the loss function, and forward-backward algorithm are the key contributions of the paper, but could benefit from a clearer description. A few things that could help:\n* The notion of \"eventType\" is perhaps overly opaque in this context. There are many overloaded concepts (such as onset, offset, velocity change, pedal, etc.) that are all considered \"events\" in similar literature. For these equations, \"eventType\" seems to just be referring to a given dimension of a frame-level binary note/pedal \"On\" variable. The equations could probably be simplified by describing it as such and using a more compact letter/symbol for the index. It's also unclear if the score matrix refers to 88 notes and pedal (89 dims) simultaneously or if there is some other factorization.\n\n* Similarly, factorizing over the notes independently is helpful for understanding eq. 1, so introducing eq. 2 first, can better motivate the equation for p(y_note/pedal | x).\n\n* Pairing Equation 1 and Algorithm 1 with a diagram/example to demonstrate visually how the output scores are converted into loss values would help to clarify some of the notation. For example, expanding on Table 1 visually for a specific example of score outputs and labels could help clarify the entire CRF section.\n\n* The forward backward algorithm could use further clarification/elaboration. It is one of the key contributions of the paper, but is mentioned only in passing for a sentence without any relevant citations / explanation. Algorithm 1 is complete, but could be made clearer with a diagram and more explanation. \n\n\nSome proof reading could benefit the paper, for example: \nL114: \"(See Section XXX)\"\nL126: \"is the unnormalized the likelihood\"", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors do not discuss the societal impacts of their work, but piano transcription is a relatively benign technology so that is reasonable.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A, piano transcription is a relatively benign application and the datasets are all public and open source.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "k7q8PNQ4sS", "reviewer": "Reviewer_pJNf", "summary": "This paper describes a piano transcription (audio->piano roll / MIDI representation) model that uses a conditional random field layer to produce the transcription as a sequence of event-intervals, rather than the commonly used frame-wise decoding/post-processing of prior work.  The proposed method has a slight increase in computational cost, but remains manageable for typical input lengths, and improves over prior work under some (but not all) transcription evaluation metrics.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The primary innovation of this paper, as I understand it, is the formulation of the transcription problem in terms of events / intervals, rather than time-based classification.  While many works have approached this kind of problem with a two-stage model---produce an activation map, and then decode it with some kind of dynamics model, eg an HMM---this paper integrates the two steps in an end-to-end fashion through a CRF output layer.  While the individual components used here are not entirely novel, the authors seem to have done a good job of adapting and optimizing the Semi-CRF method for this particular use-case.  \n\nThe paper is generally clear and well-written, and literature review covers the related recent work on piano transcription.  Section 3.1 was somewhat difficult for me to follow at first, but the example table (1) did help quite a bit to clarify the structure of the loss/likelihood function.  I appreciated the attention to detail in designing the objective function and optimizing the memory layout of the intermediate representation (section 3.1).\n\nThe experimental results (tables 2-3) come across as somewhat mixed, for a few reasons.  First, only mean scores are reported, with no mention of variance or stability; the single highest number for each metric is marked in bold, but most of the time there are comparable methods (ie within 1%) of the best, and no statistical analysis is performed here.  In the cases where there does appear to be a large effect size, it sometimes vanishes when looking at the F-scores, or the proposed method under-performs prior work.  The exception here are the offset metrics (last two columns of table 2), where the proposed method does seem to provide a consistent and meaningful improvement over prior work (~4% absolute F1 improvement over Kong et al., 2020).  The authors claim improved speed of their method as a benefit over prior work, but this is not demonstrated.  Otherwise the experiments seem to be well-designed and executed.\n\nTo summarize the results, the take-away message appears to be that the proposed method improves at detecting note offsets, and may be faster (or otherwise more efficient) than prior methods for comparable accuracy.  It is worth noting that offset detection is in practice quite a difficult task, as data is often unreliable: even with MIDI, the observed offset may not match the encoded offset due to variations in sound fonts.  It is not lost on this reviewer that the improvements are in a difficult part of the problem space, but at the same time, the improvements overall are rather incremental.  The method itself is interesting though, and I agree with the authors that it may be applicable to other contexts, though it's not obvious how well some of the specific assumptions made here (eg non-overlapping events of the same type) may transfer to applications like sound event detection.  As such, significance of this work is somewhat difficult to gauge.\n\n## Minor comments\n\n- Line 114: broken section reference\n- Line 146-7: this part seems like a non-sequitur, or at least an incomplete thought.  I understand that eq (2) is a simplifying assumption made for practicality purposes, but saying that it's similar to naive Bayes is hardly a justification.\n- Line 260: the use of $N$ (duration) and $|E|$ (event types) is inconsistent with earlier parts of the paper, where $N$ refers to the number of event types (figure 1).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No discussion of societal impact is provided.  However, this question does seem rather out of scope given the subject of the paper, so I don't have any suggestions to offer here.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "Hi9_BCNNG7I", "reviewer": "Reviewer_7qii", "summary": "This paper presents an event-based polyphonic piano transcription model using neural semi-CRFs. Unlike the state-of-the-art \"onset and frame\" model and its variations, which rely on frame-level note state predictions, this work is based on segmenting a long sequence into a set of intervals which correspond to notes and silences. They also propose a forward-backward learning algorithm for computing the score functions of the intervals and, as a result, it enables to compute the partition function. This new approach achieved comparable performance to previous state-of-the-arts and it has a great potential to audio event detection tasks. \n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "General Comments:\n\nI have been working on polyphonic piano transcription about 10 years. To my knowledge, the event-based algorithm in this paper is the first attempt beyond the frame-level note-state prediction approach, since deep learning became popular. Segmenting intervals and scoring it with event types to find note events is a great idea. With this new approach, the performance reached those of the state-of-the art models. In particular, note-level accuracy with offset is very high. This makes sense considering that this event-based method see a note event as a whole to compute the score and this note offset estimation can be better than frame-level approaches. I read the paper with a great fun. I am not sure if the semi-CRF algorithms (Algorithm 1 and 2) are very novel in terms of the NeurIPS standard, but this event-based approach will have a strong impact on automatic music transcription in the field of Music Information Retrieval (MIR) and acoustic event detection in the field of Detection and Classification of Acoustic Scenes and Events (DCASE). I strongly argue for accepting this paper.  \n\nDetailed Comments:\n- line 49: \"400-1600 frames with around 90 labeling channels\" --> This was hard to understand until I read the experiment section. What is \"labeling channel\"? \n\n- line 88 - 90: The differences of semi-CRF layer from previous works are addressed in two aspects. On top of that, the \"polyphonic nature\" where note event (or segmented intervals) can overlapped over different notes (but not over time), could be also added as a difference?\n\n- line 191-192: the element-wise multiplication hi ⊙ hj , and the first three moments for the contextual embeddings are concatenated as input of the MLP module. Why are they added?  Without the element-wise multiplication and the three moments, the performance significantly decreases?  The authors did not explain the motivation to add the multiplication and moments.  \n\n- line 199: \"We also experimented with scaling the intervalic score by the length of the interval.\" --> The length scale was part of the ablation study in Table 2 but this short sentence is very hard for me to understand what it is and why it works.\n\n- Section 4: more details of \"eventType\" will be necessary to fully understand the model. \n\n- Section 4.5: it is nice to have a section that address the time complexity.\n\n- Page 9: the last page has a lot of room to put more detailed analysis of the results. For example, some visualizations of predicted pianoroll representation and its comparison to those from the onset-and-frame models (Hawthorne et al.) and its variant (Kong et al.) with respect to the ground truth will provide insight about their difference (I expect to see the improved note-off prediction)\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This paper did not address the limitations and I also cannot find any specific one in the task and experiment scope of this paper.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Overview: The paper addresses the problem of piano transcription using semi-Markov CRF. The key contribution is to treat notes as events with finite duration. In contrast, previous methods inferred notes as frame-level events ignoring their duration. The authors demonstrate (in their updated results) the impact of their model with significant improvement in performance over previous work.

Reviews: All the reviewers recognized the novelty of this work especially in the context of the domain. The model may be applicable in audio event detection and addresses concerns about wider application. Reviewers had no reservation of its impact and contribution to the domain. Authors are encouraged to incorporate all the suggestions provided by the reviewers and their own clarifying responses.
Decision: Accept (Poster)