Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The authors take a well studied network architecture and task (MAESTRO piano transcription) and propose a straightforward extension of previous work to incorporate a CRF output layer that can directly predict temporal dependencies among the output labels. This contrasts with previous SOTA techniques that use multiple output heads (for separate parts of notes) and heuristics to infer the notes from network outputs. The CRF loss function is used to train a network similar to previous work, and shows improvements in classification accuracy, most notably in Onset+Offset F1 score.
main_review: The paper presents a straightforward approach, but impressive results of value to the music research community. The significance is limited by the domain-specific nature of the work, but the contribution is high-quality. Most significantly, the clarity of the writing and description of the CRF layer / forward-backward algorithm could be significantly improved to make the paper more useful to readers.


Originality/Significance: This work is an incremental combination of existing techniques for AMT (ConvRNN architectures, CRF) on existing benchmarks. However, the implementation of the differentiable CRF layer is not trivial, and demonstrating that it actually improves the note offset predictions as well as it does is a valuable contribution to the field. The work is not directly applicable to domains beyond piano transcription, which limits the relevance for the broader ML community, but the techniques could possibly be useful for other domains with further domain-specific adaptations. 

Quality: The authors clearly demonstrate an improvement of the SOTA on the MAESTRO dataset, which is a good target dataset for their model/task. While the numbers are clearly superior for the proposed system, it is unclear why there are some inconsistencies with the evaluation numbers for the other papers (https://arxiv.org/pdf/2010.01815.pdf, https://arxiv.org/pdf/1710.11153.pdf). For instance, Kong et al. show an F1 note + offset of 82.47, but are reported at 84.45. It is unclear from the paper the reasons for these discrepancies, and whether the public models are used to recompute metrics for this paper, which casts a little confusion over the numbers for the reported system. 

If that is the case, it's worth noting that the MAESTRO v2 dataset has several known erroneous tracks in it, and it would be more beneficial to the field to start using the MAESTRO v3 dataset for training and comparison.

For the benchmarking comparisons, it could be helpful to add a row for the time ratio of Viterbi decoding / nn model forward prediction, to help support the claim that it is indeed a small fraction.


Clarity: For the most part the ideas of the paper are presented clearly, but this is clearly the area for the most improvement. Specifically, the loss function, and forward-backward algorithm are the key contributions of the paper, but could benefit from a clearer description. A few things that could help:
* The notion of "eventType" is perhaps overly opaque in this context. There are many overloaded concepts (such as onset, offset, velocity change, pedal, etc.) that are all considered "events" in similar literature. For these equations, "eventType" seems to just be referring to a given dimension of a frame-level binary note/pedal "On" variable. The equations could probably be simplified by describing it as such and using a more compact letter/symbol for the index. It's also unclear if the score matrix refers to 88 notes and pedal (89 dims) simultaneously or if there is some other factorization.

* Similarly, factorizing over the notes independently is helpful for understanding eq. 1, so introducing eq. 2 first, can better motivate the equation for p(y_note/pedal | x).

* Pairing Equation 1 and Algorithm 1 with a diagram/example to demonstrate visually how the output scores are converted into loss values would help to clarify some of the notation. For example, expanding on Table 1 visually for a specific example of score outputs and labels could help clarify the entire CRF section.

* The forward backward algorithm could use further clarification/elaboration. It is one of the key contributions of the paper, but is mentioned only in passing for a sentence without any relevant citations / explanation. Algorithm 1 is complete, but could be made clearer with a diagram and more explanation. 


Some proof reading could benefit the paper, for example: 
L114: "(See Section XXX)"
L126: "is the unnormalized the likelihood"
limitations_and_societal_impact: The authors do not discuss the societal impacts of their work, but piano transcription is a relatively benign technology so that is reasonable.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents an event-based polyphonic piano transcription model using neural semi-CRFs. Unlike the state-of-the-art "onset and frame" model and its variations, which rely on frame-level note state predictions, this work is based on segmenting a long sequence into a set of intervals which correspond to notes and silences. They also propose a forward-backward learning algorithm for computing the score functions of the intervals and, as a result, it enables to compute the partition function. This new approach achieved comparable performance to previous state-of-the-arts and it has a great potential to audio event detection tasks. 

main_review: General Comments:

I have been working on polyphonic piano transcription about 10 years. To my knowledge, the event-based algorithm in this paper is the first attempt beyond the frame-level note-state prediction approach, since deep learning became popular. Segmenting intervals and scoring it with event types to find note events is a great idea. With this new approach, the performance reached those of the state-of-the art models. In particular, note-level accuracy with offset is very high. This makes sense considering that this event-based method see a note event as a whole to compute the score and this note offset estimation can be better than frame-level approaches. I read the paper with a great fun. I am not sure if the semi-CRF algorithms (Algorithm 1 and 2) are very novel in terms of the NeurIPS standard, but this event-based approach will have a strong impact on automatic music transcription in the field of Music Information Retrieval (MIR) and acoustic event detection in the field of Detection and Classification of Acoustic Scenes and Events (DCASE). I strongly argue for accepting this paper.  

Detailed Comments:
- line 49: "400-1600 frames with around 90 labeling channels" --> This was hard to understand until I read the experiment section. What is "labeling channel"? 

- line 88 - 90: The differences of semi-CRF layer from previous works are addressed in two aspects. On top of that, the "polyphonic nature" where note event (or segmented intervals) can overlapped over different notes (but not over time), could be also added as a difference?

- line 191-192: the element-wise multiplication hi ⊙ hj , and the first three moments for the contextual embeddings are concatenated as input of the MLP module. Why are they added?  Without the element-wise multiplication and the three moments, the performance significantly decreases?  The authors did not explain the motivation to add the multiplication and moments.  

- line 199: "We also experimented with scaling the intervalic score by the length of the interval." --> The length scale was part of the ablation study in Table 2 but this short sentence is very hard for me to understand what it is and why it works.

- Section 4: more details of "eventType" will be necessary to fully understand the model. 

- Section 4.5: it is nice to have a section that address the time complexity.

- Page 9: the last page has a lot of room to put more detailed analysis of the results. For example, some visualizations of predicted pianoroll representation and its comparison to those from the onset-and-frame models (Hawthorne et al.) and its variant (Kong et al.) with respect to the ground truth will provide insight about their difference (I expect to see the improved note-off prediction)

limitations_and_societal_impact: This paper did not address the limitations and I also cannot find any specific one in the task and experiment scope of this paper.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper describes a piano transcription (audio->piano roll / MIDI representation) model that uses a conditional random field layer to produce the transcription as a sequence of event-intervals, rather than the commonly used frame-wise decoding/post-processing of prior work.  The proposed method has a slight increase in computational cost, but remains manageable for typical input lengths, and improves over prior work under some (but not all) transcription evaluation metrics.
main_review: The primary innovation of this paper, as I understand it, is the formulation of the transcription problem in terms of events / intervals, rather than time-based classification.  While many works have approached this kind of problem with a two-stage model---produce an activation map, and then decode it with some kind of dynamics model, eg an HMM---this paper integrates the two steps in an end-to-end fashion through a CRF output layer.  While the individual components used here are not entirely novel, the authors seem to have done a good job of adapting and optimizing the Semi-CRF method for this particular use-case.  

The paper is generally clear and well-written, and literature review covers the related recent work on piano transcription.  Section 3.1 was somewhat difficult for me to follow at first, but the example table (1) did help quite a bit to clarify the structure of the loss/likelihood function.  I appreciated the attention to detail in designing the objective function and optimizing the memory layout of the intermediate representation (section 3.1).

The experimental results (tables 2-3) come across as somewhat mixed, for a few reasons.  First, only mean scores are reported, with no mention of variance or stability; the single highest number for each metric is marked in bold, but most of the time there are comparable methods (ie within 1%) of the best, and no statistical analysis is performed here.  In the cases where there does appear to be a large effect size, it sometimes vanishes when looking at the F-scores, or the proposed method under-performs prior work.  The exception here are the offset metrics (last two columns of table 2), where the proposed method does seem to provide a consistent and meaningful improvement over prior work (~4% absolute F1 improvement over Kong et al., 2020).  The authors claim improved speed of their method as a benefit over prior work, but this is not demonstrated.  Otherwise the experiments seem to be well-designed and executed.

To summarize the results, the take-away message appears to be that the proposed method improves at detecting note offsets, and may be faster (or otherwise more efficient) than prior methods for comparable accuracy.  It is worth noting that offset detection is in practice quite a difficult task, as data is often unreliable: even with MIDI, the observed offset may not match the encoded offset due to variations in sound fonts.  It is not lost on this reviewer that the improvements are in a difficult part of the problem space, but at the same time, the improvements overall are rather incremental.  The method itself is interesting though, and I agree with the authors that it may be applicable to other contexts, though it's not obvious how well some of the specific assumptions made here (eg non-overlapping events of the same type) may transfer to applications like sound event detection.  As such, significance of this work is somewhat difficult to gauge.

## Minor comments

- Line 114: broken section reference
- Line 146-7: this part seems like a non-sequitur, or at least an incomplete thought.  I understand that eq (2) is a simplifying assumption made for practicality purposes, but saying that it's similar to naive Bayes is hardly a justification.
- Line 260: the use of $N$ (duration) and $|E|$ (event types) is inconsistent with earlier parts of the paper, where $N$ refers to the number of event types (figure 1).
limitations_and_societal_impact: No discussion of societal impact is provided.  However, this question does seem rather out of scope given the subject of the paper, so I don't have any suggestions to offer here.
review_title: null

Meta-review: Overview: The paper addresses the problem of piano transcription using semi-Markov CRF. The key contribution is to treat notes as events with finite duration. In contrast, previous methods inferred notes as frame-level events ignoring their duration. The authors demonstrate (in their updated results) the impact of their model with significant improvement in performance over previous work.

Reviews: All the reviewers recognized the novelty of this work especially in the context of the domain. The model may be applicable in audio event detection and addresses concerns about wider application. Reviewers had no reservation of its impact and contribution to the domain. Authors are encouraged to incorporate all the suggestions provided by the reviewers and their own clarifying responses.
Decision: Accept (Poster)