Performance MIDI-to-Score Conversion by
Neural Beat Tracking

23rd International Society forMusic Information Retrieval Conference(ISMIR 2022)

Lele Liu*1,2, Qiuqiang Kong3, Veronica Morfi1, and Emmanouil Benetos1,2
1 Queen Mary University of London, UK 2 The Alan Turing Institute, UK 3 ByteDance Shanghai, China* The author conducted this work as an intern at ByteDance• For in-note beat prediction, we use a CRNN model• For out-of-note beat prediction, we design an objectivefunction to be minimized that encourages a low level oftempo change and add fewer out-of-note beatsI. Performance MIDI-to-Score (PM2S)
Conversion
PM2S
Conversion
Music score inMIDI/MusicXML/Lilypond/…
formatRecording from MIDI
keyboard or Transcribed
MIDI performance• Applications: Music improvisation, complete musictranscription, music performance analysis• Subtasks: Rhythm quantisation, note value prediction, key
estimation, voice separation, and possibly score type-setting
such as beaming and playing techniques annotation𝑏! – 𝑛-th beat time in the in-note beats list
𝜆 – penalty coefficient for adding out-of-note beats
𝑁 " - number of out-of-note beats• We further expand the CRNN model to predict a compact
output data representation for musical scores

II. Contributions• A deep learning (CRNN) model with a compact output forPM2S• A new rhythm quantisation method by tracking beats on aMIDI note sequence• Comparisons between different input encodings of MIDI notesequences for tracking beats from performance MIDI
• Ablation studies on input features and data augmentation
methods for tracking beats from performance MIDI
• A PM2S conversion toolbox

III. Methodology
Additional score information toget the MIDI score:

Performance MIDI in notesequence of length 𝑁:
𝑋=𝑝! , 𝑜! , 𝑑! , 𝑣!

$!"#

where for the 𝑛-th note:𝑝! - MIDI pitch
𝑜! - onset in seconds
𝑑! - duration in seconds
𝑣! - velocity in MIDI𝑌 = 𝑌! , 𝑌% , 𝑌&PM2S
Conversion
Model

where𝑌! - musical onset, note valueand hand part for all notes
𝑌% - time signature changes
𝑌& - key signature changes

Rhythm quantisation: the prediction of musical onset and notevalue for each note in the note sequence

IV. Experiments• We use a collection of classical piano performances andscores, in total 123.2 hours of piano performances in 504
distinct pieces.
• The train/valid/test splits do not have overlapping pieces• Comparative experiments show that:• Using MIDI pitch, onset shift in one-hot format, duration inraw values, and velocity achieves best performance among
difference input data encoding combinations• All four input features are helpful• Using data augmentation during model training is beneficial• Our proposed beat tracking model outperforms a baseline
beat tracking model
MV2H evaluation on performance MIDI-to-Score Conversion:

Rhythm quantisation:
Beat/downbeat tracking+Beat subdivisionprediction

Beat tracking on note sequences:• In-note beats prediction using neuralnetworks• Out-of-note beats prediction using
dynamic programming

Contact:
Email: lele.liu@qmul.ac.uk
Homepage: https://cheriell.github.io

Paper:

Code:• Results show better performance can be achieved by ourmodel in comparison with two commercial software (Finaleand MuseScore), based on the MV2H metric.
• Significantly better performance can be observed in themetrical alignment metric, due to our method’s ability to track
tempo changes.

Example& Demo: