Package: transkun
Piano Transcription with Neural Semi-CRFThis repo contains code for the paperYujia Yan, Frank Cwitkowitz, Zhiyao Duan, Skipping the Frame-Level: Event-Based Piano Transcription With Neural Semi-CRFs, Advances in Neural Information Processing Systems, 2021,OpenReview,paper,appendixpip installationpip3installtranskunThe pip package provides a quick command for transcribing piano performance audio into midi:$transkuninput.mp3output.midwith cuda:$transkuninput.mp3output.mid--devicecudaOverviewThis system works as follows:A score tensor is computed from the input audio that scores every possible intervals for whether or not being an event.The score tensor computed in (1) is then decoded by the proposed semi-CRF layer to obtain event intervals directly via dynamic programming (viterbi).Attributes, e.g., velocity and refined onset/offset position, associated with each interval are then predicted from the extracted event intervals from (2).Basic UsageThe Semi-CRF LayerThis code includes an neural semi-CRF module that is optimized for the problem domain.Here is a minimal example for using this module:importCRFimporttorchT=200NBatch=4# representing the score for the interval [TBegin, TEnd]# dimensions: [TEnd, TBegin, NBatch]# only the lower triangular part is usedscore=((torch.randn(T,T,NBatch))).cuda()# representing the score for being not an interval, dimensions [TBegin, TBegin+1]noiseScore=((torch.randn(T-1,NBatch))).cuda()# a list of list of non-overlapping intervalsintervals=[[(0,2),(4,6),(6,6),(7,8)],[(1,2),(3,5),(19,19)],[(0,0),(4,7)],[],]crf=CRF.NeuralSemiCRFInterval(score,noiseScore)## log probabilitylogP=crf.logProb(intervals)## decodingdecoded=crf.decode()## decoding starting from a given position, useful for segment based processingdecoded=crf.decode(forcedStartPos=[4]*NBatch)Transcribing piano performance into a MIDI filepython3-mtranskun.transcribe-h

usage:transkun[-h][--weightWEIGHT][--device[DEVICE]][--segmentHopSizeSEGMENTHOPSIZE][--segmentSizeSEGMENTSIZE]audioPathoutPath

positionalarguments:audioPathpathtotheinputaudiofileoutPathpathtotheoutputMIDIfile

optionalarguments:-h,--helpshowthishelpmessageandexit--weightWEIGHTpathtothepretrainedweight--device[DEVICE]Thedeviceusedtoperformthemostcomputations(optional),DEFAULT:cpu--segmentHopSizeSEGMENTHOPSIZEThesegmenthopsizeforprocessingtheentireaudiofile(s),DEFAULT:10--segmentSizeSEGMENTSIZEThesegmentsizeforprocessingtheentireaudiofile(s),DEFAULT:20Please note that segmentHopSize and segmentSize should be chosen such that there is an overlap between consecutive segments.
The included weight is trained under segmentHopSize = 10 and segmentSize = 20.This script can also be used directly as the command line command 'transkun' if the pip package is installed, e.g.,$transkuninput.mp3output.midHandling the datasetConverting to the same sampling rateWe assume the data contains only the same sampling rate 44100hz. Therefore for the maestro dataset it is necessary to perform sampling rate conversion to 44100hz for the last two years (2017 and 2018) .Justification for using the original sample rateMany exisiting works downsample the original audio files to a smaller sampling rate. And it seems quite "conventional" to do downsampling. However, from our perspective, this step is unnecessary:Spectral processing for getting the mel spectrum only incurs a small computation cost. However, a good quality downsampling algorithm is more expensive than computing the mel spectrum.Whether or not compute the spectrum from the downsampled version, the size of the input to the neural network is at the same range.There is no need to load all data into memory as the wav file is random accessible. This work directly random acesses the wav file instead of using a specific file format.Generating metadata filesAssuming all audio files have already been converted to the same sampling rate, we iterate the entire dataset to combine the groundtruth midi and metadata into a single file.The following script will generate train.pt, val.pt and test.ptpython3-mtranskun.createDatasetMaestro-h
usage:createDatasetMaestro.py[-h][--noPedalExtension]datasetPathmetadataCSVPathoutputPath

positionalarguments:datasetPathfolderpathtothemaestrodatasetmetadataCSVPathpathtothemetadatafileofthemaestrodataset(csv)outputPathpathtotheoutputfolder

optionalarguments:-h,--helpshowthishelpmessageandexit--noPedalExtensionDonotperformpedalextensionaccordingtothesustainpedalThis command will generate train.pt, dev.pt, test.pt in the outputPath.TrainingAfter generating the medata files, we can perform training using the dataset. During training, the audio waveforms will be fetched directly from the original .wav files.Firstly, generate a config template file for the model:mkdirtestcheckpoint
python3-mtranskun.generateConfigtranskun.Model_ablation>checkpoint/conf.jsonThen we call the training script.python3-mtranskun.train-hThe Evaluation ModuleComparing the output MIDI files and the groundtruth MIDI filesWe also provide an out-of-box tool for computing metrics directly from output midi files.usage:computeMetrics.py[-h][--outputJSONOUTPUTJSON][--noPedalExtension][--nProcess[NPROCESS]][--computeDeviations]estDIRgroundTruthDIR

computemetricsdirectlyfromMIDIfiles.
NotethatestDIRshouldhavethesamefolderstructureasthegroundTruthDIR.
TheMIDIfilestoevaluateshouldhavethesameextensionasthegroundtruth.
Metricsoutputedareorderedbyprecision,recall,f1,overlap.

positionalarguments:estDIRgroundTruthDIR

optionalarguments:-h,--helpshowthishelpmessageandexit--outputJSONOUTPUTJSONpathtosavetheoutputfilefordetailedmetricsperaudiofile--noPedalExtensionDonotperformpedalextensionaccordingtothesustainpedalforthegroundtruch--nProcess[NPROCESS]numberofworkersformultiprocessing--computeDeviationsoutputdetailedonset/offsetdeviationsforeachmatchednote.Currently, we do not support evaluation of multitrack MIDIs.This command can also be used directly as the command line script 'transkunEval' if the pip package is installed.Ploting the empirical cumulative distribution function to visualize the onset/offset accuracyUse the following script to plot the ECDF curve for onset/offset deviations:usage:plotDeviation.py[-h][--labels[LABELS[LABELS...]]][--offset][--TT][--output[OUTPUT]][--noDisplay]evalJsons[evalJsons...]plottheempiricalcumulativedistributionfunctionononset/offsetdeviations

positionalarguments:evalJsonsaseqeunceoftheoutputjsonfilesfromthecomputeMetricsscript,thedeviationoutputshouldbeenabled

optionalarguments:-h,--helpshowthishelpmessageandexit--labels[LABELS[LABELS...]]specifylabelstoshowonthelegend--offsetplottheoffsetdeviationcurve.Ifnotspecified,onsetdeviationcurvewillbeplotted--TTtimelimit(ms),default:50ms--output[OUTPUT]filenametosave--noDisplayDonotshowthefigure.CitationIf you find this repository helpful, please consider citing:Bibtex:@inproceedings{yan2021skipping,title={Skipping the Frame-Level: Event-Based Piano Transcription With Neural Semi-{CRF}s},author={Yujia Yan and Frank Cwitkowitz and Zhiyao Duan},booktitle={Advances in Neural Information Processing Systems},editor={A. Beygelzimer and Y. Dauphin and P. Liang and J. Wortman Vaughan},year={2021},url={https://openreview.net/forum?id=AzB2Pq7UFsA}}