Package: midi-ddsp
MIDI-DDSP: Detailed Control of Musical Performance via Hierarchical ModelingDemos|Blog Post|Colab Notebook|Paper|MIDI-DDSP is a hierarchical audio generation model for synthesizing MIDI expanded
fromDDSP.LinksCheck out the blog post 💻Read the original paper 📄Listen to some examples 🔈Try out MIDI Synthesis using MIDI-DDSP 🎵->🎻🔊Try out Web Demo on Huggingface SpacesInstall MIDI-DDSPYou could install MIDI-DDSP via pip, which allows you to use the
coolCommand-line MIDI synthesisto synthesize your MIDI.To install MIDI-DDSP via pip, simply run following command in apython3.8environment:pip install midi-ddspMIDI-DDSP is developed using tensorflow 2.7.0. Newer tensorflow version should also work.Train MIDI-DDSPTo train MIDI-DDSP, please first install midi-ddsp and clone the MIDI-DDSP repository:git clone https://github.com/magenta/midi-ddsp.gitFor dataset, please download the tfrecord files for the URMP dataset inhereto thedatafolder in
your cloned repository using the following commands:cd midi-ddsp # enter the project directory
mkdir ./data # create a data folder
gsutil cp gs://magentadata/datasets/urmp/urmp_20210324/* ./data # download tfrecords to directoryPlease checkherefor how to install and usegsutil.Finally, you can run the scripttrain_midi_ddsp.shto train the exact same model we used in the paper:sh ./train_midi_ddsp.shThe current codebase does not support training with arbitrary dataset, but we will hopefully update that in the near
future.Side note:If one download the dataset to a different location, please change thedata_dirparameter intrain_midi_ddsp.sh.The training of MIDI-DDSP takes approximately 18 hours on a single RTX 8000. The training code for now does not support
multi-GPU training. We recommend using a GPU with more than 24G of memory when training Synthesis Generator in batch
size of 16. For a GPU with less memory, please consider using a smaller batch size and change the batch size
intrain_midi_ddsp.sh.*The MIDI-DDSP cannot be installed on M1 Macbook. There is a problem with pip install package involves TensorFlow.Try to play with MIDI-DDSP yourself!Please try out MIDI-DDSP
inColab notebooks!In this notebook, you will try to use MIDI-DDSP to synthesis a monophonic MIDI file, adjust note expressions, make pitch
bend by adjusting synthesis parameters, and synthesize quartet from Bach chorales.We have trained MIDI-DDSP on the URMP dataset which support synthesizing 13 instruments: violin, viola, cello, double
bass, flute, oboe, clarinet, saxophone, bassoon, trumpet, horn, trombone, tuba. You could find how to download and use
our pre-trained model below:Command-line MIDI synthesisOn can use the MIDI-DDSP as a command-line MIDI synthesizer just like FluidSynth.To use command-line synthesis to synthesize a midi file, please first download the model weights by running:midi_ddsp_download_model_weightsTo synthesize a midi file simply run the following command:midi_ddsp_synthesize --midi_path <path-to-midi>For a starter, you can try to synthesize the example midi file in this repository:midi_ddsp_synthesize --midi_path ./midi_example/ode_to_joy.midThe command line also enables synthesize a folder of midi files. For more advance use (synthesize a folder, using
FluidSynth for instruments not supported, etc.), please seemidi_ddsp_synthesize --help.If you have a trouble downloading the model weights, please manually download
fromhere, and specify
thesynthesis_generator_weight_pathandexpression_generator_weight_pathby yourself when using the command line.
You can also specify your other model weights if you want to use your own trained model.Python UsageAfter installing midi-ddsp, you could import midi-ddsp in python and synthesize MIDI in your code.Minimal ExampleHere is a simple example to use MIDI-DDSP to synthesize a midi file:frommidi_ddspimportsynthesize_midi,load_pretrained_modelmidi_file='ode_to_joy.mid'# Load pre-trained modelsynthesis_generator,expression_generator=load_pretrained_model()# Synthesize MIDIoutput=synthesize_midi(synthesis_generator,expression_generator,midi_file)# The synthesized audiosynthesized_audio=output['mix_audio']Advance UsageHere is an advance example to synthesize theode_to_joy.mid, change the note expression controls, and adjust the
synthesis parameters:importnumpyasnpimporttensorflowastffrommidi_ddsp.utils.midi_synthesis_utilsimportsynthesize_mono_midi,conditioning_df_to_audiofrommidi_ddsp.utils.inference_utilsimportget_process_groupfrommidi_ddsp.midi_ddsp_synthesizeimportload_pretrained_modelfrommidi_ddsp.data_handling.instrument_name_utilsimportINST_NAME_TO_ID_DICT# -----MIDI Synthesis-----midi_file='ode_to_joy.mid'# Load pre-trained modelsynthesis_generator,expression_generator=load_pretrained_model()# Synthesize with violin:instrument_name='violin'instrument_id=INST_NAME_TO_ID_DICT[instrument_name]# Run model predictionmidi_audio,midi_control_params,midi_synth_params,conditioning_df=synthesize_mono_midi(synthesis_generator,expression_generator,midi_file,instrument_id,output_dir=None)synthesized_audio=midi_audio# The synthesized audio# -----Adjust note expression controls and re-synthesize-----# Make all notes weak vibrato:conditioning_df_changed=conditioning_df.copy()note_vibrato=conditioning_df_changed['vibrato'].valueconditioning_df_changed['vibrato']=np.ones_like(conditioning_df['vibrato'].values)*0.1# Re-synthesizemidi_audio_changed,midi_control_params_changed,midi_synth_params_changed=conditioning_df_to_audio(synthesis_generator,conditioning_df_changed,tf.constant([instrument_id]))synthesized_audio_changed=midi_audio_changed# The synthesized audio# There are 6 note expression controls in conditioning_df that you could change:# 'volume', 'vol_fluc', 'vibrato', 'brightness', 'attack', 'vol_peak_pos'.# Please refer to https://colab.research.google.com/github/magenta/midi-ddsp/blob/main/midi_ddsp/colab/MIDI_DDSP_Demo.ipynb#scrollTo=XfPPrdPu5sSy for the effect of each control.# -----Adjust synthesis parameters and re-synthesize-----# The original synthesis parameters:f0_ori=midi_synth_params['f0_hz']amps_ori=midi_synth_params['amplitudes']noise_ori=midi_synth_params['noise_magnitudes']hd_ori=midi_synth_params['harmonic_distribution']# TODO: make your change of the synthesis parameters here:f0_changed=f0_oriamps_changed=amps_orinoise_changed=noise_orihd_changed=hd_ori# Resynthesis the audio using DDSPprocessor_group=get_process_group(midi_synth_params['amplitudes'].shape[1],use_angular_cumsum=True)midi_audio_changed=processor_group({'amplitudes':amps_changed,'harmonic_distribution':hd_changed,'noise_magnitudes':noise_changed,'f0_hz':f0_changed,},verbose=False)midi_audio_changed=synthesis_generator.reverb_module(midi_audio_changed,reverb_number=instrument_id,training=False)synthesized_audio_changed=midi_audio_changed# The synthesized audioBibtex@inproceedings{
wu2022mididdsp,
title={{MIDI}-{DDSP}: Detailed Control of Musical Performance via Hierarchical Modeling},
author={Yusong Wu and Ethan Manilow and Yi Deng and Rigel Swavely and Kyle Kastner and Tim Cooijmans and Aaron Courville and Cheng-Zhi Anna Huang and Jesse Engel},
booktitle={International Conference on Learning Representations},
year={2022},
url={https://openreview.net/forum?id=UseMOjWENv}
}AcknowledgmentThis is not an officially supported Google product.We would like to thank @akhaliq for creating the(no longer functional).