Package: riffusion
:guitar: RiffusionRiffusion is a library for real-time music and audio generation with stable diffusion.Read about it athttps://www.riffusion.com/aboutand try it athttps://www.riffusion.com/.This is the core repository for riffusion image and audio processing code.Diffusion pipeline that performs prompt interpolation combined with image conditioningConversions between spectrogram images and audio clipsCommand-line interface for common tasksInteractive app using streamlitFlask server to provide model inference via APIVarious third party integrationsRelated repositories:Web app:https://github.com/riffusion/riffusion-appModel checkpoint:https://huggingface.co/riffusion/riffusion-model-v1CitationIf you build on this work, please cite it as follows:@article{Forsgren_Martiros_2022,
  author = {Forsgren, Seth* and Martiros, Hayk*},
  title = {{Riffusion - Stable diffusion for real-time music generation}},
  url = {https://riffusion.com/about},
  year = {2022}
}InstallTested in CI with Python 3.9 and 3.10.It's highly recommended to set up a virtual Python environment withcondaorvirtualenv:condacreate--nameriffusionpython=3.9
condaactivateriffusionInstall Python package:pipinstall-Uriffusionor clone the repository and install from source:gitclonehttps://github.com/riffusion/riffusion.gitcdriffusion
python-mpipinstall--editable.In order to use audio formats other than WAV,ffmpegis required.sudoapt-getinstallffmpeg# linuxbrewinstallffmpeg# maccondainstall-cconda-forgeffmpeg# condaIf torchaudio has no backend, you may need to installlibsndfile. Seethis issue.If you have an issue, try upgradingdiffusers. Tested with 0.9 - 0.11.Guides:Simple Install Guide for WindowsBackendsCPUcpuis supported but is quite slow.CUDAcudais the recommended and most performant backend.To use with CUDA, make sure you have torch and torchaudio installed with CUDA support. See theinstall guideorstable wheels.To generate audio in real-time, you need a GPU that can run stable diffusion with approximately 50
steps in under five seconds, such as a 3090 or A10G.Test availability with:importtorchtorch.cuda.is_available()MPSThempsbackend on Apple Silicon is supported for inference but some operations fall back to CPU,
particularly for audio processing. You may need to setPYTORCH_ENABLE_MPS_FALLBACK=1.In addition, this backend is not deterministic.Test availability with:importtorchtorch.backends.mps.is_available()Command-line interfaceRiffusion comes with a command line interface for performing common tasks.See available commands:riffusion-hGet help for a specific command:riffusionimage-to-audio-hExecute:riffusionimage-to-audio--imagespectrogram_image.png--audioclip.wavRiffusion PlaygroundRiffusion contains astreamlitapp for interactive use and exploration.Run with:riffusion-playgroundAnd access athttp://127.0.0.1:8501/Run the model serverRiffusion can be run as a flask server that provides inference via API. This server enables theweb appto run locally.Run with:riffusion-server--host127.0.0.1--port3013You can specify--checkpointwith your own directory or huggingface ID in diffusers format.Use the--deviceargument to specify the torch device to use.The model endpoint is now available athttp://127.0.0.1:3013/run_inferencevia POST request.Example input (seeInferenceInputfor the API):{"alpha":0.75,"num_inference_steps":50,"seed_image_id":"og_beat","start":{"prompt":"church bells on sunday","seed":42,"denoising":0.75,"guidance":7.0},"end":{"prompt":"jazz with piano","seed":123,"denoising":0.75,"guidance":7.0}}Example output (seeInferenceOutputfor the API):{"image":"< base64 encoded JPEG image >","audio":"< base64 encoded MP3 clip >"}TestsTests live in thetest/directory and are implemented withunittest.To run all tests:python-munittesttest/*_test.pyTo run a single test:python-munittesttest.audio_to_image_testTo preserve temporary outputs for debugging, setRIFFUSION_TEST_DEBUG:RIFFUSION_TEST_DEBUG=1python-munittesttest.audio_to_image_testTo run a single test case within a test:python-munittesttest.audio_to_image_test-kAudioToImageTest.test_stereoTo run tests using a specific torch device, setRIFFUSION_TEST_DEVICE. Tests should pass withcpu,cuda, andmpsbackends.Development GuideInstall additional packages for dev withpython -m pip install -r requirements-dev.txt.Linters:ruff,flake8,pylintFormatter:blackType checker:mypyDocstring checker:pydocstyleThese are configured inpyproject.toml.The results ofmypy .,black ., andruff .mustbe clean to accept a PR.CI is run through GitHub Actions from.github/workflows/ci.yml.Contributions are welcome through pull requests.