Package: acids-rave
RAVE: Realtime Audio Variational autoEncoderOfficial implementation ofRAVE: A variational autoencoder for fast and high-quality neural audio synthesis(article link) by Antoine Caillon and Philippe Esling.If you use RAVE as a part of a music performance or installation, be sure to cite either this repository or the article !If you want to share / discuss / ask things about RAVE you can do so in ourdiscord server!Previous versionsThe original implementation of the RAVE model can be restored usinggitcheckoutv1InstallationInstall RAVE usingpipinstallacids-raveYou will needffmpegon your computer. You can install it locally inside your virtual environment usingcondainstallffmpegColabA colab to train RAVEv2 is now available thanks tohexorcismos!UsageTraining a RAVE model usually involves 3 separate steps, namelydataset preparation,trainingandexport.Dataset preparationYou can know prepare a dataset using two methods: regular and lazy. Lazy preprocessing allows RAVE to be trained directly on the raw files (i.e. mp3, ogg), without converting them first.Warning: lazy dataset loading will increase your CPU load by a large margin during training, especially on Windows. This can however be useful when training on large audio corpus which would not fit on a hard drive when uncompressed. In any case, prepare your dataset usingravepreprocess--input_path/audio/folder--output_path/dataset/path(--lazy)TrainingRAVEv2 has many different configurations. The improved version of the v1 is calledv2, and can therefore be trained withravetrain--configv2--db_path/dataset/path--out_path/model/out--namegive_a_nameWe also provide a discrete configuration, similar to SoundStream or EnCodecravetrain--configdiscrete...By default, RAVE is built with non-causal convolutions. If you want to make the model causal (hence lowering the overall latency of the model), you can use the causal moderavetrain--configdiscrete--configcausal...New in 2.3, data augmentations are also available to improve the model's generalization in low data regimes. You can add data augmentation by adding augmentation configuration files with the--augmentkeywordravetrain--configv2--augmentmute--augmentcompressMany other configuration files are available inrave/configsand can be combined. Here is a list of all the available configurations & augmentations :TypeNameDescriptionArchitecturev1Original continuous modelv2Improved continuous model (faster, higher quality)v2_smallv2 with a smaller receptive field, adpated adversarial training, and noise generator, adapted for timbre transfer for stationary signalsv2_nopqmf(experimental) v2 without pqmf in generator (more efficient for bending purposes)v3v2 with Snake activation, descript discriminator and Adaptive Instance Normalization for real style transferdiscreteDiscrete model (similar to SoundStream or EnCodec)onnxNoiseless v1 configuration for onnx usageraspberryLightweight configuration compatible with realtime RaspberryPi 4 inferenceRegularization (v2 only)defaultVariational Auto Encoder objective (ELBO)wassersteinWasserstein Auto Encoder objective (MMD)sphericalSpherical Auto Encoder objectiveDiscriminatorspectral_discriminatorUse the MultiScale discriminator from EnCodec.OtherscausalUse causal convolutionsnoiseEnables noise synthesizer V2hybridEnable mel-spectrogram inputAugmentationsmuteRandomly mutes data batches (default prob : 0.1). Enforces the model to learn silencecompressRandomly compresses the waveform (equivalent to light non-linear amplification of batches)gainApplies a random gain to waveform (default range : [-6, 3])ExportOnce trained, export your model to a torchscript file usingraveexport--run/path/to/your/run(--streaming)Setting the--streamingflag will enable cached convolutions, making the model compatible with realtime processing.If you forget to use the streaming mode and try to load the model in Max, you will hear clicking artifacts.PriorFor discrete models, we redirect the user to themspriorlibraryhere. However, as this library is still experimental, the prior from version 1.x has been re-integrated in v2.3.TrainingTo train a prior for a pretrained RAVE model :ravetrain_prior--model/path/to/your/run--db_path/path/to/your_preprocessed_data--out_path/path/to/outputthis will train a prior over the latent of the pretrained modelpath/to/your/run, and save the model and tensorboard logs to folder/path/to/output.ScriptingTo script a prior along with a RAVE model, export your model by providing the--priorkeyword to your pretrained prior :raveexport--run/path/to/your/run--prior/path/to/your/prior(--streaming)Pretrained modelsSeveral pretrained streaming modelsare available here. We'll keep the list updated with new models.Realtime usageThis section presents how RAVE can be loaded insidenn~in order to be used live with Max/MSP or PureData.ReconstructionA pretrained RAVE model nameddarbouka.ginavailable on your computer can be loaded insidenn~using the following syntax, where the default method is set to forward (i.e. encode then decode)This does the same thing as the following patch, but slightly faster.High-level manipulationHaving an explicit access to the latent representation yielded by RAVE allows us to interact with the representation using Max/MSP or PureData signal processing tools:Style transferBy default, RAVE can be used as a style transfer tool, based on the large compression ratio of the model. We recently added a technique inspired from StyleGAN to include Adaptive Instance Normalization to the reconstruction process, effectively allowing to definesourceandtargetstyles directly inside Max/MSP or PureData, using the attribute system ofnn~.Other attributes, such asenableorgpucan enable/disable computation, or use the gpu to speed up things (still experimental).Offline usageA batch generation script has been released in v2.3 to allow transformation of large amount of filesravegeneratemodel_pathpath_1path_2--outout_pathwheremodel_pathis the path to your trained model (original or scripted),path_Xa list of audio files or directories, andout_paththe out directory of the generations.DiscussionIf you have questions, want to share your experience with RAVE or share musical pieces done with the model, you can use theDiscussion tab!DemonstrationRAVE x nn~Demonstration of what you can do with RAVE and the nn~ external for maxmsp !embedded RAVEUsing nn~ for puredata, RAVE can be used in realtime on embedded platforms !FundingThis work is led at IRCAM, and has been funded by the following projectsANR MakiMonoACTORDAFNE+N° 101061548