Package: neutone_sdk
Neutone SDKThe Neutone SDK is a tool for researchers that enables them to wrap their own audio models and run them in a DAW using ourNeutone Plugin. We offer functionality for both loading the models locally in the plugin and contributing them to the default list of models that is available to anyone running the plugin. We hope this will enable researchers to easily try their models in a DAW, but also provide creators with a collection of interesting models.Why use the Neutone SDKJUCEis the industry standard for building audio plugins. Because of this, knowledge of C++ is needed to be able to build even very simple audio plugins. However, it is rare for AI audio researchers to have extensive experience with C++ and be able to build such a plugin. Moreover, it is a serious time investment that could be spent developing better algorithms. Neutone makes it possible to build models using familiar tools such as PyTorch and with a minimal amount of Python code wrap these models such that they can be executed by the Neutone Plugin. Getting a model up and running inside a DAW can be done in less than a day without any need for C++ code or knowledge.The SDK provides support for automatic buffering of inputs and outputs to your model and on-the-fly sample rate and stereo-mono conversion. It enables a model that can only be executed with a predefined number of samples to be used in the DAW at any sampling rate and any buffer size seamlessly. Additionally, within the SDK tools for benchmarking and profiling are readily available so you can easily debug and test the performance of your models.Table of ContentsInstalling the SDKDownloading the Neutone PluginExamplesSDK DescriptionSDK UsageBenchmarking and ProfilingKnown issuesContributing to the SDKCreditsInstalling the SDKYou can installneutone_sdkusing pip:pip install neutone_sdkDownloading the PluginThe Neutone Plugin is available athttps://neutone.space. We currently offer VST3 and AU plugins that can be used to load the models created with this SDK. Please visit the website for more information.Examples and NotebooksIf you just want to wrap a model without going through a detailed description of what everything does we prepared these examples for you.The clipper example shows how to wrap a very simple PyTorch module that does not contain any AI model. Check it out for getting a high level overview of what is needed for wrapping a model. It is available atexamples/example_clipper.py.An example with a simple convolutional model based onmicro-tcncan be found atexamples/example_overdrive-random.py. This does not contain the training code and assumes you have a pretrainedmicro-tcnmodel.We also have Notebooks for more complicated models showing the entire workflow from training to exporting them using Neutone:TCN FX EmulationDDSP Timbre TransferRAVE Timbre TransferNoiseBandNet Audio ReconstructionSDK OverviewThe SDK provides functionality for wrapping existing PyTorch models in a way that can make them executable within the VST plugin. At its core the plugin is sending chunks of audio samples at a certain sample rate as an input and expects the same amount of samples at the output. The user of the SDK can specify what sample rate(s) and buffer size(s) their models perform optimally at. The SDK then guarantees that the forward pass of the model will receive audio at one of these (sample_rate, buffer_size) combinations. Four knobs are available that allow the users of the plugin to feed in additional parameters to the model at runtime. They can be enabled or disabled as needed via the SDK.Using the included export function a series of tests is automatically ran to ensure the models behave as expected and are ready to be loaded by the plugin.Benchmarking and profiling CLI tools are available for further debugging and testing of wrapped models. It is possible to benchmark the speed and latency of a model on a range of simulated common DAW (sample_rate, buffere_size) combinations as well as profile the memory and CPU usage.SDK UsageGeneral UsageWe provide several models in theexamplesdirectory. We will go through one of the simplest models, a distortion model, to illustrate.Assume we have the following PyTorch model. Parameters will be covered later on, we will focus on the inputs and outputs for now. Assume this model receives a Tensor of shape(2, buffer_size)as an input wherebuffer_sizeis a parameter that can be specified.classClipperModel(nn.Module):defforward(self,x:Tensor,min_val:float,max_val:float,gain:float)->Tensor:returntorch.clip(x,min=min_val*gain,max=max_val*gain)To run this inside the VST the simplest wrapper we can write is by subclassing the WaveformToWaveformBase baseclass.classClipperModelWrapper(WaveformToWaveformBase):@torch.jit.exportdefis_input_mono(self)->bool:returnFalse@torch.jit.exportdefis_output_mono(self)->bool:returnFalse@torch.jit.exportdefget_native_sample_rates(self)->List[int]:return[]# Supports all sample rates@torch.jit.exportdefget_native_buffer_sizes(self)->List[int]:return[]# Supports all buffer sizesdefdo_forward_pass(self,x:Tensor,params:Dict[str,Tensor])->Tensor:# ... Parameter unwrap logicx=self.model.forward(x,min_val,max_val,gain)returnxThe method that does most of the work isdo_forward_pass. In this case it is just a simple passthrough, but we will use it to handle parameters later on.By default the VST runs asstereo-stereobut when mono is desired for the model we can use theis_input_monoandis_output_monoto inform the SDK and have the inputs and outputs converted automatically. Ifis_input_monois toggled an averaged(1, buffer_size)shaped Tensor will be passed as an input instead of(2, buffer_size). Ifis_output_monois toggled,do_forward_passis expected to return a mono Tensor (shape(1, buffer_size)) that will then be duplicated across both channels at the output of the VST. This is done within the SDK to avoid unnecessary memory allocations during each pass.get_native_sample_ratesandget_native_buffer_sizescan be used to specify any preferred sample rates or buffer sizes. In most cases these are expected to only have one element but extra flexibility is provided for more complex models. In case multiple options are provided the SDK will try to find the best one for the current setting of the DAW. Whenever the sample rate or buffer size is different from the one of the DAW a wrapper is automatically triggered that converts to the correct sampling rate or implements a FIFO queue for the requested buffer size or both. This will incur a small performance penalty and add some amount of delay. In case a model is compatible with any sample rate and/or buffer size these lists can be left empty.This means that the tensorxin thedo_forward_passmethod is guaranteed to be of shape(1 if is_input_mono else 2, buffer_size)wherebuffer_sizewill be chosen at runtime from the list provided in theget_native_buffer_sizesmethod. The tensorxwill also be at one of the sampling rates from the list provided in theget_native_sample_ratesmethod.Exporting models and loading in the pluginWe provide asave_neutone_modelhelper function that saves models to disk. By default this will convert models to TorchScript and run them through a series of checks to ensure they can be loaded by the plugin. The resultingmodel.nmfile can be loaded within the plugin using theload your ownbutton. Read below for how to submit models to the default collection visible to everyone using the plugin.ParametersFor models that can use conditioning signals we currently provide four configurable knob parameters. Within theClipperModelWrapperdefined above we can include the following:classClipperModelWrapper(WaveformToWaveformBase):...defget_neutone_parameters(self)->List[NeutoneParameter]:return[NeutoneParameter(name="min",description="min clip threshold",default_value=0.5),NeutoneParameter(name="max",description="max clip threshold",default_value=1.0),NeutoneParameter(name="gain",description="scale clip threshold",default_value=1.0)]defdo_forward_pass(self,x:Tensor,params:Dict[str,Tensor])->Tensor:min_val,max_val,gain=params["min"],params["max"],params["gain"]x=self.model.forward(x,min_val,max_val,gain)returnxDuring the forward pass theparamsvariable will be a dictionary like the following:{"min":torch.Tensor([0.5]*buffer_size),"max":torch.Tensor([1.0]*buffer_size),"gain":torch.Tensor([1.0]*buffer_size)}The keys of the dictionary are specified in theget_parametersfunction.The parameters will always take values between 0 and 1 and thedo_forward_passfunction can be used to do any necessary rescaling before running the internal forward method of the model.Moreover, the parameters sent by the plugin come in at a sample level granularity. By default, we take the mean of each buffer and return a single float (as a Tensor), but theaggregate_parammethod can be used to override the aggregation method. See the full clipper export file for an example of preserving this granularity.Reporting delaySome audio models will delay the audio for a certain amount of samples. This depends on the architecture of each particular model. In order for the wet and dry signal that is going through the plugin to be aligned users are required to report how many samples of delay their model induces. Thecalc_model_delay_samplescan be used to specify the number of samples of delay. RAVE models on average have one buffer of delay (2048 samples) which is communicated statically in thecalc_model_delay_samplesmethod and can be seen in the examples. Models implemented with overlap-add will have a delay equal to the number of samples used for crossfading as seen in theDemucs model wrapperor thespectral filter example.Calculating the delay your model adds can be difficult, especially since there can be multiple different sources of delay that need to be combined (e.g. cossfading delay, filter delay, lookahead buffer delay, and / or neural networks trained on unaligned dry and wet audio). It's worth spending some extra time testing the model in your DAW to make sure the delay is being reported correctly.Lookbehind BuffersAdding a lookbehind buffer to your model can be useful for models that require a certain amount of additional context to output useful results. A lookbehind buffer can be enabled easily by indicating how many samples of lookbehind you need in theget_look_behind_samplesmethod. When this method returns a number greater than zero, thedo_forward_passmethod will always receive a tensor of shape(in_n_ch, look_behind_samples + buffer_size), but must still return a tensor of shape(out_n_ch, buffer_size)of the latest samples.We recommend avoiding using a look-behind buffer when possible since it makes your model less efficient and can result in wasted calculations during each forward pass. If using a purely convolutional model, try switching all the convolutions to cached convolutions instead.FiltersIt is common for AI models to act in unexpected ways when presented with inputs outside of the ones present in their training distribution. We provide a series of common filters (low bass, high pass, band pass, band stop) in theneutone_sdk/filters.pyfile. These can be used during the forward pass to restrict the domain of the inputs going into the model. Some of them can induce a small amount of delay, check out theexamples/example_clipper_prefilter.pyfile for a simple example on how to set up a filter.Submitting modelsThe plugin contains a default list of models aimed at creators that want to make use of them during their creative process. We encourage users to submit their models once they are happy with the results they get so they can be used by the community at large. For submission we require some additional metadata that will be used to display information about the model aimed at both creators and other researchers. This will be displayed on both theNeutone websiteand inside the plugin.Skipping the previous clipper model, here is a more realistic example based on a random TCN overdrive model inspired bymicro-tcn.classOverdriveModelWrapper(WaveformToWaveformBase):defget_model_name(self)->str:return"conv1d-overdrive.random"defget_model_authors(self)->List[str]:return["Nao Tokui"]defget_model_short_description(self)->str:return"Neural distortion/overdrive effect"defget_model_long_description(self)->str:return"Neural distortion/overdrive effect through randomly initialized Convolutional Neural Network"defget_technical_description(self)->str:return"Random distortion/overdrive effect through randomly initialized Temporal-1D-convolution layers. You'll get different types of distortion by re-initializing the weight or changing the activation function. Based on the idea proposed by Steinmetz et al."defget_tags(self)->List[str]:return["distortion","overdrive"]defget_model_version(self)->str:return"1.0.0"defis_experimental(self)->bool:returnFalsedefget_technical_links(self)->Dict[str,str]:return{"Paper":"https://arxiv.org/abs/2010.04237","Code":"https://github.com/csteinmetz1/ronn"}defget_citation(self)->str:return"Steinmetz, C. J., & Reiss, J. D. (2020). Randomized overdrive neural networks. arXiv preprint arXiv:2010.04237."Check out the documentation of the methods insidecore.py, as well as the random overdrive model on thewebsiteand in the plugin to understand where each field will be displayed.To submit a model, pleaseopen an issue on the GitHub repository. We currently need the following:A short description of what the model does and how it can contribute to the communityA link to themodel.nmfile outputted by thesave_neutone_modelhelper functionBenchmarking and ProfilingThe SDK provides three CLI tools that can be used to debug and test wrapped models.Benchmarking SpeedExample:$ python -m neutone_sdk.benchmark benchmark-speed --model_file model.nm
INFO:__main__:Running benchmark for buffer sizes (128, 256, 512, 1024, 2048) and sample rates (48000,). Outliers will be removed from the calculation of mean and std and displayed separately if existing.
INFO:__main__:Sample rate:  48000 | Buffer size:    128 | duration:  0.014±0.002 | 1/RTF:  5.520 | Outliers: [0.008]
INFO:__main__:Sample rate:  48000 | Buffer size:    256 | duration:  0.028±0.003 | 1/RTF:  5.817 | Outliers: []
INFO:__main__:Sample rate:  48000 | Buffer size:    512 | duration:  0.053±0.003 | 1/RTF:  6.024 | Outliers: []
INFO:__main__:Sample rate:  48000 | Buffer size:   1024 | duration:  0.106±0.000 | 1/RTF:  6.056 | Outliers: []
INFO:__main__:Sample rate:  48000 | Buffer size:   2048 | duration:  0.212±0.000 | 1/RTF:  6.035 | Outliers: [0.213]Running the speed benchmark will automatically run random inputs through the model at a sample rate of 48000 and buffer sizes of (128, 256, 512, 1024, 2048) and report the average time taken to execute inference for one buffer. From this the1/RTFis calculated which represents how much faster than realtime the model is. As this number gets higher, the model will use fewer resources within the DAW. It is necessary for this number to be bigger than 1 for the model to be able to be executed in realtime on the machine that the benchmark is ran on.The sample rates and buffer sizes being tested, as well as the number of times the benchmark is internally repeated to calculate the averages and the number of threads used for the computation are available as parameters. Runpython -m neutone_sdk.benchmark benchmark-speed --helpfor more information. When specifiying custom sample rates or buffer sizes each individual one needs to be passed to the CLI separately. For example:--sample_rate 48000 --sample_rate 44100 --buffer_size 32 --buffer_size 64.While the speed benchmark should be fast as the models are generally speaking required to be realtime it is possible to get stuck if the model is too slow. Make sure you choose an appropiate number of sample rates and buffer sizes to test.Benchmarking LatencyExample:$python-mneutone_sdk.benchmarkbenchmark-latencymodel.nmINFO:__main__:Nativebuffersizes:[2048],Nativesamplerates:[48000]INFO:__main__:Modelexports/ravemodel/model.nmhasthefollowingdelaysforeachsamplerate/buffersizecombination(lowestdelayfirst):
INFO:__main__:Samplerate:48000|Buffersize:2048|Totaldelay:0|(Bufferingdelay:0|Modeldelay:0)INFO:__main__:Samplerate:48000|Buffersize:1024|Totaldelay:1024|(Bufferingdelay:1024|Modeldelay:0)INFO:__main__:Samplerate:48000|Buffersize:512|Totaldelay:1536|(Bufferingdelay:1536|Modeldelay:0)INFO:__main__:Samplerate:48000|Buffersize:256|Totaldelay:1792|(Bufferingdelay:1792|Modeldelay:0)INFO:__main__:Samplerate:44100|Buffersize:128|Totaldelay:1920|(Bufferingdelay:1920|Modeldelay:0)INFO:__main__:Samplerate:48000|Buffersize:128|Totaldelay:1920|(Bufferingdelay:1920|Modeldelay:0)INFO:__main__:Samplerate:44100|Buffersize:256|Totaldelay:2048|(Bufferingdelay:2048|Modeldelay:0)INFO:__main__:Samplerate:44100|Buffersize:512|Totaldelay:2048|(Bufferingdelay:2048|Modeldelay:0)INFO:__main__:Samplerate:44100|Buffersize:1024|Totaldelay:2048|(Bufferingdelay:2048|Modeldelay:0)INFO:__main__:Samplerate:44100|Buffersize:2048|Totaldelay:2048|(Bufferingdelay:2048|Modeldelay:0)Running the speed benchmark will automatically compute the latency of the model at combinations ofsample_rate=(44100, 48000)andbuffer_size=(128, 256, 512, 1024, 2048). This gives a general overview of what will happen for common DAW settings. The total delay is split into buffering delay and model delay. The model delay is reported by the creator of the model in the model wrapper as explainedabove. The buffering delay is automatically computed by the SDK taking into consideration the combination of(sample_rate, buffer_size)specified by the wrapper (the native ones) and the one specified by the DAW at runtime. Running the model at its native(sample_rate, buffer_size)combination(s) will incur minimum delay.Similar to the speed benchmark above, the tested combinations of(sample_rate, buffer_size)can be specified from the CLI. Runpython -m neutone_sdk.benchmark benchmark-latency --helpfor more info.Profiling$python-mneutone_sdk.benchmarkprofile--model_fileexports/ravemodel/model.nm
INFO:__main__:Profilingmodelexports/ravemodel/model.nmatsamplerate48000andbuffersize128STAGE:2023-09-2814:34:5396328:4714960ActivityProfilerController.cpp:311]CompletedStage:WarmUp
30it[00:00,37.32it/s]STAGE:2023-09-2814:34:5496328:4714960ActivityProfilerController.cpp:317]CompletedStage:Collection
STAGE:2023-09-2814:34:5496328:4714960ActivityProfilerController.cpp:321]CompletedStage:PostProcessing
INFO:__main__:DisplayingTotalCPUTime
INFO:__main__:--------------------------------------------------------------------------------------------------------------------------------NameSelfCPU%SelfCPUCPUtotal%CPUtotalCPUtimeavgCPUMemSelfCPUMem# of Calls--------------------------------------------------------------------------------------------------------------------------------forward98.54%799.982ms102.06%828.603ms26.729ms0b-918.17Kb31aten::convolution0.12%963.000us0.95%7.739ms175.886us530.62Kb-143.50Kb44...
...
FulloutputremovedfromGitHub.The profiling tool will run the model at a sample rate of 48000 and a buffer size of 128 under the PyTorch profiler and output a series of insights, such as the Total CPU Time, Total CPU Memory Usage (per function) and Grouped CPU Memory Usage (per group of function calls). This can be used to identify bottlenecks in your model code (even within the model call within thedo_forward_passcall).Similar to benchmarking, it can be ran at different combinations of sample rates and buffer sizes as well as different numbers of threads. Runpython -m neutone_sdk.benchmark profile --helpfor more info.Known issuesFreezing models on save can cause instabilities and thus freezing is disabled by default. We recommend trying to save models both with and without freeze.Displaying some metadata information does not currently work with local model loading in the plugin.Lookahead buffers will be implemented at the SDK level in the near future but is currently possible with additional code. An example is available inthis file.M1 acceleration is currently not supported.Wrapping more complicated models can result in obscure TorchScript errors.Contributing to the SDKWe welcome any contributions to the SDK. Please add types wherever possible and use theblackformatter for readability.The current roadmap is:Adding a TCN library and an overhaul of the TCN-based example modelsSupporting non-realtime modelsSupporting models with multiple inputs and / or outputsSupporting more and different types of parametersLooking into alternatives for TorchScriptCreditsThe audacitorch project was a major inspiration for the development of the SDK.Check it out here