Package: audacitorch
audacitorchThis package contains utilities for prepping PyTorch audio models for use in Audacity. More specifically, it provides abstract classes for you to wrap your waveform-to-waveform and waveform-to-labels models (see theDeep Learning for Audacitywebsite to learn more about deep learning models for audacity).Table of ContentsDownloading Audacity with Deep LearningInstallingContributing Models to AudacityChoosing an Effect TypeWaveform to Waveform modelsWaveform to Labels modelsModel MetadataMaking Your Model Built-In to AudacityDebugging Your Model in AudacityExample - Waveform-to-WaveformMaking sure your model is compatible withtorchscriptExample - Exporting a Pretrained Asteroid ModelDownload Audacity with Deep LearningOur work has not yet been merged to the main build of Audacity, though it will be soon. You can keep track of its progress by viewing ourpull request. In the meantime, you can download an alpha version ofAudacity + Deep Learning here.InstallingYou can installaudacitorchusing pip:pip install -e "git+https://github.com/hugofloresgarcia/audacitorch.git#egg=audacitorch"Contributing Models to AudacitySupported Torch versionsaudacitorchrequires for your model to be able to run inTorch 1.9.0, as that's what the Audacity torchscript interpreter uses.Deep Learning Effect and AnalyzerAudacity is equipped with a wrapper framework for deep learning models written in PyTorch. Audacity contains two deep learning tools:Deep Learning EffectandDeep Learning Analyzer.Deep Learning Effectperforms waveform to waveform processing, and is useful for audio-in-audio-out tasks (such as source separation, voice conversion, style transfer, amplifier emulation, etc.), whileDeep Learning Analyzerperforms waveform to labels processing, and is useful for annotation tasks (such as sound event detection, musical instrument recognition, automatic speech recognition, etc.).audacitorchcontains two abstract classes for serializing two types of models: waveform-to-waveform and waveform-to-labels. The classes areWaveformToWaveformBase, andWaveformToLabelsBase, respectively.Choosing an Effect TypeWaveform to Waveform modelsAs shown in theeffect diagram, Waveform-to-waveform models receive a single multichannel audio track as input, and may write to a variable number of new audio tracks as output.Example models for waveform-to-waveform effects include source separation, neural upsampling, guitar amplifier emulation, generative models, etc. Output tensors for waveform-to-waveform models must be multichannel waveform tensors with shape(num_output_channels, num_samples). For every audio waveform in the output tensor, a new audio track is created in the Audacity project.Waveform to Labels modelsAs shown in theeffect diagram, Waveform-to-labels models receive a single multichannel audio track as input, and may write to an output label track as output. The waveform-to-labels effect can be used for many audio analysis applications, such as voice activity detection, sound event detection, musical instrument recognition, automatic speech recognition, etc. The output for waveform-to-labels models must be a tuple of two tensors. The first tensor corresponds to the class indexes for each label present in the waveform, shape(num_timesteps,). The second tensor must contain timestamps with start and stop times for each label, shape(num_timesteps, 2).What If My Model Uses a Spectrogram as Input/Output?If your model uses a spectrogram as input/output, you'll need to wrap your forward pass with some torchscript-compatible preprocessing/postprocessing. We recommend usingtorchaudio, writing your own preprocessing transforms in their ownnn.Module, or writing your PyTorch-only preprocessing and placing it inWaveformToWaveform.do_forward_passorWaveformToLabels.do_forward_pass. See thecompatibilitysection for more info.Model MetadataCertain details about the model, such as its sample rate, tool type (e.g. waveform-to-waveform or waveform-to-labels), list of labels, etc. must be provided by the model contributor in a separatemetadata.jsonfile. In order to help users choose the correct model for their required task, model contributors are asked to provide a short and long description of the model, the target domain of the model (e.g. speech, music, environmental, etc.), as well as a list of tags or keywords as part of the metadata. Seeherefor an example metadata dictionary.Metadata Specrequired fields:sample_rate(int)range(0, 396000)Model sample rate. Input tracks will be resampled to this value.domains(List[str])List of data domains for the model. The list should contain any of the following strings (any others will be ignored):["music", "speech", "environmental", "other"]short_description(str)max 60 charsshort description of the model. should contain a brief message with the model's purpose, e.g. "Use me for separating vocals from the background!".long_description(str)max 280 charslong description of the model. Shown in the detailed view of the model UI.tags(List[str])list of tags (to be shown in the detailed view)each tag should be 15 characters maxmax 5 tags per model.labels(List[str)output labels for the model. Depending on the effect type, this field means different thingswaveform-to-waveformname of each output source (e.g.drums,bass,vocal). To create the track name for each output source, each one of the labels will be appended to the mixture track's name.waveform-to-labels:This should be classlist for model. The class indexes output by the model during a forward pass will be used to index into this classlist.effect_type(str)Target effect for this model. Must be one of["waveform-to-waveform", "waveform-to-labels"].multichannel(bool)Ifmultichannelis set totrue, stereo tracks are passed to the model as multichannel audio tensors, with shape(2, n). Note that this means that the input could either be a mono track with shape(1, n)or stereo track with shape(2, n).Ifmultichannelis set tofalse, stereo tracks are downmixed, meaning that the input audio tensor will always be shape(1, n).Making Your Model Built-In To AudacityBy default, users have to click on theAdd From HuggingFacebutton on the Audacity Model Manager and enter the desired repo's ID to install a community contributed model. If you, instead, would like your community contributed model to show up in Audacity's Model Manager by default, please open a requesthere.Example - Waveform-to-Waveform modelHere's a minimal example for a model that simply boosts volume by multiplying the incoming audio by a factor of 2.We can sum up the whole process into 4 steps:Developing your modelWrapping your model usingaudacitorchCreating a metadata documentExporting to HuggingFaceDeveloping your modelFirst, we create our model. There are no internal constraints on what the internal model architecture should be, as long as you can usetorch.jit.scriptortorch.jit.traceto serialize it, and it is able to meet the input-output constraints specified in waveform-to-waveform and waveform-to-labels models.importtorchimporttorch.nnasnnclassMyVolumeModel(nn.Module):defforward(self,x:torch.Tensor)->torch.Tensor:# do the neural net magic!x=x*2returnxMaking sure your model is compatible withtorchscriptPyTorch makes it really easy to deploy your Python models in C++ by usingtorchscript, an intermediate representation format for torch models that can be called in C++. Many of Python's built-in functions are supported by torchscript. However, not all Python operations are supported by the torchscript environment, meaning that you are only allowed to use a subset of Python operations in your model code. Seethe torch.jit docsto learn more about writing torchscript-compatible code.If your model computes spectrograms (or requires any kind of preprocessing/postprocessing), make sure those operations are compatible with torchscript, liketorchaudio's operation set.Useful links:Torchscript referencePytorch's tutorial on torchscript modelsA 1:1 mapping of the features in python to their support in torchscript(recommended)Mastering Torchscript: Tracing vs Scripting, Device Pinning, Direct Graph ModificationWrapping your model usingaudacitorchNow, we create a wrapper class for our model. Because our model returns an audio waveform as output, we'll useWaveformToWaveformBaseas our parent class. For bothWaveformToWaveformBaseandWaveformToLabelsBase, we need to implement thedo_forward_passmethod with our processing code. See thedocstringsfor more details.fromaudacitorchimportWaveformToWaveformBaseclassMyVolumeModelWrapper(WaveformToWaveformBase):defdo_forward_pass(self,x:torch.Tensor)->torch.Tensor:# do any preprocessing here!# expect x to be a waveform tensor with shape (n_channels, n_samples)output=self.model(x)# do any postprocessing here!# the return value should be a multichannel waveform tensor with shape (n_channels, n_samples)returnoutputCreating a metadata documentAudacity models need a metadata file. See the metadataspecto learn about the required fields.metadata={'sample_rate':48000,'domain_tags':['music','speech','environmental'],'short_description':'Use me to boost volume by 3dB :).','long_description':'This description can be a max of 280 characters aaaaaaaaaaaaaaaaaaaa.','tags':['volume boost'],'labels':['boosted'],'effect_type':'waveform-to-waveform','multichannel':False,}All set! We can now proceed to serialize the model to torchscript and save the model, along with its metadata.frompathlibimportPathfromaudacitorch.utilsimportsave_model,validate_metadata,\get_example_inputs,test_run# create a root dir for our modelroot=Path('booster-net')root.mkdir(exist_ok=True,parents=True)# get our modelmodel=MyVolumeModel()# wrap itwrapper=MyVolumeModelWrapper(model)# serialize it using torch.jit.script, torch.jit.trace,# or a combination of both.# option 1: torch.jit.script# using torch.jit.script is preferred for most cases,# but may require changing a lot of source codeserialized_model=torch.jit.script(wrapper)# option 2: torch.jit.trace# using torch.jit.trace is typically easier, but you# need to be extra careful that your serialized model behaves# properly after tracingexample_inputs=get_example_inputs()serialized_model=torch.jit.trace(wrapper,example_inputs[0],check_inputs=example_inputs)# take your model for a test run!test_run(serialized_model)# check that we created our metadata correctlysuccess,msg=validate_metadata(metadata)assertsuccess# save!save_model(serialized_model,metadata,root)Exporting to HuggingFaceYou should now have a directory structure that looks like this:/booster-net/
/booster-net/model.pt
/booster-net/metadata.jsonThis will be the repository for your audacity model. Make sure to add a readme with theaudacitytag in the YAML metadata, so it show up on the explore tab of Audacity's Deep Learning Tools.Create aREADME.mdinsidebooster-net/, and add the following header:in README.md---
tags: audacity
---Awesome! It's time to push to HuggingFace. See theirdocumentationfor adding a model to the HuggingFace model hub.Debugging Your Model in AudacityAfter serializing, you may need to debug your model inside Audacity, to make sure that it handles inputs correctly, doesn't crash while processing, and produces the correct output.
While debugging, make sure your model isn't available through other users through theExplore HuggingFacebutton by temporarily removing theaudacitytag from your README file.
If your model fails internally while processing audio, you may see something like this:To debug, you can access the error logs through the Help menu, inHelp->Diagnostics->Show Log.... Any torchscript errors that may occur during the forward pass will be redirected here.Example - Exporting a PretrainedAsteroidmodelSee thisexample notebook, where we serialize a pretrained ConvTasNet model for speech separation using theAsteroidsource separation library.Example - Exporting a PretrainedS2TmodelSee thisexample notebook, where we serialize a pretrained speech to text transformer from Facebook.