Package: nn-tilde
Demonstration videoInstallationGrab thelatest release of nn~! Be sure to download the correct version for your installation.Mac users, you should download the arm64 version ifand only ifyour Mac is a M1-2, and your installation of Max/MSP or PureData is not using Rosetta !Windows users, for now it is required that all.dllfiles in thenn~package are copied next to the ˋMax.exeˋ executable.MaxMSPUncompress the.tar.gzfile in the Package folder of your Max installation, i.e. inDocuments/Max 8/Packages/.You can then instantiate annn~object. Depending on your installation, you might get a quarantine warning, such as this oneIn most cases, proceeding with the removal will disable this warning until the next update ofnn~. If MacOS continues to block the external, you might have to compile it yourself.Right click on thenn~object to open the help patch, and follow the tabs to learn more about this project.PureDataUncompress the.tar.gzfile in the Package folder of your Pd installation, i.e. inDocuments/Pd/externals/. You can then add a new path in thePd/File/Preferences/Pathmenu pointing to thenn_tildefolder.On MacOS, unlike Max/MSP, PureData doesn't include an automatic quarantine removal procedure. Therefore we have to do it manually. Fire up a terminal, andcdto thenn_tildefolder. The following command will fix the issuexattr-r-dcom.apple.quarantine.UsagePretrained modelsAt its core,nn~is a translation layer between Max/MSP or PureData and thelibtorch c++ interface for deep learning. Alone,nn~is like an empty shell, andrequires pretrained modelsto operate. You can find a fewRAVEmodelshere, or a fewvschaos2modelshere.Pretrained model fornn~aretorchscript files, with a.tsextension. Create a folder somewhere on your computer, for example inDocuments/pretrained_models/and add this path to Max or PureData include path (Max:Options/File Preferences, PureData:File/Preferences/Path).Loading a modelOnce this is done, you can load a model using the following syntax (here to load a pretrained model located inDocuments/pretrained_models/decoder_only.ts)Max / MSPPureDataNote that youhaveto include the.tsextension in the PureData version. Depending on the model loaded, there will be a different number of inlets / outlets, corresponding to the different inputs and outputs of the model.Selecting a methodA given pretrained model might have several differentmethods, with different effects and usage. For example, theRAVE modelhas three different methods, corresponding to different subparts of the model. For the sake of the example, we will describe them in the following table:Method nameDescriptionInputsOutputsencodeEncodes an audio signal into a latent trajectoryaudio signalmultiple latent trajectoriesdecodeDecodes a latent trajectory into an audio signalmultiple latent trajectoriesaudio signalforward(default)Encodesanddecodes an audio signalaudio signalaudio signalThe user can switch between methods during instanciation by adding a second argument tonn~calling the desired method.Max / MSPPureDataUsing attributesIt is possible to configure models after their initialization using specialattributes, whose type and effect is entirely defined by the model itself, with the exception of theenableattribute which can be set to either 0 or 1 to enable or disable the model, potentially saving up computation.Model attributes can be set usingmessages, with the following syntax:setATTRIBUTE_NAMEATTRIBUTE_VAL_1ATTRIBUTE_VAL_2Using Max/MSP and PureData graphical objects, this can lead to an intuitive way to modify the behavior of the model, as shown below where we have two model attributes (i.e. generation temperature and generation mode), and the specialenableattribute.Max / MSPPureDataThe attribute list and values are againmodel dependantand should retrieved in the documentation of the model.Buffer configurationInternally,nn~has a circular buffer mechanism that helps maintain a reasonable computational load. You can modify its size through the use of an additional integer after the method declaration, as shown belowMax / MSPPureDataMulticanal (Max/MSP)The Max/MSP release ofnn~includes additional externals, namelymc.nn~andmcs.nn~, allowing the use of the multicanal abilities of Max 8+ to simplify the patching process withnn~and optionally decrease the computational load.In the following examples, two audio files are being encoded then decoded by the same model in parallelThis patch can be improved both visuallyandcomputationally speaking by usingmc.nn~and usingbatch operationsUsingmc.nn~we build the multicanal signalsover the different batches. In the example above, each multicanal signal will have 2 different canals. We also propose themcs.nn~external that builds multicanal signalsover the different dimensions, as shown in the example belowIn the example above, the two multicanals signals yielded by thenn~ rave encode 2object have 16 canals each, corresponding to the 16 latent dimensions. This can help patching, while keeping the batching abilities ofmc.nn~by creating an explicit number of inlets / oulets corresponding to the number of examples we want to process in parallel.To recap, the regularnn~operates on a single example, and has as many inlets / outlets as the model has inputs / outputs. Themc.nn~external is likenn~, but can process multiple examplesat the same time. Themcs.nn~variant is a bit different, and can process mulitple examples at the same time, but willhave one inlet / outlet per examples.Special messagesenable [0 / 1]Enable / Disable computation to save up computation without deleting the model. Similar to how abypassfunction would work.reloadDynamically reloads the model. Can be useful if you want to periodically update the state of a model during a training.Build InstructionsmacOSDownload the latest libtorch (CPU)hereand unzip it to a known directoryRun the following commands:gitclonehttps://github.com/acids-ircam/nn_tilde--recursivecdnn_tilde
mkdirbuildcdbuild
cmake../src/-DCMAKE_PREFIX_PATH=/path/to/libtorch-DCMAKE_BUILD_TYPE=Release
makeCopy the produced.mxoexternal inside~/Documents/Max 8/Packages/nn_tilde/externals/You can buildnn~for PureData by adding-DPUREDATA_INCLUDE_DIR=/Applications/Pd-X.XX-X.app/Contents/Resources/src/to the cmake call.WindowsDownload Libtorch (CPU) and dependencieshereand unzip to a known directory.Install Visual Studio and the C++ toolsRun the following commands:gitclonehttps://github.com/acids-ircam/nn_tilde--recurse-submodulescdnn_tilde
mkdirbuildcdbuild
cmake..\src-Ax64-DCMAKE_PREFIX_PATH="<unzipped libtorch directory>"cmake--build.--configReleaseRaspberry PiWhile nn~ can be compiled and used on Raspberry Pi, you may have to consider using lighter deep learning models. We currently only support 64bit OS.Install nn~ for PureData usingcurl-shttps://raw.githubusercontent.com/acids-ircam/nn_tilde/master/install/raspberrypi.sh|bashFundingThis work is led at IRCAM, and has been funded by the following projectsANR MakiMonoACTORDAFNE+N° 101061548