Package: torchdeploy
torch::deploy(MultiPy)torch::deploy(MultiPy for non-PyTorch use cases) is a C++ library that enables you to run eager mode PyTorch models in production without any modifications to your model to support tracing.torch::deployprovides a way to run using multiple independent Python interpreters in a single process without a shared global interpreter lock (GIL). For more information on howtorch::deployworks
internally, please see the relatedarXiv paper.To learn how to usetorch::deployseeInstallationandExamples.Requirements:PyTorch 1.13+ or PyTorch nightlyLinux (ELF based)x86_64 (Beta)arm64/aarch64 (Prototype)ℹ️torch::deployis ready for use in production environments, but is in Beta and may have some rough edges that we're continuously working on improving. We're always interested in hearing feedback and usecases that you might have. Feel free to reach out!InstallationBuilding via DockerThe easiest way to build deploy and install the interpreter dependencies is to do so via docker.gitclone--recurse-submoduleshttps://github.com/pytorch/multipy.gitcdmultipyexportDOCKER_BUILDKIT=1dockerbuild-tmultipy.The built artifacts are located inmultipy/runtime/build.To run the tests:dockerrun--rmmultipymultipy/runtime/build/test_deployInstalling viapip installWe support installing both python modules and the runtime libs usingpip install, with the caveat of having to manually install the C++ dependencies
first. This serves as a single-command source build, essentially being a wrapper
aroundpython setup.py develop, once all the dependencies have been installed.To start with, the multipy repo should be cloned first:gitclone--recurse-submoduleshttps://github.com/pytorch/multipy.gitcdmultipy# (optional) if using existing checkoutgitsubmodulesync&&gitsubmoduleupdate--init--recursiveInstalling System DependenciesThe runtime system dependencies are specified inbuild-requirements.txt. To install them on Debian-based systems, one could run:sudoaptupdate
xargssudoaptinstall-y-qq--no-install-recommends<build-requirements.txtPython Environment SetupWe support bothcondaandpyenv+virtualenvto create isolated environments to build and run in. Sincemultipyrequires a position-independent version of python to launch interpreters with, forcondaenvironments we use the prebuiltlibpython-static=3.xlibraries fromconda-forgeto link with at build time, and forvirtualenv/pyenvwe compile python with-fPICto create the linkable library.NOTEWe support Python versions 3.7 through 3.10 formultipy; note that forcondaenvironments thelibpython-staticlibraries are available for3.8onwards. Withvirtualenv/pyenvany version from 3.7 through 3.10 can be used, as the PIC library is built explicitly.Click to expandExample commands for installing conda:curl-fsSL-v-o~/miniconda.sh-Ohttps://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh&&\chmod+x~/miniconda.sh&&\~/miniconda.sh-b-p/opt/conda&&\rm~/miniconda.shVirtualenv / pyenv can be installed as follows:pip3installvirtualenv
gitclonehttps://github.com/pyenv/pyenv.git~/.pyenvInstalling python, pytorch and related dependenciesMultipy requires the latest version of pytorch to run models successfully, and we recommend fetching the latestnightliesfor pytorch and also cuda, if required.In acondaenvironment, we would do the following:condacreate-nnewenv
condaactivatenewenv
condainstallpython=3.8
condainstall-cconda-forgelibpython-static=3.8# cudacondainstallpytorchtorchvisiontorchaudiocudatoolkit=11.3-cpytorch-nightly# cpu onlycondainstallpytorchtorchvisiontorchaudiocpuonly-cpytorch-nightlyFor apyenv/virtualenvsetup, one could do:exportCFLAGS="-fPIC -g"~/.pyenv/bin/pyenvinstall--force3.8.6
virtualenv-p~/.pyenv/versions/3.8.6/bin/python3~/venvs/multipysource~/venvs/multipy/bin/activate
pipinstall-rdev-requirements.txt# cudapip3install--pretorchtorchvisiontorchaudio--extra-index-urlhttps://download.pytorch.org/whl/nightly/cu113# cpu onlypip3install--pretorchtorchvisiontorchaudio--extra-index-urlhttps://download.pytorch.org/whl/nightly/cpuRunningpip installOnce all the dependencies are successfully installed, most importantly including a PIC-library of python and the latest nightly of pytorch, we can run the following, in eithercondaorvirtualenv, to install both the python modules and the runtime/interpreter libraries:# from base multipy directorypipinstall-e.The C++ binaries should be available in/opt/dist.Alternatively, one can install only the python modules without invokingcmakeas follows:pipinstall-e.--install-option="--cmakeoff"NOTEAs of 10/11/2022 the linking of prebuilt static fPIC versions of python downloaded fromconda-forgecan be problematic on certain systems (for example Centos 8), with linker errors likelibpython_multipy.a: error adding symbols: File format not recognized. This seems to be an issue withbinutils, and the steps inhttps://wiki.gentoo.org/wiki/Project:Toolchain/Binutils_2.32_upgrade_notes/elfutils_0.175:_unable_to_initialize_decompress_status_for_section_.debug_infocan help. Alternatively, the user can go with thevirtualenv/pyenvflow above.DevelopmentManually buildingmultipy::runtimefrom sourceBothdockerandpip installoptions above are wrappers around thecmakebuild of multipy's runtime. For development purposes it's often helpful to
invokecmakeseparately.See the install section for how to correctly setup the Python environment.# checkout repogitclone--recurse-submoduleshttps://github.com/pytorch/multipy.gitcdmultipy# (optional) if using existing checkoutgitsubmodulesync&&gitsubmoduleupdate--init--recursivecdmultipy# install python parts of `torch::deploy` in multipy/multipy/utilspipinstall-e.--install-option="--cmakeoff"cdmultipy/runtime# configure runtime to build/cmake-S.-Bbuild# if you need to override the ABI setting you can passcmake-S.-Bbuild-D_GLIBCXX_USE_CXX11_ABI=<0/1># compile the files in build/cmake--buildbuild--configRelease-jRunning unit tests formultipy::runtimeWe first need to generate the neccessary examples. First make sure your python environment hastorch. Afterwards, oncemultipy::runtimeis built, run the following (executed automatically fordockerandpipabove):cd multipy/multipy/runtime
python example/generate_examples.py
cd build
./test_deployExamplesSee theexamples directoryfor complete examples.Packaging a modelfor multipy::runtimemultipy::runtimecan load and run Python models that are packaged withtorch.package. You can learn more abouttorch.packagein thetorch.packagedocumentation.For now, let's create a simple model that we can load and run inmultipy::runtime.fromtorch.packageimportPackageExporterimporttorchvision# Instantiate some modelmodel=torchvision.models.resnet.resnet18()# Package and export it.withPackageExporter("my_package.pt")ase:e.intern("torchvision.**")e.extern("numpy.**")e.extern("sys")e.extern("PIL.*")e.extern("typing_extensions")e.save_pickle("model","model.pkl",model)Note that since "numpy", "sys", "PIL" were marked as "extern",torch.packagewill
look for these dependencies on the system that loads this package. They will not be packaged
with the model.Now, there should be a file namedmy_package.ptin your working directory.Load the model in C++#include<multipy/runtime/deploy.h>#include<multipy/runtime/path_environment.h>#include<torch/script.h>#include<torch/torch.h>#include<iostream>#include<memory>intmain(intargc,constchar*argv[]){if(argc!=2){std::cerr<<"usage: example-app <path-to-exported-script-module>\n";return-1;}// Start an interpreter manager governing 4 embedded interpreters.std::shared_ptr<multipy::runtime::Environment>env=std::make_shared<multipy::runtime::PathEnvironment>(std::getenv("PATH_TO_EXTERN_PYTHON_PACKAGES")// Ensure to set this environment variable (e.g. /home/user/anaconda3/envs/multipy-example/lib/python3.8/site-packages));multipy::runtime::InterpreterManagermanager(4,env);try{// Load the model from the multipy.package.multipy::runtime::Packagepackage=manager.loadPackage(argv[1]);multipy::runtime::ReplicatedObjmodel=package.loadPickle("model","model.pkl");}catch(constc10::Error&e){std::cerr<<"error loading the model\n";std::cerr<<e.msg();return-1;}std::cout<<"ok\n";}This small program introduces many of the core concepts ofmultipy::runtime.AnInterpreterManagerabstracts over a collection of independent Python
interpreters, allowing you to load balance across them when running your code.PathEnvironmentenables you to specify the location of Python
packages on your system which are external, but necessary, for your model.Using theInterpreterManager::loadPackagemethod, you can load amultipy.packagefrom disk and make it available to all interpreters.Package::loadPickleallows you to retrieve specific Python objects
from the package, like the ResNet model we saved earlier.Finally, the model itself is aReplicatedObj. This is an abstract handle to
an object that is replicated across multiple interpreters. When you interact
with aReplicatedObj(for example, by callingforward), it will select
an free interpreter to execute that interaction.Build and execute the C++ exampleAssuming the above C++ program was stored in a file called,example-app.cpp, a
minimalCMakeLists.txtfile would look like:cmake_minimum_required(VERSION3.12FATAL_ERROR)project(multipy_tutorial)set(MULTIPY_PATH".."CACHEPATH"The repo where multipy is built or the PYTHONPATH")# include the multipy utils to help link againstinclude(${MULTIPY_PATH}/multipy/runtime/utils.cmake)# add headers from multipyinclude_directories(${MULTIPY_PATH})# link the multipy prebuilt binaryadd_library(multipy_internalSTATICIMPORTED)set_target_properties(multipy_internalPROPERTIESIMPORTED_LOCATION${MULTIPY_PATH}/multipy/runtime/build/libtorch_deploy.a)caffe2_interface_library(multipy_internalmultipy)add_executable(example-appexample-app.cpp)target_link_libraries(example-appPUBLIC"-Wl,--no-as-needed -rdynamic"dlpthreadutilmultipyc10torch_cpu)Currently, it is necessary to buildmultipy::runtimeas a static library.
In order to correctly link to a static library, the utilitycaffe2_interface_libraryis used to appropriately set and unset--whole-archiveflag.Furthermore, the-rdynamicflag is needed when linking to the executable
to ensure that symbols are exported to the dynamic table, making them accessible
to the deploy interpreters (which are dynamically loaded).Updating LIBRARY_PATH and LD_LIBRARY_PATHIn order to locate dependencies provided by PyTorch (e.g.libshm), we need to update theLIBRARY_PATHandLD_LIBRARY_PATHenvironment variables to include the path to PyTorch's C++ libraries. If you installed PyTorch using pip or conda, this path is usually in the site-packages. An example of this is provided below.exportLD_LIBRARY_PATH="$LD_LIBRARY_PATH:/home/user/anaconda3/envs/multipy-example/lib/python3.8/site-packages/torch/lib"exportLIBRARY_PATH="$LIBRARY_PATH:/home/user/anaconda3/envs/multipy-example/lib/python3.8/site-packages/torch/lib"The last step is configuring and building the project. Assuming that our code
directory is laid out like this:example-app/
    CMakeLists.txt
    example-app.cppWe can now run the following commands to build the application from within theexample-app/folder:cmake-S.-Bbuild-DMULTIPY_PATH="/home/user/repos/multipy"# the parent directory of multipy (i.e. the git repo)cmake--buildbuild--configRelease-jNow we can run our app:./example-app/path/to/my_package.ptContributingWe welcome PRs! See theCONTRIBUTINGfile.LicenseMultiPy is BSD licensed, as found in theLICENSEfile.LegalTerms of UsePrivacy PolicyCopyright (c) Meta Platforms, Inc. and affiliates.
All rights reserved.