Package: torchlambda
torchlambdais a tool to deployPyTorchmodels
onAmazon's AWS LambdausingAWS SDK for C++andcustom C++ runtime.Using statically compiled dependencieswhole package is shrunk to only30MB.Due to small size of compiled source code users can pass their models asAWS Lambda layers.Services likeAmazon S3are no longer necessary to load your model.torchlambdahas it's PyTorch & AWS dependencies always tested & up to date because of dailycontinuous deploymentruns.DocsDeploymentPackagePythonPyTorchDockerCodeBeatImages:heavy_check_mark: Why should I usetorchlambda?Lightweight & latest dependencies- compiled source code weights only30MB. Previous approach to PyTorch network deployment on AWS Lambda (fastai) uses outdated PyTorch (1.1.0) as dependency layer and requires AWS S3 to host your model. Now you can only use AWS Lambda and host your model as layer and PyTorchmasterand latest stable release are supported on a daily basis.Cheaper and less resource hungry- available solutions run server hosting incoming requests all the time. AWS Lambda (and torchlambda) runs only when the request comes.Easy automated scalingusually autoscaling is done withKubernetesor similar tools (seeKubeFlow). This approach requires knowledge of another tool, setting up appropriate services (e.g.Amazon EKS). In AWS Lambda case you just push your neural network inference code and you are done.Easy to use- no need to learn new tool.torchlambdahas at most4commands and deployment is done viaYAMLsettings. No need to modify your PyTorch code.Do one thing and do it well- most deployment tools are complex solutions
including multiple frameworks and multiple services.torchlambdafocuses
solely on inference of PyTorch models on AWS Lambda.Write programs to work together- This tool does not repeat PyTorch & AWS's functionalities (likeaws-cli). You can also use your favorite third party tools (saysaws,Terraformwith AWS andMLFlow,PyTorch-Lightningto train your model).Test locally, run in the cloud-torchlambdausesAmazon Linux 2Dockerimagesunder the hood & allows you to uselambci/docker-lambdato test your deployment onlocalhostbefore pushing deployment to the cloud (seeTest Lambda deployment locallytutorial).Extensible when you need it- All you usually need are a few lines of YAML settings, but if you wish to fine-tune your deployment you can usetorchlambda build--flags(changing various properties of PyTorch and AWS dependencies themselves). You can also write your own C++ deployment code (generate template viatorchlambda templatecommand).Small is beautiful-3000LOC (most being convenience wrapper creating this tool)
make it easy to jump into source code and check what's going on under the hood.:house: Table Of ContentsInstallationTutorialsResNet18 deployment on AWS LambdaTest Lambda deployment locallybase64image encodingCommandssettingstemplatebuildlayerYAML settings file referenceC++ code:page_with_curl: BenchmarksBenchmarks can be seen inBENCHMARKS.mdfile and are comprised of around ~30000 test cases.Results are divided based on settings used, model type, payload, AWS Lambda timing etc. Below is an example of how inference performance changes due to higher resolution images and type of encoding:128x128256x256512x5121024x1024base64120.622165.184311.129995.249byte133.315203.628498.3911738.97char128.331209.306517.4821822.56short135.859207.389497.8181740.91int133.42216.163519.5021783.02long126.979228.497516.981760.93float135.825223.045515.2451802.25double137.281209.267536.9591811.83Clearly the bigger image, the more important it is to usebase64encoding. For all results and descriptionclick here.:question: ContributingIf you find an issue or you think some functionality may be useful to you, pleaseopen new Issueorcreate Pull Requestwith your changes, thanks!