Package: fastnn
Fast Neural Networks (FastNN)A framework for deploying serializable and optimizable neural net models at scale in production via. the NVIDIA Triton Inference Server.FastNN Docker Release Selector (Ubuntu 18.04)Documentation- Guides, Models, API ReferencesFeatures:Data ProcessingIntuitive data processing modules for encoding human-readible data into tensors compatible with deep learning librariesModel ExportingFastNN torch modules and tools for exporting models via.TorchScripttracing and scripting to a production environment. Now includes Text Generation.Model ZooVarious exported models hosted in this repo via. git-lfs and AWS S3. Includes models from the HuggingFace's Transformers and
TorchVisionModel DeploymentDeploy models using Triton Inference Server on CPU/GPU-compatible server(s) with helm or dockerFastNN ClientClient wrapper for Triton Inference Server's client module for programmatic requests with pythonPre-Requisites:Git LFS is required if you'd like to use any of the models provided by FastNN in./model_repository.Cloning this repository without Git LFS will clone a repository with LFS pointers, not the actual model.After the repository is cloned and Git LFS is installed, set up withgit lfs install.Download specific models with:gitlfspull--include="model_repository/<path-to-model-dir>"--exclude=""Download ALL models with:gitlfspull# Or#git lfs pull --include="*" --exclude=""Quickstart and Installation:Pre-requisites:Requirements: Python 3.9+, PyTorch 2+, Triton ClientOptional: CUDA Compatible GPU, NVIDIA Drivers, cudnn (PyTorch pre-built wheels)To install PyTorch with TorchVision, please refer to the installation instructions on their web pagehere.The tritonclient package wheels are not hosted on the public PyPI server. We need to add the address of NVIDA's private python package index to the environment. You can complete these steps and install the tritonclient package by running the following.# If you cloned this repo, you can just uncomment and run the one line below#sh ./scripts/install_triton_client.pipinstallnvidia-pyindex
pipinstalltritonclient[all]Install via. pipOnce the above requirements are set, you can install fastnn with the command below:pipinstallfastnnIf you are comfortable with the latest default stable releases of PyTorch you can skip step 1 in the pre-requisites and runpip install fastnn[torch]instead.Install from source with Poetry for developmentYou will need to install poetry by referring to the installation instructions on their web pagehere.After cloning the repository, just runpoetry installto install with the .lock file.Activate the virtual environment with the command below:poetryshellDockerOfficial FastNN images are hosted onDocker Hub.Select FastNN package and image versions by referencing thedocumentation. Development and runtime environments are available.Jupyter lab and notebook servers are accessible with notebook examples and terminal accesslocalhost:8888with every image.Run the latest FastNN image by running below:dockerrun--gpusall--rm-it-p8888:8888aychang/fastnn:latestRun images with specific configurations as can see in the example command below:dockerrun--gpusall--rm-it-p8888:8888aychang/fastnn:0.1.0-cuda11.0-runtime-ubuntu18.04-py3.7Triton Inference Server: Local or Kubernetes ClusterLocal DeploymentRequirements:Docker 19.03+GPU Inference Requirements:NVIDIA CUDA-Compatible GPUNVIDIA Container Toolkit (nvidia-docker)Local deployment of the Triton Server uses the EXPLICIT model control mode. Local models must be explicitly specified with the--load-modelargument in./scripts/start_triton_local.shexportMODEL_REPOSITORY=$(pwd)/model_repository
sh./scripts/start_triton_local.shHelm Chart install in Kubernetes ClusterRequirements: kubectl 1.17+, Helm 3+, Kubernetes 1.17+You can currently install the local FastNN helm chart with the following instuctions:cd./k8s
helminstallfastnn.exportMODEL_REPOSITORY=$(pwd)/model_repositoryNote: The current local helm chart installation deploys Triton using the NONE model control mode. All models available in the S3 Model Zoo will be deployed...good luck.
Deployed models and model control mode can be edited in the helm chart deployment configuration file.LicenseThis project is licensed under the terms of the MIT license.