Package: torch-nos
Website|Docs|Tutorials|Playground|Blog|DiscordNOSis a fast and flexible PyTorch inference server that runs on any cloud or AI HW.🛠️ Key Features👩‍💻Easy-to-use: Built forPyTorchand designed to optimize, serve and auto-scale Pytorch models in production without compromising on developer experience.🥷Multi-modal & Multi-model: Serve multiple foundational AI models (LLMs,Diffusion,Embeddings,Speech-to-TextandObject Detection) simultaneously, in a single server.⚙️HW-aware Runtime:Deploy PyTorch models effortlessly on modern AI accelerators (NVIDIA GPUs, AWS Inferentia2, AMD - coming soon, and even CPUs).☁️Cloud-agnostic Containers:Run on any cloud (AWS, GCP, Azure, Lambda Labs, On-Prem) with our ready-to-use inference server containers.🔥 What's New[Feb 2024]✍️ [blog]Introducing the NOS Inferentia2 (inf2) runtime.[Jan 2024]✍️ [blog]Serving LLMs on a budgetwithSkyServe.[Jan 2024]📚 [docs]NOS x SkyPilot Integrationpage![Jan 2024]✍️ [blog]Getting started with NOS tutorialsis availablehere![Dec 2023]🛝 [repo] We open-sourced theNOS playgroundto help you get started with more examples built on NOS!🚀 QuickstartWe highly recommend that you go to ourquickstart guideto get started. To install the NOS client, you can run the following command:condacreate-nnospython=3.8-y
condaactivatenos
pipinstalltorch-nosOnce the client is installed, you can start the NOS server via the NOSserveCLI. This will automatically detect your local environment, download the docker runtime image and spin up the NOS server:nosserveup--http--logging-levelINFOYou are now ready to run yourfirst inference requestwith NOS! You can run any of the following commands to try things out. You can set the logging level toDEBUGif you want more detailed information from the server.👩‍💻What can NOS do?💬 Chat / LLM Agents (ChatGPT-as-a-Service)NOS provides an OpenAI-compatible server with streaming support so that you can connect your favorite OpenAI-compatible LLM client to talk to NOS.API / UsagegRPC API ⚡fromnos.clientimportClientclient=Client("[::]:50051")model=client.Module("TinyLlama/TinyLlama-1.1B-Chat-v1.0")response=model.chat(message="Tell me a story of 1000 words with emojis",_stream=True)REST APIcurl\-XPOSThttp://localhost:8000/v1/chat/completions\-H"Content-Type: application/json"\-d'{"model": "TinyLlama/TinyLlama-1.1B-Chat-v1.0","messages": [{"role": "user","content": "Tell me a story of 1000 words with emojis"}],"temperature": 0.7,"stream": true}'🏞️ Image Generation (Stable-Diffusion-as-a-Service)Build MidJourney discord bots in seconds.API / UsagegRPC API ⚡fromnos.clientimportClientclient=Client("[::]:50051")sdxl=client.Module("stabilityai/stable-diffusion-xl-base-1-0")image,=sdxl(prompts=["hippo with glasses in a library, cartoon styling"],width=1024,height=1024,num_images=1)REST APIcurl\-XPOSThttp://localhost:8000/v1/infer\-H'Content-Type: application/json'\-d'{"model_id": "stabilityai/stable-diffusion-xl-base-1-0","inputs": {"prompts": ["hippo with glasses in a library, cartoon styling"],"width": 1024, "height": 1024,"num_images": 1}}'🧠 Text & Image Embedding (CLIP-as-a-Service)Buildscalable semantic search of images/videosin minutes.API / UsagegRPC API ⚡fromnos.clientimportClientclient=Client("[::]:50051")clip=client.Module("openai/clip-vit-base-patch32")txt_vec=clip.encode_text(texts=["fox jumped over the moon"])REST APIcurl\-XPOSThttp://localhost:8000/v1/infer\-H'Content-Type: application/json'\-d'{"model_id": "openai/clip-vit-base-patch32","method": "encode_text","inputs": {"texts": ["fox jumped over the moon"]}}'🎙️ Audio Transcription (Whisper-as-a-Service)Performreal-time audio transcriptionusing Whisper.API / UsagegRPC API ⚡frompathlibimportPathfromnos.clientimportClientclient=Client("[::]:50051")model=client.Module("openai/whisper-small.en")withclient.UploadFile(Path("audio.wav"))asremote_path:response=model(path=remote_path)# {"chunks": ...}REST APIcurl\-XPOSThttp://localhost:8000/v1/infer/file\-H'accept: application/json'\-H'Content-Type: multipart/form-data'\-F'model_id=openai/whisper-small.en'\-F'file=@audio.wav'🧐 Object Detection (YOLOX-as-a-Service)Run classical computer-vision tasks in 2 lines of code.API / UsagegRPC API ⚡frompathlibimportPathfromnos.clientimportClientclient=Client("[::]:50051")model=client.Module("yolox/medium")response=model(images=[Image.open("image.jpg")])REST APIcurl\-XPOSThttp://localhost:8000/v1/infer/file\-H'accept: application/json'\-H'Content-Type: multipart/form-data'\-F'model_id=yolox/medium'\-F'file=@image.jpg'⚒️ Custom modelsWant to run models not supported by NOS? You can easily add your own models following the examples in theNOS Playground.📄 LicenseThis project is licensed under theApache-2.0 License.📡 TelemetryNOS collects anonymous usage data usingSentry. This is used to help us understand how the community is using NOS and to help us prioritize features. You can opt-out of telemetry by settingNOS_TELEMETRY_ENABLED=0.🤝 ContributingWe welcome contributions! Please see ourcontributing guidefor more information.🔗  Quick Links💬 Send us an email atsupport@autonomi.aior join ourDiscordfor help.📣 Follow us onTwitter, andLinkedInto keep up-to-date on our products.<style> .md-typeset h1, .md-content__button { display: none; } </style>