Package: inference-gpu
🎬 pip install inferenceRoboflowInference is the easiest way to use and deploy computer vision models.
Inference supports running object detection, classification, instance segmentation, and even foundation models (like CLIP and SAM).
You cantrain and deploy your own custom modelor use one of the 50,000+fine-tuned models shared by the community.There are three primaryinferenceinterfaces:A Python-native package (pip install inference)A self-hosted inference server (inference server start)Afully-managed, auto-scaling API.You can run Inference on an edge device like an NVIDIA Jetson, or on cloud computing platforms like AWS, GCP, and Azure.🏃 Getting StartedGet up and running withinferenceon your local machine in 3 minutes.pipinstallinference# or inference-gpu if you have CUDASetupyour Roboflow Private API Keyby exporting aROBOFLOW_API_KEYenvironment variable or
adding it to a.envfile.exportROBOFLOW_API_KEY=your_key_hereRunan open-source Rock, Paper, Scissors modelon your webcam stream:importinferenceinference.Stream(source="webcam",# or rtsp stream or camera idmodel="rock-paper-scissors-sxsw/11",# from Universeon_prediction=lambdapredictions,image:(print(predictions)# now hold up your hand: 🪨 📄 ✂️))[!NOTE]Currently, the stream interface only supports object detectionNow let's extend the example to useSupervisionto visualize the predictions and display them on screen with OpenCV:importcv2importinferenceimportsupervisionassvannotator=sv.BoxAnnotator()inference.Stream(source="webcam",# or rtsp stream or camera idmodel="rock-paper-scissors-sxsw/11",# from Universeoutput_channel_order="BGR",use_main_thread=True,# for opencv displayon_prediction=lambdapredictions,image:(print(predictions),# now hold up your hand: 🪨 📄 ✂️cv2.imshow("Prediction",annotator.annotate(scene=image,detections=sv.Detections.from_inference(predictions))),cv2.waitKey(1)))👩‍🏫 More ExamplesThe/examplesdirectory contains code samples for working with and extendinginferenceincluding using foundation models like CLIP, HTTP and UDP clients, and an insights dashboard, along with community examples (PRs welcome)!🎥 Inference in actionCheck out Inference running on a video of a football game:https://github.com/roboflow/inference/assets/37276661/121ab5f4-5970-4e78-8052-4b40f2eec173💻 Why Inference?Inference provides a scalable method through which you can manage inferences for your vision projects.Inference is composed of:Thousands ofpre-trained community modelsthat you can use as a starting point.Foundation models like CLIP, SAM, and OCR.A tight integration withSupervision.An HTTP server, so you don’t have to reimplement things like image processing and prediction visualization on every project and you can scale your GPU infrastructure independently of your application code, and access your model from whatever language your app is written in.Standardized APIs for computer vision tasks, so switching out the model weights and architecture can be done independently of your application code.A model registry, so your code can be independent from your model weights & you don't have to re-build and re-deploy every time you want to iterate on your model weights.Active Learning integrations, so you can collect more images of edge cases to improve your dataset & model the more it sees in the wild.Seamless interoperability withRoboflowfor creating datasets, training & deploying custom models.And more!📌 Use the Inference ServerYou can learn more about Roboflow Inference Docker Image build, pull and run in ourdocumentation.Run on x86 CPU:dockerrun-it--net=hostroboflow/roboflow-inference-server-cpu:latestRun on NVIDIA GPU:dockerrun-it--network=host--gpus=allroboflow/roboflow-inference-server-gpu:latest👉 more docker run optionsRun on arm64 CPU:dockerrun-p9001:9001roboflow/roboflow-inference-server-arm-cpu:latestRun on NVIDIA Jetson with JetPack4.x:dockerrun--privileged--net=host--runtime=nvidiaroboflow/roboflow-inference-server-jetson:latestRun on NVIDIA Jetson with JetPack5.x:dockerrun--privileged--net=host--runtime=nvidiaroboflow/roboflow-inference-server-jetson-5.1.1:latestExtras:Some functionality requires extra dependencies. These can be installed by specifying the desired extras during installation of Roboflow Inference.extradescriptionclipAbility to use the coreCLIPmodel (by OpenAI)gazeAbility to use the coreGazemodelhttpAbility to run the http interfacesamAbility to run the coreSegment Anythingmodel (by Meta AI)Note:Both CLIP and Segment Anything require pytorch to run. These are included in their respective dependencies however pytorch installs can be highly environment dependent. See theofficial pytorch install pagefor instructions specific to your enviornment.Example install with CLIP dependencies:pipinstall"inference[clip]"Inference ClientTo consume predictions from inference server in Python you can
use theinference-sdkpackage.pipinstallinference-sdkfrominference_sdkimportInferenceHTTPClientimage_url="https://media.roboflow.com/inference/soccer.jpg"# Replace ROBOFLOW_API_KEY with your Roboflow API Keyclient=InferenceHTTPClient(api_url="http://localhost:9001",# or https://detect.roboflow.com for Hosted APIapi_key="ROBOFLOW_API_KEY")withclient.use_model("soccer-players-5fuqs/1"):predictions=client.infer(image_url)print(predictions)Visit ourdocumentationto discover capabilities ofinference-clientslibrary.Single Image InferenceAfter installinginferencevia pip, you can run a simple inference
on a single image (vs the video stream example above) by instantiating
amodeland using theinfermethod (don't forget to setup yourROBOFLOW_API_KEYenvironment variable or.envfile):frominference.models.utilsimportget_modelmodel=get_model(model_id="soccer-players-5fuqs/1")# you can also infer on local images by passing a file path,# a PIL image, or a numpy arrayresults=model.infer(image="https://media.roboflow.com/inference/soccer.jpg",confidence=0.5,iou_threshold=0.5)print(results)Getting CLIP EmbeddingsYou can run inference withOpenAI's CLIP modelusing:frominference.modelsimportClipimage_url="https://media.roboflow.com/inference/soccer.jpg"model=Clip()embeddings=model.embed_image(image_url)print(embeddings)Using SAMYou can run inference withMeta's Segment Anything modelusing:frominference.modelsimportSegmentAnythingimage_url="https://media.roboflow.com/inference/soccer.jpg"model=SegmentAnything()embeddings=model.embed_image(image_url)print(embeddings)🏗️ inference ProcessTo standardize the inference process throughout all our models, Roboflow Inference has a structure for processing inference requests. The specifics can be found on each model's respective page, but overall it works like this for most models:✅ Supported ModelsLoad from RoboflowYou can use models hosted on Roboflow with the following architectures through Inference:YOLOv5 Object DetectionYOLOv5 Instance SegmentationYOLOv8 Object DetectionYOLOv8 ClassificationYOLOv8 SegmentationYOLACT SegmentationViT ClassificationCore ModelsCore Models are foundation models and models that have not been fine-tuned on a specific dataset.The following core models are supported:CLIPL2CS (Gaze Detection)Segment Anything (SAM)📝 LicenseThe Roboflow Inference code is distributed under anApache 2.0 license. The models supported by Roboflow Inference have their own licenses. View the licenses for supported models below.modellicenseinference/models/clipMITinference/models/gazeMIT,Apache 2.0inference/models/samApache 2.0inference/models/vitApache 2.0inference/models/yolactMITinference/models/yolov5AGPL-3.0inference/models/yolov7GPL-3.0inference/models/yolov8AGPL-3.0Inference CLIWe've created a CLI tool with useful commands to make theinferenceusage easier. Check outdocs.🚀 EnterpriseWith a Roboflow Inference Enterprise License, you can access additional Inference features, including:Server cluster deploymentDevice managementActive learningYOLOv5 and YOLOv8 commercial licenseTo learn more,contact the Roboflow team.📚 documentationVisit ourdocumentationfor usage examples and reference for Roboflow Inference.🏆 contributionWe would love your input to improve Roboflow Inference! Please see ourcontributing guideto get started. Thank you to all of our contributors! 🙏💻 explore more Roboflow open source projectsProjectDescriptionsupervisionGeneral-purpose utilities for use in computer vision projects, from predictions filtering and display to object tracking to model evaluation.AutodistillAutomatically label images for use in training computer vision models.Inference(this project)An easy-to-use, production-ready inference server for computer vision supporting deployment of many popular model architectures and fine-tuned models.NotebooksTutorials for computer vision tasks, from training state-of-the-art models to tracking objects to counting objects in a zone.CollectAutomated, intelligent data collection powered by CLIP.