Package: boxmot
BoxMOT: pluggable SOTA tracking modules for segmentation, object detection and pose estimation modelsIntroductionThis repo contains a collections of pluggable state-of-the-art multi-object trackers for segmentation, object detection and pose estimation models. For the methods using appearance description, both heavy (CLIPReID) and lightweight state-of-the-art ReID models (LightMBN,OSNetand more) are available for automatic download. We provide examples on how to use this package together with popular object detection models such as:Yolov8,Yolo-NASandYOLOX.TrackerHOTA↑MOTA↑IDF1↑BoTSORT77.878.988.9DeepOCSORT77.478.489.0OCSORT77.478.489.0HybridSORT77.377.988.8ByteTrack75.674.686.0StrongSORTNOTES: performed on the 10 first frames of each MOT17 sequence. The detector used is ByteTrack's YoloXm, trained on: CrowdHuman, MOT17, Cityperson and ETHZ. Each tracker is configured with its original parameters found in their respective official repository.Tutorials* [Yolov8 training (link to external repository)](https://docs.ultralytics.com/modes/train/) 
* [Deep appearance descriptor training (link to external repository)](https://kaiyangzhou.github.io/deep-person-reid/user_guide.html) 
* [ReID model export to ONNX, OpenVINO, TensorRT and TorchScript](https://github.com/mikel-brostrom/yolo_tracking/wiki/ReID-multi-framework-model-export) 
* [Evaluation on custom tracking dataset](https://github.com/mikel-brostrom/yolo_tracking/wiki/How-to-evaluate-on-custom-tracking-dataset) 
* [ReID inference acceleration with Nebullvm](https://colab.research.google.com/drive/1APUZ1ijCiQFBR9xD0gUvFUOC8yOJIvHm?usp=sharing)ExperimentsIn inverse chronological order:Evaluation of the params evolved for first half of MOT17 on the complete MOT17Segmentation model vs object detetion model on MOT metricsEffect of masking objects before feature extractionconf-thres vs HOTA, MOTA and IDF1Effect of KF updates ahead for tracks with no associations on MOT17Effect of full images vs 1280 input to StrongSORT on MOT17Effect of different OSNet architectures on MOT16Yolov5 StrongSORT vs BoTSORT vs OCSORTYolov5BoTSORTbranch:https://github.com/mikel-brostrom/Yolov5_StrongSORT_OSNet/tree/botsortYolov5 StrongSORT OSNet vs other trackers MOT17StrongSORT MOT16 ablation studyYolov5 StrongSORT OSNet vs other trackers MOT16 (deprecated)NewsCentroid-based cost function added to OCSORT and DeepOCSORT (suitable for: small and/or high speed objects and low FPS videos) (Januari 2024)Custom Ultralytics pacakge updated from 8.0.124 to 8.0.224 (December 2023)HybridSORT available (August 2023)SOTA CLIP-ReID people and vehicle models available (August 2023)Why BOXMOT?Today's multi-object tracking options are heavily dependant on the computation capabilities of the underlaying hardware. BOXMOT provides a great variety of setup options that meet different hardware limitations: CPU only, low memory GPUs... Everything is designed with simplicity and flexibility in mind. If you don't get good tracking results on your custom dataset with the out-of-the-box tracker configurations, use theexamples/evolve.pyscript for tracker hyperparameter tuning.InstallationStart withPython>=3.8environment.If you want to run the YOLOv8, YOLO-NAS or YOLOX examples:git clone https://github.com/mikel-brostrom/yolo_tracking.git
cd yolo_tracking
pip install -v -e .but if you only want to import the tracking modules you can simply:pip install boxmotYOLOv8 | YOLO-NAS | YOLOX examplesTrackingYolo models$pythonexamples/track.py--yolo-modelyolov8n# bboxes onlypythonexamples/track.py--yolo-modelyolo_nas_s# bboxes onlypythonexamples/track.py--yolo-modelyolox_n# bboxes onlyyolov8n-seg# bboxes + segmentation masksyolov8n-pose# bboxes + pose estimationTracking methods$pythonexamples/track.py--tracking-methoddeepocsortstrongsortocsortbytetrackbotsortTracking sourcesTracking can be run on most video formats$pythonexamples/track.py--source0# webcamimg.jpg# imagevid.mp4# videopath/# directorypath/*.jpg# glob'https://youtu.be/Zgi9g1ksQHc'# YouTube'rtsp://example.com/media.mp4'# RTSP, RTMP, HTTP streamSelect ReID modelSome tracking methods combine appearance description and motion in the process of tracking. For those which use appearance, you can choose a ReID model based on your needs from thisReID model zoo. These model can be further optimized for you needs by thereid_export.pyscript$pythonexamples/track.py--source0--reid-modellmbn_n_cuhk03_d.pt# lightweightosnet_x0_25_market1501.ptmobilenetv2_x1_4_msmt17.engineresnet50_msmt17.onnxosnet_x1_0_msmt17.ptclip_market1501.pt# heavyclip_vehicleid.pt...Filter tracked classesBy default the tracker tracks all MS COCO classes.If you want to track a subset of the classes that you model predicts, add their corresponding index after the classes flag,pythonexamples/track.py--source0--yolo-modelyolov8s.pt--classes1617# COCO yolov8 model. Track cats and dogs, onlyHereis a list of all the possible objects that a Yolov8 model trained on MS COCO can detect. Notice that the indexing for the classes in this repo starts at zeroMOT compliant resultsCan be saved to your experiment folderruns/track/exp*/bypythonexamples/track.py--source...--save-motEvaluationEvaluate a combination of detector, tracking method and ReID model on standard MOT dataset or you custom one by$python3examples/val.py--yolo-modelyolo_nas_s.pt--reid-modelosnetx1_0_dukemtcereid.pt--tracking-methoddeepocsort--benchmarkMOT16--yolo-modelyolox_n.pt--reid-modelosnet_ain_x1_0_msmt17.pt--tracking-methodocsort--benchmarkMOT17--yolo-modelyolov8s.pt--reid-modellmbn_n_market.pt--tracking-methodstrongsort--benchmark<your-custom-dataset>EvolutionWe use a fast and elitist multiobjective genetic algorithm for tracker hyperparameter tuning. By default the objectives are: HOTA, MOTA, IDF1. Run it by$pythonexamples/evolve.py--tracking-methodstrongsort--benchmarkMOT17--n-trials100# tune strongsort for MOT17--tracking-methodocsort--benchmark<your-custom-dataset>--objectiveHOTA# tune ocsort for maximizing HOTA on your custom tracking datasetThe set of hyperparameters leading to the best HOTA result are written to the tracker's config file.Custom object detection model tracking exampleMinimalisticimportcv2importnumpyasnpfrompathlibimportPathfromboxmotimportDeepOCSORTtracker=DeepOCSORT(model_weights=Path('osnet_x0_25_msmt17.pt'),# which ReID model to usedevice='cuda:0',fp16=False,)vid=cv2.VideoCapture(0)whileTrue:ret,im=vid.read()# substitute by your object detector, output has to be N X (x, y, x, y, conf, cls)dets=np.array([[144,212,578,480,0.82,0],[425,281,576,472,0.56,65]])tracks=tracker.update(dets,im)# --> (x, y, x, y, id, conf, cls, ind)Completeimportcv2importnumpyasnpfrompathlibimportPathfromboxmotimportDeepOCSORTtracker=DeepOCSORT(model_weights=Path('osnet_x0_25_msmt17.pt'),# which ReID model to usedevice='cuda:0',fp16=True,)vid=cv2.VideoCapture(0)color=(0,0,255)# BGRthickness=2fontscale=0.5whileTrue:ret,im=vid.read()# substitute by your object detector, input to tracker has to be N X (x, y, x, y, conf, cls)dets=np.array([[144,212,578,480,0.82,0],[425,281,576,472,0.56,65]])tracks=tracker.update(dets,im)# --> (x, y, x, y, id, conf, cls, ind)xyxys=tracks[:,0:4].astype('int')# float64 to intids=tracks[:,4].astype('int')# float64 to intconfs=tracks[:,5]clss=tracks[:,6].astype('int')# float64 to intinds=tracks[:,7].astype('int')# float64 to int# in case you have segmentations or poses alongside with your detections you can use# the ind variable in order to identify which track is associated to each seg or pose by:# segs = segs[inds]# poses = poses[inds]# you can then zip them together: zip(tracks, poses)# print bboxes with their associated id, cls and confiftracks.shape[0]!=0:forxyxy,id,conf,clsinzip(xyxys,ids,confs,clss):im=cv2.rectangle(im,(xyxy[0],xyxy[1]),(xyxy[2],xyxy[3]),color,thickness)cv2.putText(im,f'id:{id}, conf:{conf}, c:{cls}',(xyxy[0],xyxy[1]-10),cv2.FONT_HERSHEY_SIMPLEX,fontscale,color,thickness)# show image with bboxes, ids, classes and confidencescv2.imshow('frame',im)# break on pressing qifcv2.waitKey(1)&0xFF==ord('q'):breakvid.release()cv2.destroyAllWindows()Tiled inferencefromsahiimportAutoDetectionModelfromsahi.predictimportget_sliced_predictionimportcv2importnumpyasnpfrompathlibimportPathfromboxmotimportDeepOCSORTtracker=DeepOCSORT(model_weights=Path('osnet_x0_25_msmt17.pt'),# which ReID model to usedevice='cpu',fp16=False,)detection_model=AutoDetectionModel.from_pretrained(model_type='yolov8',model_path='yolov8n.pt',confidence_threshold=0.5,device="cpu",# or 'cuda:0')vid=cv2.VideoCapture(0)color=(0,0,255)# BGRthickness=2fontscale=0.5whileTrue:ret,im=vid.read()# get sliced predictionsresult=get_sliced_prediction(im,detection_model,slice_height=256,slice_width=256,overlap_height_ratio=0.2,overlap_width_ratio=0.2)num_predictions=len(result.object_prediction_list)dets=np.zeros([num_predictions,6],dtype=np.float32)forind,object_predictioninenumerate(result.object_prediction_list):dets[ind,:4]=np.array(object_prediction.bbox.to_xyxy(),dtype=np.float32)dets[ind,4]=object_prediction.score.valuedets[ind,5]=object_prediction.category.idtracks=tracker.update(dets,im)# --> (x, y, x, y, id, conf, cls, ind)iftracks.shape[0]!=0:xyxys=tracks[:,0:4].astype('int')# float64 to intids=tracks[:,4].astype('int')# float64 to intconfs=tracks[:,5].round(decimals=2)clss=tracks[:,6].astype('int')# float64 to intinds=tracks[:,7].astype('int')# float64 to int# print bboxes with their associated id, cls and confforxyxy,id,conf,clsinzip(xyxys,ids,confs,clss):im=cv2.rectangle(im,(xyxy[0],xyxy[1]),(xyxy[2],xyxy[3]),color,thickness)cv2.putText(im,f'id:{id}, conf:{conf}, c:{cls}',(xyxy[0],xyxy[1]-10),cv2.FONT_HERSHEY_SIMPLEX,fontscale,color,thickness)# show image with bboxes, ids, classes and confidencescv2.imshow('frame',im)# break on pressing qifcv2.waitKey(1)&0xFF==ord('q'):breakvid.release()cv2.destroyAllWindows()ContributorsContactFor Yolo tracking bugs and feature requests please visitGitHub Issues.
For business inquiries or professional support requests please send an email to:yolov5.deepsort.pytorch@gmail.com