Package: juxtapose
🍿 IntroJuxtapose is a 2D multi person pose detection, tracking, and estimation inference toolbox for sports + kinematics analysis. VisitDocs.🍄 OverviewCode mostly adopted from four repos ->ultralytics,mmdeploy,mmdetection,mmpose.Supported Detectors:rtmdet-s,rtmdet-m,rtmdet-l,groundingdino,yolov8Supported Pose Estimators:rtmpose-s,rtmpose-m,rtmpose-lSupported Trackers:bytetrack,botsort🥒 Updates2024/01/11Added Nextra docs + deployed to Vercel atsdk.juxt.space.2024/01/07Reduce dependencies by removing MMCV, MMDet, MMPose SDK, run fully on ONNX.2023/11/01Added juxtapose to PYPI repository so that we can install it usingpip install juxtapose.2023/08/25Added customregion of interests (ROI) drawing toolsthat enables multi ROIs filtering while performing pose estimation/tracking. Seeusage below.2023/08/15AddedGroundingDino&YOLOv8object detector.2023/08/09Added keypoints streaming to csv file using csv module.2023/07/31AddedByteTrackandBotSORT. Completed engineering effort for top down inferences in any sources. Seesupported sources below.2023/06/15ConvertedRTMDET (s/m/l)andRTMPOSE (s/m/l)to ONNX usingMMDeploy.👉 Getting StartedInstall Using PIPpip install juxtaposeNote: If you faced any issues, kindly review thisgithub issue🧀 Local Developmentgitclonehttps://github.com/ziqinyeow/juxtapose
pipinstall.🤩 Feel The Magic🌄 Basic UsagefromjuxtaposeimportRTM# Init a rtm model (including rtmdet, rtmpose, tracker)model=RTM(det="rtmdet-m",# see type hintingpose="rtmpose-m",# see type hintingtracker="bytetrack",# see type hintingdevice="cpu",# see type hinting)# Inference with directory (all the images and videos in the dir will get inference sequentially)model("data")# Inference with imagemodel("data/football.jpeg",verbose=False)# verbose -> disable terminal printing# Inference with videomodel("data/bike.mp4")# Inference with the YouTube Sourcemodel("https://www.youtube.com/watch?v=1vYvTbDJuFs&ab_channel=PeterGrant",save=True)🎨 Select Region of Interests (ROIs)It will first prompt the user to draw the ROIs, pressrto remove the existing ROI drawn.
After drawing, pressSPACEorENTERorqto accept the ROI drawn. The model will filter
out the bounding boxes based on the ROIs.😁 Note: PressSPACEagain to redraw the bounding boxes. See custom implementation withcv2here.fromjuxtaposeimportRTMmodel=RTM(det="groundingdino",pose="rtmpose-l",tracker="none")model("data/bike.mp4",roi="rect")# rectangle roi# 1. Draw ROI first# 2. Press r or R to reset ROI# 3. Press SPACE or Enter or q or Q to continue with the ROI🚴‍♂️ Accessing result for each frame: More Flexibility# Adding custom plotimportcv2fromjuxtaposeimportRTM,Annotatormodel=RTM()annotator=Annotator(thickness=3,font_color=(128,128,128))# see rtm.utils.plotting# set show to true -> cv2.imshow the frame (you can use cv2 to plot anything in the frame)# set plot to false -> if you want to ignore default plot -> see rtm.rtm (line `if plot:`)forresultinmodel("data/bike.mp4",show=True,plot=False,stream=True):# do what ever you want with the dataim,bboxes,kpts=result.im,result.bboxes,result.kpts# e.g custom plot anything using cv2 APIcv2.putText(im,"custom text",(100,100),cv2.FONT_HERSHEY_SIMPLEX,1.0,(128,128,128))# use the annotator class -> see rtm.utils.plottingannotator.draw_bboxes(im,bboxes,labels=[f"children_{i}"foriinrange(len(bboxes))])annotator.draw_kpts(im,kpts,thickness=4)annotator.draw_skeletons(im,kpts)⚽️ Custom Forward Pass: Full Flexibility# Custom model forward passimportcv2importtorchfromjuxtaposeimportRTMDet,RTMPose,Annotatorframe=cv2.imread("data/football.jpeg")device="cuda"iftorch.cuda.is_available()else"cpu"# s, m, lrtmdet=RTMDet("l",device=device)rtmpose=RTMPose("l",device=device)annotator=Annotator()bboxes,scores,labels=rtmdet(frame)# [[x1, y1, x2, y2], ...], [], []kpts=rtmpose(frame,bboxes=bboxes)# shape: (number of human, 17, 2)annotator.draw_bboxes(frame,bboxes,labels=[f"person_{i}"foriinrange(len(bboxes))])annotator.draw_kpts(frame,kpts,thickness=4)annotator.draw_skeletons(frame,kpts)cv2.imshow("frame",frame)cv2.waitKey(0)cv2.destroyAllWindows()Supported SourcesAdopted from ultralytics repository -> seehttps://docs.ultralytics.com/modes/predict/SourceArgumentTypeNotesimage'image.jpg'str or PathSingle image file.URL'https://ultralytics.com/images/bus.jpg'strURL to an image.screenshot'screen'strCapture a screenshot.PILImage.open('im.jpg')PIL.ImageHWC format with RGB channels.OpenCVcv2.imread('im.jpg')np.ndarray of uint8 (0-255)HWC format with BGR channels.numpynp.zeros((640,1280,3))np.ndarray of uint8 (0-255)HWC format with BGR channels.torchtorch.zeros(16,3,320,640)torch.Tensor of float32 (0.0-1.0)BCHW format with RGB channels.CSV'sources.csv'str or PathCSV file containing paths to images, videos, or directories.video'video.mp4'str or PathVideo file in formats like MP4, AVI, etc.directory'path/'str or PathPath to a directory containing images or videos.glob'path/*.jpg'strGlob pattern to match multiple files. Use the * character as a wildcard.YouTube'https://youtu.be/Zgi9g1ksQHc'strURL to a YouTube video.stream'rtsp://example.com/media.mp4'strURL for streaming protocols such as RTSP, RTMP, or an IP address.