Package: tf-bodypix
TensorFlow BodyPix (TF BodyPix)A Python implementation ofbody-pix.Goals of this project is:Python library, making it easy to integrate the BodyPix modelCLI with limited functionality, mostly for demonstration purposePrerequisitsPython 3.7+InstallInstall with all dependencies:pipinstalltf-bodypix[all]Install with minimal or no dependencies:pipinstalltf-bodypixExtras are provided to make it easier to provide or exclude dependencies
when using this project as a library:extra namedescriptiontfTensorFlow(required). But you may use your own build.tfjsTensorFlow JS Model support, usingtfjs-graph-convertertflitetflite-runtimeimageImage loading viaPillow, required by the CLI.videoVideo support viaOpenCVwebcamWebcam support viaOpenCVandpyfakewebcamallAll of the libraries (excepttflite-runtime)Python APIfrompathlibimportPathimporttensorflowastffromtf_bodypix.apiimportdownload_model,load_model,BodyPixModelPaths# setup input and output pathsoutput_path=Path('./data/example-output')output_path.mkdir(parents=True,exist_ok=True)input_url=('https://www.dropbox.com/s/7tsaqgdp149d8aj/serious-black-businesswoman-sitting-at-desk-in-office-5669603.jpg?dl=1')local_input_path=tf.keras.utils.get_file(origin=input_url)# load model (once)bodypix_model=load_model(download_model(BodyPixModelPaths.MOBILENET_FLOAT_50_STRIDE_16))# get prediction resultimage=tf.keras.preprocessing.image.load_img(local_input_path)image_array=tf.keras.preprocessing.image.img_to_array(image)result=bodypix_model.predict_single(image_array)# simple maskmask=result.get_mask(threshold=0.75)tf.keras.preprocessing.image.save_img(f'{output_path}/output-mask.jpg',mask)# colored mask (separate colour for each body part)colored_mask=result.get_colored_part_mask(mask)tf.keras.preprocessing.image.save_img(f'{output_path}/output-colored-mask.jpg',colored_mask)# posesfromtf_bodypix.drawimportdraw_poses# utility function using OpenCVposes=result.get_poses()image_with_poses=draw_poses(image_array.copy(),# create a copy to ensure we are not modifing the source imageposes,keypoints_color=(255,100,100),skeleton_color=(100,100,255))tf.keras.preprocessing.image.save_img(f'{output_path}/output-poses.jpg',image_with_poses)CLICLI Helppython-mtf_bodypix--helporpython-mtf_bodypix<subcommand>--helpList Available Modelspython-mtf_bodypixlist-modelsThe result will be a list of all of thebodypixTensorFlow JS models available in thetfjs-models bucket.Those URLs can be passed as the--model-patharguments below, or to thedownload_modelmethod of the Python API.The CLI will download and cache the model from the provided path. If no--model-pathis provided, it will use a default model (mobilenet).To list TensorFlow Lite models instead:python-mtf_bodypixlist-tflite-modelsInputs and OutputsMost commands will work with inputs (source) and outputs.The source path can be specified via the--sourceparameter.The following inputs are supported:typedescriptionimageStatic image (e.g..png)videoVideo (e.g..mp4)webcamLinux Webcam (/dev/videoNorwebcam:0)If the source path points to an external file (e.g.https://), then it will be downloaded and locally cached.The output path can be specified via--output, unless--show-outputis used.The following outpus are supported:typedescriptionimage_writerWrite to a static image (e.g..png)v4l2Linux Virtual Webcam (/dev/videoN)windowDisplay a window (by using--show-output)Example commandsCreating a simple body maskpython-mtf_bodypix\draw-mask\--source\"https://www.dropbox.com/s/7tsaqgdp149d8aj/serious-black-businesswoman-sitting-at-desk-in-office-5669603.jpg?dl=1"\--show-output\--threshold=0.75Image Source:Serious black businesswoman sitting at desk in officeAdd the mask over the original image using--mask-alphapython-mtf_bodypix\draw-mask\--source\"https://www.dropbox.com/s/7tsaqgdp149d8aj/serious-black-businesswoman-sitting-at-desk-in-office-5669603.jpg?dl=1"\--show-output\--threshold=0.75\--mask-alpha=0.5Image Source:Serious black businesswoman sitting at desk in officeColorize the body mask depending on the body partpython-mtf_bodypix\draw-mask\--source\"https://www.dropbox.com/s/7tsaqgdp149d8aj/serious-black-businesswoman-sitting-at-desk-in-office-5669603.jpg?dl=1"\--show-output\--threshold=0.75\--mask-alpha=0.5\--coloredImage Source:Serious black businesswoman sitting at desk in officeAdditionally select the body partspython-mtf_bodypix\draw-mask\--source\"https://www.dropbox.com/s/7tsaqgdp149d8aj/serious-black-businesswoman-sitting-at-desk-in-office-5669603.jpg?dl=1"\--show-output\--threshold=0.75\--mask-alpha=0.5\--partsleft_faceright_face\--coloredImage Source:Serious black businesswoman sitting at desk in officeAdd mask overlay to a videopython-mtf_bodypix\draw-mask\--source\"https://www.dropbox.com/s/s7jga3f0dreavlb/video-of-a-man-laughing-and-happy-1608393-360p.mp4?dl=1"\--show-output\--threshold=0.75\--mask-alpha=0.5\--coloredVideo Source:Video Of A Man Laughing And HappyAdd pose overlay to a videopython-mtf_bodypix\draw-pose\--source\"https://www.dropbox.com/s/pv5v8dkpj5wung7/an-old-man-doing-a-tai-chi-exercise-2882799-360p.mp4?dl=1"\--show-output\--threshold=0.75Blur background of a videopython-mtf_bodypix\blur-background\--source\"https://www.dropbox.com/s/s7jga3f0dreavlb/video-of-a-man-laughing-and-happy-1608393-360p.mp4?dl=1"\--show-output\--threshold=0.75\--mask-blur=5\--background-blur=20Video Source:Video Of A Man Laughing And HappyReplace the background of a videopython-mtf_bodypix\replace-background\--source\"https://www.dropbox.com/s/s7jga3f0dreavlb/video-of-a-man-laughing-and-happy-1608393-360p.mp4?dl=1"\--background\"https://www.dropbox.com/s/b22ss59j6pp83zy/brown-landscape-under-grey-sky-3244513.jpg?dl=1"\--show-output\--threshold=0.75\--mask-blur=5Video Source:Video Of A Man Laughing And HappyBackground:Brown Landscape Under Grey SkyCapture Webcam and adding mask overlaypython-mtf_bodypix\draw-mask\--sourcewebcam:0\--show-output\--threshold=0.75\--mask-alpha=0.5\--coloredCapture Webcam and adding mask overlay, writing to v4l2loopback device(replace/dev/videoNwith the actual virtual video device)python-mtf_bodypix\draw-mask\--sourcewebcam:0\--output/dev/videoN\--threshold=0.75\--mask-alpha=0.5\--coloredCapture Webcam and blur background, writing to v4l2loopback device(replace/dev/videoNwith the actual virtual video device)python-mtf_bodypix\blur-background\--sourcewebcam:0\--background-blur20\--output/dev/videoN\--threshold=0.75Capture Webcam and replace background, writing to v4l2loopback device(replace/dev/videoNwith the actual virtual video device)python-mtf_bodypix\replace-background\--sourcewebcam:0\--background\"https://www.dropbox.com/s/b22ss59j6pp83zy/brown-landscape-under-grey-sky-3244513.jpg?dl=1"\--threshold=0.75\--output/dev/videoNBackground:Brown Landscape Under Grey SkyTensorFlow Lite Model support (experimental)The model path may also point to a TensorFlow Lite model (.tfliteextension). Whether that actually improves performance may depend on the platform and available hardware.You could convert one of the available TensorFlow JS models to TensorFlow Lite using the following command:python-mtf_bodypix\convert-to-tflite\--model-path\"https://storage.googleapis.com/tfjs-models/savedmodel/bodypix/mobilenet/float/075/model-stride16.json"\--optimize\--quantization-type=float16\--output-model-file"./mobilenet-float-multiplier-075-stride16-float16.tflite"The above command is provided for convenience.
You may use alternative methods depending on your preference and requirements.Relevant links:TensorFlow Lite converterTF Lite post_training_quantizationTF GitHub #40183.TensorFlow Lite Runtime support (experimental)This project can also be used withtflite-runtimeinstead of full TensorFlow (e.g. by using thetfliteextra).
However,TensorFlow Lite converterwould require full TensorFlow.
In order to avoid it, one needs to use a TensorFlow Lite model (see previous section).Docker UsageYou could also use the Docker image if you prefer.
The entrypoint will by default delegate to the CLI, except forpythonorbashcommands.# pull latest image (you may also use tags)dockerpullde4code/tf-bodypix# mount real and virtual webcam devices on linuxdockerrun--rm\--device/dev/video0\--device/dev/video2\de4code/tf-bodypix\blur-background\--source/dev/video0\--output/dev/video2\--background-blur20\--threshold=0.75# mount x11 display on linuxdockerrun--rm\--net=host\--volume/tmp/.X11-unix:/tmp/.X11-unix\--volume${HOME}/.Xauthority:/root/.Xauthority\--envDISPLAY\de4code/tf-bodypix\replace-background\--source\"https://www.dropbox.com/s/s7jga3f0dreavlb/video-of-a-man-laughing-and-happy-1608393-360p.mp4?dl=1"\--background\"https://www.dropbox.com/s/b22ss59j6pp83zy/brown-landscape-under-grey-sky-3244513.jpg?dl=1"\--show-output\--threshold=0.75\--mask-blur=5Example MediaHere are a few example media files you could try.Images:Serious black businesswoman sitting at desk in office(Source)Woman Wearing Gray Notch Lapel Suit Jacket(Source)Smiling Woman Standing In Front Of A Colorful Flag(Source)Man and Woman Smiling Inside Building(Source)Two Woman in Black Sits on Chair Near Table(Source)Female barista in beanie and apron resting chin on had(Source)Smiling Woman Holding White Android Smartphone While Sitting Front of Table(Source)Woman Having Coffee and Rice Bowl(Source)Woman Smiling While Holding a Coffee Cup(Source)Videos:Video Of A Man Laughing And Happy(Source)A Group Of People In A Business Meeting(Source)An Old Man Doing A Tai Chi Exercise(Source)Background:Brown Landscape Under Grey Sky(Source)Experimental Downstream ProjectsLayered Visionis an experimental project using thetf-bodypixPython API.AcknowledgementsOriginal TensorFlow JS Implementation of BodyPixLinux-Fake-Background-Webcam, an implementation of theblog postdescribing using the TensorFlow JS implementation with Python via a Socket API.tfjs-to-tffor providing an easy way to convert TensorFlow JS modelsvirtual_webcam_backgroundfor a great pure Python implementation