Package: layered-vision
Layered VisionGoals of this project is:A tool to allow the composition of images or videos via a configuration file (e.g. as a virtual webcam).This project is still very much experimental and may change significantly.InstallInstall with all dependencies:pipinstalllayered-vision[all]Install with minimal dependencies:pipinstalllayered-visionExtras are provided to make it easier to provide or exclude dependencies
when using this project as a library:extra namedescriptionbodypixForbodypixfilterwebcamVirtual Webcam support viapyfakewebcamyoutubeYouTube support viapafyandyoutube_dlmediapipeSelfie Segmentation usingMediaPipe.allAll of the librariesVirtual Webcam For LinuxYou do not need to use a webcam to use the project, as you could feed a video file.
But if you do want to use a webcam (currently only supported on Linux), this section provides a bit more information.On Linux,/dev/video0often refers to the true webcam device.You can usev4l2loopbackto create a virtual webcam device. e.g. you could/dev/video2.Most applications looking for a webcam should then be able to use that virtual device.
(Applications might include Chromium, Skype etc.)Once installed, you can create/dev/video2via the following command:modprobev4l2loopbackdevices=1video_nr=2exclusive_caps=1card_label="VirtualCam 1"To create the device after every reboot, you might want to create/etc/modprobe.d/v4l2loopback.conf:options v4l2loopback devices=1 video_nr=2 exclusive_caps=1 card_label="VirtualCam 1"ConfigurationThe configuration format is file isYAML.There are a number ofexample configuration files.LayersEvery configuration file will contain layers. Layers are generally described from top to down.
With the last layer usually being the output layer.The source to the output layer will be the layer above.A very simple configuration file that downloads thenumpylogo and saves it to a file might look like (example-config/save-image.yml):layers:-id:ininput_path:"https://raw.githubusercontent.com/numpy/numpy/v1.20.1/branding/logo/logomark/numpylogoicon.png"-id:outoutput_path:"numpy-logo.png"You could also have two outputs (example-config/two-outputs.yml):layers:-id:ininput_path:"https://raw.githubusercontent.com/numpy/numpy/v1.20.1/branding/logo/logomark/numpylogoicon.png"-id:out1output_path:"data/numpy-logo1.png"-id:out2output_path:"data/numpy-logo2.png"In that case the source layer for bothout1andout2isin.By usingwindowas theoutput_path, the image is displayed in a window (example-config/display-image.yml):layers:-id:ininput_path:"https://raw.githubusercontent.com/numpy/numpy/v1.20.1/branding/logo/logomark/numpylogoicon.png"width:480height:300repeat:true-id:outoutput_path:windowInput Layer (Source)A layer that has aninput_pathproperty.The following inputs are currently supported:type namedescriptionimageStatic image (e.g..png)videoVideo (e.g..mp4)webcamLinux Webcam (/dev/videoN)fillFills a new image with a coloryoutubeYouTube stream (e.g.https://youtu.be/f0cGgOv3l4c,  seeexample config)mssScreen capture usingmss(seeexample config)Theinput_pathmay point to a remote location (as is the case withall examples). In that case it will be downloaded and cached locally.In most cases thetype nameis inferred from theinput_path.
You can also specify the type explicitly via thetypeproperty or by prefixing the path, e.g.:webcam:/dev/video0.Output Layer (Sink)A layer that has anoutput_pathproperty.The following outputs are currently supported:type namedescriptionimage_writerWrite to a static image (e.g..png)v4l2Linux Virtual Webcam (/dev/videoN)windowDisplay a windowwebProvide output as JPEG streamAs is the case with theinput_path, in most cases thetype nameis inferred from theoutput_path.
You can also specify the type explicitly via thetypeproperty or by prefixing the path, e.g.:v4l2:/dev/video2.Web Stream (Experimental)The output may also be provide as a JPEG stream. That way it can be viewed in a browser.The following configuration options are supported:namedefault valuedescriptionhost0.0.0.0Host to listen to,0.0.0.0to listen on any host. This could also be set to say127.0.0.1to prevent the stream from being accessed from another machine.port8280The port to listen to.With the default configuration, openinghttp://localhost:8280will provide a link to the stream.
The stream will contineously provide JPEG frames to the browser (as a single request).Currently only one stream consumer is supported.Filter LayerA layer that has afilterproperty.The following filters are currently supported:namedescriptionbox_blurBlurs the image or channel.bodypixUses thebodypixmodel to mask a person.chroma_keyUses a chroma key (colour) to add a maskcopyCopies the input. Mainly useful as a placeholder layer withbranches.dilateDilates the image or channel. For example to increase the alpha mask after usingerodeerodeErodes the image or channel. That could be useful to remove outliers from an alpha mask.bilateralApplies abilateral filter, usingd,sigma_colorandsigma_spaceparameters.motion_blurAdds a motion blur to the image or channel. That could be used to make an alpha mask move more slowlymp_selfie_segmentationUses theMediaPipe's Selfie Segmentationto mask a person (similar to bodypix).pixelatePixelates the input.fillFills the input or a selected channel with a color / value. e.g. withcolorset toblueinvertInverts the input. e.g.blacktowhitemultiplyMultiplies the input with a constant value. e.g. to adjust thealphachannelwarp_perspectiveWarps the perspective of the input image given a list oftarget_points. e.g. to display it in a corner of the output imageEveryfiltermay have additional properties. Please refer to theexamples(or come back in the future) for more detailed information. In particulardisplay-video-segmentation-replace-background-template.ymlprovides examples of most filters (often disabled by default).Filter: mp_selfie_segmentation (Experimental)MediaPipe's Selfie Segmentationallows background segmentation (similar to bodypix). As it is more optimized, it will usually be faster than using bodypix.The following parameters are supported:namedefault valuedescriptionmodel_selection1The model to use,0or1(please referMediaPipe's Selfie Segmentation documentationfor further details).threshold0.1The threshold for the segmentation mask.cache_model_result_secs0.0The number of seconds to cache the mask for.python-mlayered_visionstart\--config-file\"example-config/display-video-segmentation-replace-background-template.yml"\--set"bodypix.enabled=false"\--set"mp_selfie_segmentation.enabled=true"Branches LayerA layer that has anbranchesproperty.
Eachbranchis required to have alayersproperty.
The input to each set ofbranch layersis the input to thebrancheslayer.
Thebranchesare then combined (added on top of each other).
To makebranchesuseful, at least the lastbranch imageshould have an alpha mask.Error HandlingBy default, any error such as an invalid path or filter parameter,
will result in an exception being thrown, causing the application to exit.To instead display an image, you could define an input layer with the idon_error:layers:-id:on_error# Source: https://pixabay.com/vectors/test-pattern-tv-tv-test-pattern-152459/input_path:"https://www.dropbox.com/s/29ycjg9ubht776y/test-pattern-152459_640.png?dl=1"repeat:true# ...CLICLI Helppython-mlayered_vision--helporpython-mlayered_vision<subcommand>--helpExample Commandpython-mlayered_visionstart--config-file=example-config/display-image.ymlYou could also load the config from a remote location:python-mlayered_visionstart--config-file\"https://raw.githubusercontent.com/de-code/layered-vision/develop/example-config/display-video-chroma-key-replace-background.yml"It is also possible to override config values via command line arguments, e.g.:python-mlayered_visionstart--config-file=example-config/display-image.yml\--setout.output_path=/path/to/output.pngYou could also try replacing the background with a YouTube stream:python-mlayered_visionstart\--config-file\"https://raw.githubusercontent.com/de-code/layered-vision/develop/example-config/webcam-bodypix-replace-background-to-v4l2loopback.yml"\--setbg.input_path="https://youtu.be/f0cGgOv3l4c"\--setbg.fps=30\--setin.input_path="/dev/video0"\--setout.output_path="/dev/video2"Note: you may need to specify the fpsIf a local configuration file was specified, the application will attempt to reload it on change.Docker UsageYou could also use the Docker image if you prefer.
The entrypoint will by default delegate to the CLI, except forpythonorbashcommands.dockerpullde4code/layered-vision

dockerrun--rm\--device/dev/video0\--device/dev/video2\de4code/layered-visionstart\--config-file\"https://raw.githubusercontent.com/de-code/layered-vision/develop/example-config/webcam-bodypix-replace-background-to-v4l2loopback.yml"\--setbg.input_path="https://www.dropbox.com/s/4debg4lrgn5g36l/toy-train-3288425.mp4?dl=1"\--setin.input_path="/dev/video0"\--setout.output_path="/dev/video2"(Background:Toy Train)Acknowledgementsvirtual_webcam_background, a somewhat similar project (more focused on bodypix)OBS Studio, conceptually a source of inspiration. (with UI etc)