Package: gest
gestHand gestures as an input deviceWhyFor health related reasons, I had to stop using a mouse and a keyboard.Talonallowed me to type with my voice and move the cursor with my eyes. This project was started to complement this setup with hand gestures.Development statusThe project is in an early stage of development. I use it on daily basis, so it should be good enough for some.What is implemented:pinching gesture recognition, in one hand orientationheatmap output, separate for left and right hand, indicating pinched point positiondemo for testing recognition modelsexample script for simulating mouse clicks and scrollingscripts for producing and reviewing training dataBiasThe gesture recognition model was trained on images of my hands, taken with my hardware in my working environment, so it is probably heavily biased.
I hope people who want to use it, but recognition quality prevents them from it, would capture some images of their hands using included tooling and donate it to the project, so that over time it works well for everyone.InstallationUse Python 3.6, 3.7 or 3.8 andin a virtual environmentrunpip install gestIf you clone this repository, you can get the exact versions of required libraries that I am using withPoetrypoetry installWalkthroughDemoFirst check how the included model works for you. Runpython -m gest.demoand see if it recognizes your gestures as here:If you have multiple cameras, you can pick one likepython -m gest.demo --camera 2Camera numbers are not necessarily consecutive.
Two cameras may be accessible as 0 and 2.
This option is supported by other commands as well.Example scriptIn the presentation on top I am runningpython -m gest.examples.two_handed_scroll_and_clickIt only acts if it detects both hands pinching and based on their relative position:double clicks if you cross your handsscrolls up or down if your hands pinch at different heightsleft clicks if your hands (almost) touchright clicks if your hands are on the same height, but not close horizontally (this action is delayed by a fraction of a second to prevent accidental use)Controlling CPU loadFor everyday use, you don't want to dedicate too much resources to gesture recognition. You can control it by settingOMP_NUM_THREADS, as inOMP_NUM_THREADS=2 python -m gest.examples.two_handed_scroll_and_clickTry different values to find balance between responsiveness and CPU load.Custom scriptsThe demo and example scripts serve two additional purposes:
they can be used as templates for custom scripts
and they define the public API for the purpose of semantic versioning.Training data annotationCapturingpython -m gest.annotation.capture --countdown 5 data_directorywill help you create annotated images.
Once you start automatic annotation (pressato start/stop) it will ask you to pinch a given point with your left or right hand, or to not pinch ("background").You will have 5 seconds before the image is captured (the--countdown).You will also see the last annotated image for quick review. It can be deleted withd.Reviewingpython -m gest.annotation.review --time 1 data_directory closed_pinch_leftwill let you review all images annotated as left hand pinch indata_directory, showing you each for 1 second if you start/stop automatic advancing witha. Otherwise you can go to the next/previous image withn/p. Delete incorrectly annotated images withd.You should also reviewclosed_pinch_rightandbackground.Annotation guidelinesIt makes sense to annotate realistic training data that the model performs poorly on, like ifit mistakenly detects a pinch when you pick up the phone,it doesn't detect pinching when you wear a skin colored shirt.If it performs poorly overall, it's good to capture the images in many short sessions, with different lighting, clothes, background, camera angle.The point isn't though to look for tricky cases or stretch the definition of a pinching gesture to include a different hand orientation (eg. with pinching fingers pointing towards the camera).Donating annotated dataContact meb.marcinkowski@leomail.pl