Package: hamoco
Hamocohamoco(handy mouse controller) is a python application that allows you to control your mouse from your webcam using various hand gestures. You have a laptop equipped with a webcam? Well, good news, that's all you need to feel like Tom Cruise inMinority Report! Kind of.DemonstrationIn the example below, the hand is used to move the pointer, open a file by double-clicking on it, scroll through it, select a paragraph and cut it. The file is then dragged and dropped into a folder.How does it work?By using the power ofPyAutoGUIto control the mouse,OpenCVto process the video feed, andMediaPipeto track hands,hamocopredicts the nature of a hand pose in real-time thanks to a neural network built withKerasand uses it to perform various kinds of mouse pointer actions.Installation1.FromPyPI:pipinstallhamoco2.From thecode repository:git clone https://github.com/jorisparet/hamoco
cd hamoco
pip install -r requirements.txt
pip install .The installation copies three scripts in the default script folder ofpip:hamoco-runhamoco-datahamoco-trainLinuxThe default folder should be under/home/<user>/.local/bin/. Make sure this location (or the correct one, if different) is included in your$PATHenvironment variable to be able to run the scripts from the console. If not, type the following commandexport PATH=$PATH:/path/to/hamoco/scripts/in the console or add it your.bashrcfile.WindowsThe default folder should be underC:\Users\<user>\AppData\Local\Programs\Python\<python_version>\Scripts\. Make sure this location (or the correct one, if different) is included in your$PATHenvironment variable to be able to run the scripts from the console. If not, type the following commandset PATH=%PATH%;C:\path\to\hamoco\scripts\in the console, or selectEdit the system environment variablesin the search bar, clickEnvironment Variables…, clickPATH, clickEdit...and add the correct path to the scripts.Requirements:PyAutoGUINumPyOpenCVMediaPipeTensorFlowQuick startRunning the scriptshamocois composed of three executable scripts:hamoco-run,hamoco-data, andhamoco-train, that are listed below. Run these scripts directly from the console,e.g.hamoco-run --sensitivity 0.5 --show.hamoco-runhamoco-runis themain application. It activates the webcam and allows to use hand gestures to take control of the mouse pointer. Several basic actions can then be performed, such asleft click,right click,drag and dropandscrolling. Note that it requiresa bit of practicebefore getting comfortable with the controls. Various settings can be adjusted to customize the hand controller to your liking, such as the global sensivitity, parameters for motion smoothing and much more. Typehamoco-run --helpfor more information on the available options.Examples:hamoco-run --sensitivity 0.4 --scrolling_threshold 0.2: adapts the sensitivity and sets a custom threshold value to trigger scrolling motions.hamoco-run --min_cutoff_filter 0.05 --show: sets a custom value for the cutoff frequency used for motion smoothing and opens a window that shows the processed video feed in real-time.hamoco-run --scrolling_speed 20: sets a custom value for the scrolling speed. Note that for a given value, results may differ significantly depending on the operating system.hamoco-run --margin 0.2 --stop_sequence THUMB_SIDE CLOSE INDEX_MIDDLE_UP: adapts the size of the detection margin (indicated by the dark frame in the preview windows using--show), and changes the sequence of consecutive poses to stop the application.Configuration files with default values for the control parameters can be found in the installation folder, underhamoco/config/. Simply edit the file that corresponds to your operating system (posix.jsonforLinuxandnt.jsonforWindows) to save your settings permanently, and hence avoid specifying the parameters by hand in the console.Hand poses & Mouse actions:OPEN: the pointer is free and follows the center of the palm (indicated by the white square) ;CLOSE: the pointer stops all actions. The hand can be moved anywhere in the frame without moving the pointer. This is used to reset the origin of motion (see thenota benebelow) ;INDEX_UP: performs a left-click at the current pointer location. Execute twice rapidly for a double-click ;PINKY_UP: performs a right click at the current pointer location ;INDEX_MIDDLE_UP: holds the left mouse button down and moves the pointer by following the center of the palm. This is used for selection and drag & drop ;THUMB_SIDE: enables vertical scrolling using the first triggering location as origin. Scrolling up or down is done by moving the hand up or down relative to the origin while keeping the same hand pose ;N.B.note that, much like a real mouse, the recorded motion of the pointer isrelativeto its previous position. When your mouse reaches the edge of your mouse pad, you simply lift it and land it back somewhere on the pad to start moving again. Similarly, if your hand reaches the edge of the frame, the pointer will stop moving: simply close your fist and move it back into the frame to reset the origin of motion (exactly like when lifting and moving a real mouse).The various hand poses are illustrated below:Exiting the application:There are two ways to exit the application:In the preview mode (--showoption enabled), simply click on the preview window and pressESC;Execute a predetermined sequence of consecutive hand poses. The default sequence can be found in the help message (hamoco-run --help). A new sequence can be specified with the--stop_sequenceoption followed by the consecutive hand poses, or it can simply be changed in the.jsonconfiguration file.hamoco-datahamoco-dataactivates the webcam and allows to record your own labeled data for hand poses in order to train a custom neural-network-based classification model for the main application. This model can then be used in place of the one provided by default and will be more performant, as it will be trained on your personal and natural hand poses (seehamoco-train). Typehamoco-data --helpfor more information on the available options.This application requires two arguments:pose: a string that indicates the type of hand pose you intend to record. It should be one of:OPEN,CLOSE,INDEX_UP,PINKY_UP,THUMB_SIDE,INDEX_MIDDLE_UP.path_to_data: path to the folder inside of which you want the recorded data to be saved.Examples:hamoco-data OPEN data/ --delay 1.0: starts the recording for theOPENhand pose, stores the resulting data in thedatafolder (provided it exists!), and takes a new snapshot every second.hamoco-data INDEX_UP data/ --delay 0.25 --images: starts the recording for theINDEX_UPhand pose, stores the resulting data in thedatafolder, takes a new snapshot every 0.25s, and saves the images (in addition to the numeric data file used for training the model). Saving images can be useful if you want to manually check if your hand was in a correct position when its numerical data was recorded, and hence keep or remove specific data files accordingly.hamoco-data CLOSE data/ --reset --stop_after 200: starts the recording of theCLOSEhand pose, stores the resulting data in thedatafolder, deletes every previously recorded file for this hand pose, and automatically stop the recording after taking 200 snapshots.hamoco-trainProvided a path to a directory with compatible data,hamoco-traintrains a customizable NN-based classification model to predict a hand pose. This classification model can then be used in the main application in place of the one provided by default. Typehamoco-train --helpfor more information on the available options.This application requires two arguments:path_to_model: path to save the newly trained model.path_to_data: path to the data folder to use to train the model (seehamoco-data).Examples:hamoco-train my_custom_model.h5 data/ --hiden_layers 50 25 --epochs 20: trains and save a model namedmy_custom_model.h5that contains two hidden layers (with dimensions 50 and 25 respectively) over 20 epochs, by using the compatible data in thedatafolder.hamoco-train my_custom_model.h5 data/ --epochs 10 --learning_rate 0.1: trains and save a model namedmy_custom_model.h5with default dimensions over 20 epochs and with a learning rate of 0.1, by using the compatible data in thedatafolder.Your model can then be used in the main application with the--modelflag ofhamoco-run,e.g.hamoco-run --model <path_to_your_model>, or you can change the.jsonconfiguration file to point to it.AuthorJoris Paret