Hand Gesture Classifier for IoT Control
A small Keras network that classifies 16 hand gesture classes (15 gestures and "no gesture") from MediaPipe hand landmarks. It was trained for the course project hand-gesture-iot-control (Digital Image Processing, HCM-UTE, team of 4, 2025), where the gestures turn a fan and two lights on and off through an ESP32 and MQTT.
Files
| Path | Content |
|---|---|
SavedModel/saved_model_best/ |
The classifier as a TensorFlow SavedModel (TensorFlow 2.16.1) |
SavedModel/metadata.pkl |
Label list and feature names used by the project scripts |
labels.json |
The same label list in JSON, so you do not need to load the pickle |
hand_landmarker.task |
Google's MediaPipe Hand Landmarker model, included unchanged for convenience (Apache 2.0, from the MediaPipe models) |
Input
46 float32 values per hand, computed from the 21 MediaPipe landmarks (image x, y):
- Subtract the wrist (landmark 0) from every point, then divide by the largest absolute x or y value, so the points are in [-1, 1].
- Features 0 to 41: x and y of the 21 points in order.
- Features 42 and 43: unit vector from the wrist to the mean of the 4 knuckles (landmarks 5, 9, 13, 17).
- Features 44 and 45: unit vector from the index knuckle (5) to the pinky knuckle (17).
The exact code is normalize_features() in UI/Step5_RealtimePredictiion/tkinter_detection_classification_esp32.py in the GitHub repo.
Output
Softmax probabilities over the 16 labels in labels.json. Labels starting with S_ are symmetric gestures (either hand). A_LH_ and A_RH_ are FanLeft and FanRight done with the left or right hand.
How to use
import json
import numpy as np
import tensorflow as tf
model = tf.saved_model.load("SavedModel/saved_model_best")
predict = model.signatures["serving_default"]
labels = json.load(open("labels.json"))["labels"]
features = np.zeros((1, 46), dtype=np.float32) # replace with real features
probs = predict(input_layer_1=tf.constant(features))["output_0"].numpy()[0]
print(labels[int(probs.argmax())], float(probs.max()))
Training
| Setting | Value |
|---|---|
| Data | 18,639 photos taken by the team (640x640), 29,648 samples after mirroring the symmetric gestures |
| Model | Dense 256, 128, 64 with batch norm and dropout (0.4, 0.3), softmax over 16 classes |
| Split | 70% train, 15% validation, 15% test, random and stratified by class |
| Training | Adam, learning rate 1e-3, batch 128, early stopping on validation accuracy, class weights, landmark rotation and noise augmentation |
Results
On 1,282 photos taken separately from the training data, each photo run through MediaPipe and this classifier:
| Outcome | Photos | Share |
|---|---|---|
| Correct | 1,242 | 96.9% |
| Wrong class | 22 | 1.7% |
| No hand found by MediaPipe | 18 | 1.4% |
Leaving out the 18 photos with no hand found, 98.3% are correct. The main confusions are FanSpeed3 with Light2On and FanOff with Start. Validation accuracy during training was about 99.5%, but the random split by photo makes that number optimistic.
Limitations
- Trained on photos of our team members only. Accuracy drops in very low or very bright light and at extreme hand angles.
- This SavedModel was exported by an earlier version of the training script, so the training settings above may not match it exactly.
- In the live app, the prediction is also filtered by confidence, entropy and a voting window before any command is sent. The model alone does not do this.
Links
- Code: https://github.com/Ticasslo/hand-gesture-iot-control
- Report (Vietnamese): https://github.com/Ticasslo/hand-gesture-iot-control/blob/main/17_BaoCao.pdf
Team
Huỳnh Ngọc Thắng, Phạm Trần Thiên Đăng, Phạm Võ Nhất Kha, Huỳnh Thanh Nhân