Vision-Based Hand Gesture Customization from a Single Demonstration
Authors: name: Soroush Shahi, name: Cori Tymoszek Park, name: Richard Kang, name: Asaf Liberman, name: Oron Levy, name: Jun Gong, name: Abdelkareem Bedri, name: Gierad Laput

Hand gesture recognition is becoming a more prevalent mode of human-computer
interaction, especially as cameras proliferate across everyday devices. Despite
continued progress in this field, gesture customization is often underexplored.
Customization is crucial since it enables users to define and demonstrate
gestures that are more natural, memorable, and accessible. However,
customization requires efficient usage of user-provided data. We introduce a
method that enables users to easily design bespoke gestures with a monocular
camera from one demonstration. We employ transformers and meta-learning
techniques to address few-shot learning challenges. Unlike prior work, our
method supports any combination of one-handed, two-handed, static, and dynamic
gestures, including different viewpoints. We evaluated our customization method
through a user study with 20 gestures collected from 21 participants, achieving
up to 97% average recognition accuracy from one demonstration. Our work
provides a viable path for vision-based gesture customization, laying the
foundation for future advancements in this domain.