Poster 1213

IPN Hand: A Video Dataset and Benchmark for
Real-Time Continuous Hand Gesture Recognition
Gibran Benitez-Garcia1, Jesus Olivares-Mercado2,Gabriel Sanchez-Perez2 and Keiji Yanai1
1The University of Electro-Communications, Tokyo, Japan
2Instituto Politecnico Nacional, Mexico City, Mexico

1

UEC Yanai Lab. Tokyo.

ⓒ 20192020 UEC Tokyo.

Continuous Hand Gesture Recognition
Problem:• Recognize dynamic hand gestures withzero lag.
• Currently available datasets only
include segmented gestures clips (oneper video).

Objective:• Introduce a new dataset able toevaluate real-time continuous hand
gesture recognition with sufficient
size for deep learning models.
2UEC Yanai Lab. Tokyo.

ⓒ 2020 UEC Tokyo.

Contributions• A new dataset named IPN Hand, which includes:– Multiple gestures per video with temporal labels (200 videos in total).
– Sufficient number of samples (4,218 instances) from different subjects (50 in total).
– Dynamic and static hand gestures (13 in total) for controlling touchless screens.
– Natural movements of the hand as non-gesture segments.
– Real-world scenario with different backgrounds (28 in total) .
– Publicly available including RGB frames, optical flow maps, and hand masks.
• An alternative input for multimodal real-time hand gesture recognition:– Real-time hand semantic segmentation to obtain hand masks.
– Evaluation of 3D-CNNs with multimodal inputs: RGB-Seg, and RGB-flow.

3

UEC Yanai Lab. Tokyo.

ⓒ 2020 UEC Tokyo.

Challenges of the dataset• Continuous gestures without transition states:

4

UEC Yanai Lab. Tokyo.

ⓒ 2020 UEC Tokyo.

Challenges of the dataset• Natural behaviors of users' hands as non-gesture states:

5

UEC Yanai Lab. Tokyo.

ⓒ 2020 UEC Tokyo.

Challenges of the dataset• Intra-class variability of gestures' duration:

Example gesture: “Double click with one finger”

6

UEC Yanai Lab. Tokyo.

ⓒ 2020 UEC Tokyo.

Challenges of the dataset• Different real-world backgrounds:

7

UEC Yanai Lab. Tokyo.

ⓒ 2020 UEC Tokyo.

Dataset gestures (static)

8

Class 1: Pointing with one finger

Class 2: Pointing with two fingers

Point-1f: 1010 instances

Point-2f: 1007 instances

UEC Yanai Lab. Tokyo.

ⓒ 2020 UEC Tokyo.

Dataset gestures (dynamic)

9

Class 3: Click with one finger

Class 4: Click with two fingers

Click-1f: 200 instances

Click-2f: 200 instances

UEC Yanai Lab. Tokyo.

ⓒ 2020 UEC Tokyo.

Dataset gestures (dynamic)

10

Class 5: Throw up

Class 6: Throw down

Th-up: 200 instances

Th-down: 200 instances

UEC Yanai Lab. Tokyo.

ⓒ 2020 UEC Tokyo.

Dataset gestures (dynamic)

11

Class 7: Throw left

Class 8: Throw rightTh-left: 200 instances

Th-right: 200 instances

UEC Yanai Lab. Tokyo.

ⓒ 2020 UEC Tokyo.

Dataset gestures (dynamic)

12

Class 9: Open twice

Class 10: Double click with one finger

Open-2: 200 instances

2click-1: 200 instances

UEC Yanai Lab. Tokyo.

ⓒ 2020 UEC Tokyo.

Dataset gestures (dynamic)

13

Class 11: Double click with two fingers

Class 12: Zoom in

2click-2: 200 instances

Zoom-in: 200 instances

UEC Yanai Lab. Tokyo.

ⓒ 2020 UEC Tokyo.

Dataset gestures (dynamic)
Class 13: Zoom out
Zoom-out: 200 instances

14

UEC Yanai Lab. Tokyo.

ⓒ 2020 UEC Tokyo.

Benchmark evaluation• Continuous hand gesture recognition:– Based on a two hierarchical 3D-CNNs approach [1].

UEC Yanai Lab. Tokyo.

ⓒ 2020UEC Tokyo.
15 [1] Kopuklu, O, et.al. Online Dynamic Hand Gesture Recognition
Including
Efficiency Analysis. T-BIOM 2020

Evaluation results• Continuous hand gesture recognition:– Levenshtein accuracy is used as evaluation metric for continuous recognition.

16

UEC Yanai Lab. Tokyo.

ⓒ 2020 UEC Tokyo.

Evaluation results• Isolated hand gesture recognition:– The best accuracy is obtained by RGB-flow with 86.32% (13 classes only).

17

P1 P2 C1 C2 T-u T-d T-l T-r O2 2c1 2c2 Z-i Z-o
Point-1f 92 6 0 0 1 0 0 1 0 0 0 0 0
Point-2f 4 95 0 0 0 0 0 1 0 0 0 0 0
Click-1f 4 4 73 0 0 2 0 0 0 13 0 2 2Click-2f 0 6 0 63 0 0 0 4 0 2 21 0 4Th-up 2 2 0 0 85 2 0 0 10 0 0 0 0Th-down 4 0 0 0 2 92 0 0 2 0 0 0 0Th-left 0 0 0 0 0 0 94 2 2 0 0 0 2Th-right 2 0 0 0 0 2 0 96 0 0 0 0 0
Open-2 0 0 0 0 2 0 0 0 87 0 0 12 02click-1f 6 0 31 0 0 0 0 0 0 60 2 2 02click-2f 0 4 0 33 0 0 0 0 0 2 60 2 0
Zoom-in 0 0 0 0 0 0 2 2 0 0 0 90 6
Zoom-o 0 4 2 4 0 ⓒ 20200 UEC0 Tokyo.
4 0 0 0 8 79
UEC Yanai Lab. Tokyo.

Conclusion• Conclusions:– Our new dataset is able to evaluate hand gesture recognition for isolated
and continuous benchmarks.
– Due to non-gesture segments and specific challenges, the SOTA 3D-CNNs
models only achieve 42.47% accuracy on the continuous benchmark.
– This demonstrates that our dataset will help to push advances in this field.

• Available at:– https://github.com/GibranBenitez/IPN-hand

18

UEC Yanai Lab. Tokyo.

ⓒ 2020 UEC Tokyo.

Poster 1213

Thank You

Code & Models19

UEC Yanai Lab. Tokyo.

ⓒ 2020 UEC Tokyo.