IPN Hand: A Video Dataset and Benchmark for
Real-Time Continuous Hand Gesture Recognition
Gibran Benitez-Garcia ∗ , Jesus Olivares-Mercado† , Gabriel Sanchez-Perez † and Keiji Yanai∗∗ Department of Informatics, The University of Electro-Communications, Tokyo, Japan

Email: gibran@ieee.org, yanai@cs.uec.ac.jp† Instituto Politecnico Nacional, ESIME Culhuacan, Mexico City, Mexico

Email: jolivares@ipn.mx, gasanchezp@ipn.mx

Abstract—Continuous hand gesture recognition (HGR) is an
essential part of human-computer interaction with a wide range
of applications in the automotive sector, consumer electronics,
home automation, and others. In recent years, accurate andefficient deep learning models have been proposed for HGR.
However, in the research community, the current publicly available datasets lack real-world elements needed to build responsive
and efficient HGR systems. In this paper, we introduce a new
benchmark dataset named IPN Hand with sufficient size, variety,and real-world elements able to train and evaluate deep neuralnetworks. This dataset contains more than 4,000 gesture samples
and 800,000 RGB frames from 50 distinct subjects. We design
13 different static and dynamic gestures focused on interaction
with touchless screens. We especially consider the scenario whencontinuous gestures are performed without transition states, andwhen subjects perform natural movements with their hands
as non-gesture actions. Gestures were collected from about 30
diverse scenes, with real-world variation in background andillumination. With our dataset, the performance of three 3D-CNN
models is evaluated on the tasks of isolated and continuous realtime HGR. Furthermore, we analyze the possibility of increasing
the recognition accuracy by adding multiple modalities derived
from RGB frames, i.e., optical flow and semantic segmentation,
while keeping the real-time performance of the 3D-CNN model.
Our empirical study also provides a comparison with the publicly
available nvGesture (NVIDIA) dataset. The experimental results
show that the state-of-the-art ResNext-101 model decreases about
30% accuracy when using our real-world dataset, demonstratingthat the IPN Hand dataset can be used as a benchmark, and may
help the community to step forward in the continuous HGR. Our
dataset and pre-trained models used in the evaluation are publicly
available at github.com/GibranBenitez/IPN-hand.

I. I NTRODUCTION
Gestures are a natural form of human communication [1].
Hand gesture recognition (HGR) is an essential part of humancomputer interaction. Systems using vision-based interaction
and control are more common nowadays [2]–[4]. Compared to
traditional inputs such as mouse and keyboard, vision-based
interfaces can be more practical and natural based on theintuitiveness of designed gestures. HGR has a wide range ofapplications in the automotive sector, consumer electronics,
public transit, gaming, home automation, and others [4]–[6].
For these applications, HGR systems must be designed towork online and deal with continuous gestures that users
may input. There are mainly four issues that systems must
deal with for continuous HGR applications: (i) continuous
gestures without transition states, (ii) natural behaviors of

(a)

(b)

(c)

(d)

Fig. 1.
Some examples that show the challenges of our dataset. (a)Continuous gestures without transition states. (b) Clutter backgrounds. (c)Natural interaction with objects. (d) Weak illumination conditions.

users’ hands similar to target gestures, (iii) intra-class variability of gestures’ duration, and (iv) lag between performinga gesture and its classification. It is worth mentioning that (i)is particularly important for some real-world applications suchas interacting with touchless screens. For example, someone
can open a picture and zoom in, by using two continuous
gestures (double-click + zoom-in) without a transition statethat forces the hand returning to a neutral position before thesecond gesture. This example is shown in Figure 1(a).
Thanks to the advances on deep neural networks, in recent
years, accurate and efficient deep learning models have been
proposed to overcome the challenges on continuous HGR [7]–[11]. However, these methods were evaluated using datasetsthat do not cover the main real-world issues. Currently, it is
not easy to find a benchmark dataset able to evaluate the fourmain issues of continuous HGR. Most of the hand gesturedataset used for continuous HGR, like ChaLearn ConGD [12],nvGesture [7] and EgoGesture [13], do not include continuous gestures without transition states (i), nor natural hand
movements as non-gesture actions (ii). To the best of ourknowledge, there a no publicly available hand gesture datasetsthat cover these two issues of continuous HGR. Note that,some works have designed specific datasets for controlling
automotive interfaces [14], [15], that partially include theseissues. However, the datasets are not publicly available.

TABLE IC OMPARISON OF THE PUBLIC CONTINUOUS GESTURE DATASETS
DatasetChaLearn ConGD, 2016 [12]nvGesture, 2016 [7]EgoGesture, 2018 [13]IPN Hand (ours)

Instances
47,933
1,532
24,161
4,218

Videos
22,535
1,532
2,081
200

Instance/video
2.1
1.0
11.6
21.1

In this paper, we introduce a new dataset called IPN Hand
for the task of continuous hand gesture recognition. The
dataset contains more than four thousand RGB gesture samples
and 800 thousand frames from 50 distinct subjects. We design
13 classes of static and dynamic gestures for interaction withtouchless screens. Our dataset has the most realistic scenario
for continuous HGR than other hand gesture datasets. IPN
Hand includes the largest number of continuous gestures pervideo and the largest speed of intra-class variation for different subjects when they were performing the same gesture.
Besides, our dataset is more complex as it was collected fromabout thirty representative scenes with considerable variation,
including clutter backgrounds, strong and weak illumination
conditions, static and dynamic background environments. We
specially design two scenarios that reflects the real-world
issues of continuous HGR: when continuous gestures are
performed without transition states, and when subjects perform
natural movements with their hands as non-gesture actions.
Some examples of the main challenges of our dataset are
shown in Figure 1.
Given our dataset, we evaluate three state-of-the-art (SOTA)
methods based on 3D-CNN models for the tasks of isolated
and continuous real-time HGR. Furthermore, we analyze thepossibility of increasing the recognition accuracy by adding
multiple modalities derived from RGB frames. Specifically, weevaluate the data level fusion of RGB with semantic segmentation of target hands as an alternative of the RGB+Optical Flowor RGB+Depth modalities for real-time HGR. Our empirical
study also provides a comparison with the publicly available
nvGesture (NVIDIA) dataset. The experimental results on
the IPN Hand dataset demonstrate that it can be used as abenchmark for the data-hungry 3D-CNN methods, which may
help the community to step forward in the continuous HGR.
II. R ELATED W ORKS
A. Datasets for continuous HGR
Existing continuous HGR datasets differ by factors such as
scale, the number of classes, sensors used, and the domainof gestures. However, all of them must include non-gesture
frames to emulate the online behavior of real applications.
These frames are important to define the realism of thedesigned dataset. Therefore, we analyze the commonly used
datasets based on this element.
The ChaLearn LAP ConGD dataset [12] is derived from 9
different gesture domains, from Italian sign language, activities
to pantomime. It contains 249 classes and more than 40
thousand instances from 21 subjects, which makes ChaLearn
ConGD the largest dataset for continuous HGR. This dataset

Classes
249
25
83
13

ConGD

Subjects
21
20
50
50

Scenes151
628

nvGesture

View
3rd
3rd
1st
3rd

Modalities
RGB-D
RGB-D
RGB-D
RGB

EgoGesture

IPN HandFig. 2.
Comparison of gesture vs. non-gesture frames from the public
continuous gesture datasets. Note that only the first row (blue) shows examples
of gesture frames.

contains videos with one to five continuous gestures withtransition states. The nvGesture dataset [7] is designed tocontrol in-car automotive devices, and it includes 25 gesture
types from 20 subjects, consisting of 1 532 instances. Notethat this dataset is commonly used for online HGR evaluation,even though it only contains videos with isolated gestures. The
EgoGesture dataset [13] is a benchmark dataset for egocentric(first-person) view, which consists of 83 classes with more than
20 thousand instances from 50 subjects. This dataset includes
at max 12 instances per video. To the best of our knowledge,
our proposed IPN Hand dataset contains the largest numberof instances per video (21), and different scenes (28), making
it suitable for evaluating continuous HGR systems. Detailed
comparison between our IPN Hand and related gesture datasets
can be found in Table I.
Figure 2 shows the comparison of the non-gesture frames
from the public continuous gesture datasets. Most of the nongesture frames from the ChaLearn ConGD dataset consist oftransition states between gestures, usually showing the subject
in a neutral position as in the first column of the figure. On theother hand, the nvGesture shows drivers with their hands atthe steering wheel, while EgoGesture shows only backgroundswith the hands out of view as non-gesture frames. It is clearthat our dataset presents more challenge on distinguishinggesture vs. non-gesture frames, since we include subjects
performing natural movements with their hands.
B. Continuous HGRContinuous HGR can be divided into two stages: gesture
spotting and gesture classification. Gesture spotting aims to detect temporal video segments which contain gesture instances,while the classification stage aims to classify the gesture ofeach spotted segment. Multi-stream architectures have been
widely employed for both tasks. Simonyan et al. [16] were
the pioneers of fusing features from two modalities, using one
stream with RGB images and the other with flow fields for
isolated HGR. On This multi-modality approach is prevalent
as well for continuous HGR, as shown in the 2017 ChaLearn
Look At People (LAP) continuous gesture detection (ConGD)
challenge [17], where all entries used multi-stream architectures of at least RGB and depth. For instance, the winners [8]introduced a two-stream 3D-CNN combining hand-location
features of RGB and depth modalities by explicitly overlaying
a black mask on the input frames. They firstly spotted thegestures based on a dataset-specific observation: subjects raise
their hands at the beginning of gestures and put them down
again at the end. One year after ConGD challenge, Zhu etal. [9] overcame the results of the winners by proposing
a temporal dilated 3D-CNN architecture to binary classify
gesture/non-gesture frames, and 3D-CNN+LSTM for gestureclassification. The current SOTA method of ConGD uses a 12channels architecture with extra RNN layers to simultaneously
spot and classify continuous gestures [10]. Recently, Kopuklu
et al. [11] proposed a hierarchical structure of 3D-CNN
architectures to detect and classify continuous hand gestures.
Their spotting method consists of detecting gesture frames
using a shallow 3D-CNN model on eight consecutive frames,
while a SOTA model is used for classification only if a gestureis detected.
To evaluate our dataset for real-world applications of continuous HGR, detection and classification must work online oreven with a negative lag between performing a gesture and its
classification feedback. Therefore, we follow the hierarchicalstructure of 3D-CNNs [11], since it can detect a gesture whena confidence measure reaches a threshold level before thegesture ends (early-detection). Furthermore, we evaluate themulti-modality accuracy derived from RGB frames, by keeping in mind the real-time performance. We employ a data level
fusion to avoid the significant increase in the computational
cost of the 3D-CNN models. Besides, we propose to use
semantic segmentation results as an alternative to the absent
depth modality and the computational expensive optical flow.
III. T HE IPN H AND DATASET
A. Data collection
To collect the dataset, the RGB videos were recorded in
the resolution of 640 × 480 with the frame rate of 30 fps.
The participants were asked to record the gestures using theirown PC or laptop by keeping the defined resolution and frame
rate. Thus, the distance between the camera and each subject
varies, since we instructed all participants to be located in
a comfortable position to manipulate the screen of their PC
with hand gestures. In this way, the videos were generated
with 28 different scenes in total. For some participants that
do not have access to a camera that can cover the recording
specifications, we prepared three scenes (scene 2-4, shown in
Figure 3(b)-(d)), including clutter and variable illumination(a) 1.Plain(b) 2.WindowA(c) 3.OfficeA(d) 4.OfficeB(e) 5.DarkRoom(f) 6.WindowB

Fig. 3. Some examples of the scenes with more videos in the dataset. Notethat all subjects are developing the same gesture, ”pointing with two fingers”.

backgrounds. When collecting data, we first teach the subjects
how to perform each gesture and tell them the gesture names(short descriptions). Then we generate four gesture name lists
with random order for recording four videos per subject. Thus,
the subject was told the gesture name and performed thegesture accordingly. They were asked to continuously perform
21 gestures with 3 random breaks as a single session which
was recorded as a video.
B. Dataset characteristics
1) Gesture Classes: We design the gestures in our dataset
focused on interaction with touchless screens. We include
gestures able to control the location of the pointer on thescreen (pointer), and to manipulate interfaces (actions). We
take some gestures used in common smartphones, as they arenatural and easy to remember by the users. Thus, we designed
13 gestures (shown in Figure 4) defined to control the punterand actions.
For the punter location, we defined two static gestures ofpointing to the screen with one and two fingers, respectively.
Note that these gestures are static due to the gesture itself
does not need temporal information to be detected. However,
the application of this gesture includes the hand movement
to control the pointer. For the action gestures, we defined 11
gestures, including click with one and two fingers, throw tofour positions (up, down, left & right), double click with one
and two fingers, zoom-in, and zoom-out. Table II provides thename and descriptions of each gesture in our dataset. Notethat we also include segments were natural hand movements
are performed as non-gestures states.
2) Subjects: A small number of subjects could make theintra-class variation very limited. Hence, we asked 50 subjects
for our data collection. In the 50 subjects, there are 16 females
and 34 males. The average age of the subjects is 21.2, where
the minimum age is 18, and the maximum age is 25. All of
the subjects are currently students at Instituto Politecnico from
Mexico.
3) Extreme illumination and clutter backgrounds: To evaluate the robustness to the illumination change of the baseline

Fig. 4. Examples of the 13 gesture classes included in our dataset. For visualization purposes, semantic segmentation masks were blended to the RGB images.

TABLE IIS TATICS PER GESTURE OF OUR IPN H AND DATASET
id

GestureNameInstancesMean
duration(std)

Duration
nstdk

0

No gesture

No-gest1431

147 (133)

0.904

1

Pointing with
one finger

Point-1f

1010

219 (67)

0.308

2

Pointing with
two fingers

Point-2f

1007

224 (69)

0.309

3

Click with
one finger

Click-1f

200

56 (29)0.517

4

Click with
two fingers

Click-2f

200

60 (43)0.718

5Throw up

Th-up

200

62 (25)0.400

6Throw down

Th-down

201

65 (28)0.424

7Throw left

Th-left

200

66 (27)0.400

8Throw rightTh-right

200

64 (28)0.439

9Open twice

Open-2200

76 (31)0.410

10

Double clickwith one finger

2click-1f

200

68 (28)0.412

11

Double clickwith two fingers

2click-2f

200

70 (30)0.435

12Zoom in

Zoom-in200

65 (29)0.440

13Zoom out

Zoom-o200

64 (28)0.432

Fig. 5. Distribution of the 28 different scenes included in the dataset.

methods, we have data collected under extreme conditions
such as facing a window with strong sunlight (Figure 3(b)),
or in a room with almost null artificial light (Figure 1(d)).
Some scenes with static background placed with student-life
stuff (Figure 1(b)), and dynamic background with walking
people appearing in the camera view (Figure 3(d)-(f)) werealso included.
C. Dataset statics
Fifty distinct subjects participated in performing 13 classes
of gestures in 28 diverse scenes. Totally 4,218 gesture instances and 800,491 frames were collected in RGB. Figure 5
illustrates the sample distribution on each video among the 28

scenes. In the figure, the horizontal axis and the vertical axis
indicate the ID of the scenes and the number of the videos,respectively. In addition, Figure 3 shows examples of the six
scenes with more videos on our dataset.
When data collection, 21 gestures are considered as asession and recorded in a single video. Thus, we get 200 RGB
videos in total. Note that the order of the gestures performed issemi-randomly generated, trying to include one dynamic after
one static gesture, which resembles the realistic interaction
with a touchless screen. The start and end frame index of eachgesture instance in the video were manually labeled, which
provides enough information to train and evaluate continuous
HGR approaches in a fully supervised manner. In the dataset,the minimum length of a gesture is 9 frames. The maximum
length of a gesture is 650 frames.
In Table III, we show the data statistics of our dataset
compared with the continuous HGR datasets that are currently
publicly available. The dataset statistics include the numberof total frames, the mean video durations, the mean of thegesture sample durations, the standard deviation of gesture
durations, the percentage of the training data. The mean andstandard deviation of the gesture sample durations is calculated
over the samples from all gesture classes in the dataset.
Following [13], we use the normalized standard deviation

TABLE III
S TATICS OF THE PUBLIC CONTINUOUS GESTURE DATASETS
DatasetChaLearn ConGD, 2016 [12]nvGesture, 2016 [7]EgoGesture, 2018 [13]IPN Hand (ours)

Frames1,714,629
122,560
2,953,224
800,491

Mean video duration
76.1
80.0
1,419.1
4,002.5

Mean gesture duration
41
71
38
140

Duration std
18.5
32.3
13.9
93.9

Duration nstdk
0.37
0.3
0.33
0.43

% of train
0.635
0.685
0.595
0.739

A. Hierarchical 3D-CNNs for continuous HGR

Fig. 6. The flowchart of the continuous HGR approach based on two
hierarchical 3D-CNNs models.

for gesture duration in each gesture class to describe thespeed variation of different subjects when performing the same
gesture. The normalized standard deviation of durations in a
gesture class k is calculated as follows:
s
PN k
k̄ 2
1
i (li − l )nstdk = k̄(1)Nlwhere in gesture class k, lik represents the duration of the ith
sample, lk̄ is the average duration of samples, and N is thenumber of samples. We get the average nstdk over all gestureclasses, for the whole dataset.
From Table III, we can see that our IPN Hand datset has
the largest duration nstdk (0.43), significantly larger than
EgoGesture, which is the second largest. This demonstrates
that our dataset has a large speed variation for different
subjects when performing the same gesture. In resume, ourproposed dataset covers all the real-world issues for continuous
HGR, as described in the introduction: (i) continuous gestures
without transition states, (ii) natural behaviors of users’ hands
similar to target gestures, (iii) intra-class variability of gestures’ duration (nstdk ).
IV. B ENCHMARK E VALUATION
We evaluate three SOTA 3D-CNN models as baselines forthe tasks of isolated and continuous HGR, with our new IPN
Hand dataset. For continuous HGR, we adopt a two-model
hierarchical architecture to detect and classify the continuous
input stream of frames.

As mentioned before, we use the framework proposedby Kopuklu et al. [11] to detect and classify continuous
hand gestures. The flowchart of the two-model hierarchicalarchitecture is shown in Figure 6. Sliding windows with aconstant stride run through incoming video frames where
the detector queue is placed at the very beginning of theclassifier queue. If the detector recognizes a gesture, then theclassifier is activated. The detector’s output is post-processed
for more robust performance, and the final decision is madeusing a single-time activation block where only one activation
occurs per performed gesture. Note that the sliding windowof the detector is the only who is working every single stride.
In contrast, the classification window is activated based on
detectors output, so the inference time is not always the sumof both models.
Since we also evaluate multi-modal 3D-CNNs models,
we first calculate the semantic segmentation masks of eachstreaming frame. For this task, we use the efficient semantic segmentation approach of HarDNet (Harmonic Dense
Net) [18] which achieves SOTA results with a network inspired by DenseNet [19]. Its core component, the HarDBlock(Harmonic Dense Block), is specifically designed to address
the problem of the GPU memory traffic. Therefore, we canachieve real-time performance on each frame before applying
3D-CNNs models.
B. Experimental setup
We randomly split the data by subject into training (74%),
and testing (26%) sets, resulting in 148 training, and 52 testing
videos. The numbers of gesture instances in training, andtesting splits are 3 117, and 1 101, respectively. Thus, 37 and13 subjects were designed for training and testing, respectively.
1) Implementation details: We compare the 3D-CNN models of C3D [20], and the 3D versions [21] of Resnet-50 [22]and ResNeXt-101 [23], as deep models for classification. On
the other hand, for the shallow 3D-CNN detector we use
the ResLight (Resnet-10) as proposed in [11]. All 3D-CNN
models were trained using a fully supervised strategy using
the manually segmented gestures.
For the multi-modality tests, we train a HarDNet [18]model, and use the SPyNet [24] approach for calculation of semantic segmentation and optical flow, respectively. We trained
the HarDNet model with a synthetic hand pose estimation
dataset [25] which contains more than 40 thousand images
with hands fully annotated at pixel-level1 . For the SPyNet, we
1 https://lmb.informatik.uni-freiburg.de/projects/hand3d/

TABLE IV
R ESULTS OF ISOLATED HGR TASK USING OUR IPN H AND DATASET
Model
C3D
ResNeXt-101
ResNeXt-101
ResNeXt-101
ResNet-50
ResNet-50
ResNet-50

Input sequence
32-frames
32-frames
32-frames
32-frames
32-frames
32-frames
32-frames

Modality
RGB
RGB
RGB-Flow
RGB-Seg
RGB
RGB-Flow
RGB-Seg

Results
77.75
83.59
86.32
84.77
73.1
74.65
75.11

Parameters50.75 M
47.51 M
47.56 M
47.56 M
46.25 M
46.27 M
46.27 M

Model Size
387 MB363 MB363 MB363 MB
353 MB
353 MB
353 MB

Inference time76.2 ms
27.7 ms
28.9 ms
28.9 ms
17.8 ms
18.2 ms
18.2 ms

TABLE VC OMPARISON OF THE EXTRA PROCESSES FOR MULTI - MODALITY MODELS .
Process

Model

Params.

Segmentation
Optical FlowHarDNetSPyNet

4.114 M
1.440 MModel
size
15.8 MB5.50 MB

Inferencetime
8.1 ms21.9 msused the open-source implementation and pre-trained modelof [26] to calculate real-time densely optical flow from each
input frame.
All 3D-CNN models were pre-trained on the Jester
dataset [27], while the HarDNet on the ImageNet dataset [28].
Contrary, since ResLight is a compact model, we trained itfrom scratch using all non-gesture and gesture instances of the
IPN Hand training set. The inference time (FPS) was measured
on an Intel Core i7-9700K desktop with a single NVIDIAGTX 1080ti GPU. More implementation details, as well as
training and evaluation codes can be found in our open-source
repository2
C. Isolated HGR task
We evaluate this task with the conventional classification
metric. We segment the video sequences into isolated gesturesamples based on the beginning and ending frames manuallyannotated. The learning task is to predict class labels for eachgesture sample. We use classification accuracy, which is thepercent of correctly labeled examples, and the confusion matrix of the predictions, as evaluation metrics for this learningtask.
1) Experimental results using the IPN Hand dataset:
Table IV presents the results of evaluated models with different
modalities using our IPN Hand dataset. As expected, thebest results were obtained by the ResNeXt-101 model, which
can barely achieve real-time performance with the different
modalities. However, it is clearly faster and more accurate than
the robust C3D model. On the other hand, ResNet-50 is themost efficient model, but present the lowest accuracy results
among the evaluated approaches.
It is worth noting that our RGB-seg achieves competitive
results compared to the RGB-flow, which is significant since
the optical flow process is more computationally expensive. To
evaluate the efficiency of the multi-modal approaches, Table V
shows the inference time and the model size of extra-processes
related to semantic segmentation and optical flow estimation.
2 https://github.com/GibranBenitez/IPN-handFig. 7. Confusion matrix of the best result obtained from ResNext-101 modelwith RGB-flow.
TABLE VIR ESULTS OF ISOLATED HGR TASK USING THE NV G ESTURE DATASET
Model
ResNeXt-101
ResNeXt-101
ResNeXt-101

Input sequence
32-frames
32-frames
32-frames

Modality
RGB
RGB-Flow
RGB-Seg

Results
79.46
82.36
82.15

From this table, we can see that the semantic segmentation ismore than two times faster than the optical flow, making the
RGB-seg alternative feasible for real-time applications.
In addition, in the Figure 7, we show the confusion matrixof the best result obtained for isolated HGR, ResNext-101
with RGB-flow. As expected, the problems are related to thegestures that are closer such as clicks with double clicks.
2) Experimental results using the nvGesture dataset: As
mentioned before, we also evaluate the performance of our
RGB-seg alternative with a common online HGR dataset.
Table VI presents the results of the evaluated models withdifferent modalities using the nvGesture dataset. From theseresults, we can notice that the trend of the multi-modal resultsfrom IPN Hand is still valid (RGB-flow > RGB-seg > RGB).
In addition, Table VII shows the result of our real-time
RGB-seg alternative compared to SOTA methods of nvGesturedataset. From this table, we validate the use of semantic
segmentation as an important source of valuable featuresfor HGR. The results of ResNeXt-101 with RGB-seg, arecompetitive to SOTA methods that even employ different
modalities, such as depth. These findings are significant due
to the semantic segmentation process only takes 8 ms with an

TABLE VII
C OMPARISION WITH SOTA METHODS OF NV G ESTURE DATASET.
Model
C3D [20]
R3DCNN [7]
MTUT [29]R3DCNN [7]
MFF [30]
R3DCNN [7]
ResNeXt-101(a) RGB

Modality
RGB
RGB
RGB*
RGB+Flow
RGB+Flow
Depth+Flow
RGB+Seg(b) RGB-seg

Results
73.8
74.1
81.3
79.3
84.7
82.4
82.2

(a) Detection results(c) RGB-flow

Fig. 8. Confusion matrices of gesture detection on the IPN hand dataset
using different modalities for the Resnet-Light model.

input image of 360 × 240 pixels.
D. Continuous HGR task
We use the Levenshtein accuracy [11] as evaluation metricfor the continuous HGR taks. This metric employs the Levenshtein distance to measure the distance between sequencesby counting the number of item-level changes (insertion,
deletion, or substitutions) to transform one sequence into theother. In the case of continuous hand gestures, the difference
between the sequence of predicted and ground truth gestures
is measured. For example, if a ground truth sequence is[1, 2, 3, 4, 5, 6, 7, 8, 9] and predicted gestures of a video is[1, 2, 7, 4, 5, 6, 6, 7, 8, 9], the Levenshtein distance is 2. Definedby the deletion of one of the ”6” which is detected two times,
and the substitution of ”7” with ”3”. Thus, the Levenshtein
accuracy is obtained by averaging this distance over thenumber of true target classes. In our example, the accuracy
is 1 − (2/9) × 100 = 77.78%.
We obtain the average Levenshtein accuracy over the 52
testing videos to asses the continuous HGR performance. Besides, we also evaluate the detection accuracy of the 3D-CNN
detectors with different multi-modalities. Note that detectors
are trained and evaluated using isolated gesture vs. non-gesturesamples. So that we use a binary classification accuracy, andthe confusion matrix of the predictions, as evaluation metrics
for these models.
1) Experimental results: First, we evaluate the detector
model with the different modalities in Table VIII. As expected,
the same trend is maintained, but since ResLight-10 is amuch more compact model, the inference time is not affected.
It is also important to analyze the misclassification of thedetector, therefore Figure 8 shows the confusion matrices fromeach modality. It is clear that the benefits from multi-modal
approaches are reflected in the detection of non-gesture frames,
since the RGB approach misrecognized 85% of these frames.
We evaluate the complete process of the hierarchical twomodel approach using Resnet-50 and ResNeXt-101 as classifiers, and ResLight-10 as a detector. In addition, we also(b) Classification results
Fig. 9. Qualitative results of the proposed networks. The most significant
improvements occur on pixels belonging to large objects.

analyze the multi-modal alternatives covering real-time performance. Table IX shows the Levenshtein accuracy, as wellas the model size and inference time of each evaluated modeland modality. The results of this test make more evident theadvantages of using the RGB-seg alternative for real-time
continuous HGR. For instance, the inference time results of
ResNeXt-101 RGB are comparable with those of ResNet-50
RGB-seg. However, the Levenshtein accuracy of the latter issignificantly better (around 10% of improvement). Finally, weshow the temporal visualization of continuous predictions from
one testing video in Figure 9.
V. C ONCLUSION
In this paper, we introduced a new benchmark dataset forcontinuous HGR that includes real-world issues, such as continuous gestures without transition states, natural behaviors ofusers’ hands, and large intra-class variability of gestures’ duration. Besides, we evaluate the data level fusion of RGB withsemantic segmentation as an alternative of the RGB+Optical
Flow or RGB+Depth modalities for real-time HGR. From theexperimental results, we conclude that RGB-seg is a suitable
multi-modal alternative for real-time continuous hand gesturerecognition. Furthermore, we believe that the proposed dataset
can be used as a benchmark and help the community to movesteps forward on the continuous HGR.
ACKNOWLEDGMENT
This work was supported by JSPS KAKENHI Grant Number 15H05915, 17H01745, 17H06100 and 19H04929.

TABLE VIIIR ESULTS OF THE DETECTOR MODEL USING OUR IPN H AND DATASET
Model
ResLight-10
ResLight-10
ResLight-10

Input sequence
8-frames
8-frames
8-frames

Modality
RGB
RGB-Flow
RGB-Seg

Results
75.4
82.43
80.06

Parameters0.895 M
0.908 M
0.908 M

Model Size
6.83 MB
6.94 MB
6.94 MB

Inference time
2.4 ms
2.9 ms
2.9 msTABLE IXR ESULTS OF THE HIERARCHICAL TWO - MODEL APPROACH FOR CONTINUOUS HGR USING OUR IPN H AND DATASET
Model
ResNeXt-101
ResNeXt-101
ResNeXt-101
Resnet-50
Resnet-50
Resnet-50

Modality
RGB
RGB-Flow
RGB-seg
RGB
RGB-Flow
RGB-seg

Results
25.34
42.47
39.01
19.78
39.47
33.27

detector
6.83 MB12.4 MB22.7 MB6.83 MB12.4 MB22.7 MB

Model sizeclassifier
363 MB363 MB363 MB
353 MB
353 MB
353 MBR EFERENCES[1] A. Kendon, “Gesticulation and speech: Two aspects of the process ofutterance,” The relationship of verbal and nonverbal communication,
no. 25, p. 207, 1980.
[2] M. Leo, G. Medioni, M. Trivedi, T. Kanade, and G. M. Farinella,“Computer vision for assistive technologies,” Computer Vision andImage Understanding, vol. 154, pp. 1–15, 2017.
[3] L. P. Berg and J. M. Vance, “Industry use of virtual reality in productdesign and manufacturing: a survey,” Virtual reality, vol. 21, no. 1, pp.
1–17, 2017.
[4] S. S. Rautaray and A. Agrawal, “Vision based hand gesture recognition
for human computer interaction: a survey,” Artificial intelligence review,
vol. 43, no. 1, pp. 1–54, 2015.
[5] C. A. Pickering, K. J. Burnham, and M. J. Richardson, “A researchstudy of hand gesture recognition technologies and applications forhuman vehicle interaction,” in 2007 3rd Institution of Engineering andTechnology conference on automotive electronics. IET, 2007, pp. 1–15.
[6] F. Parada-Loira, E. González-Agulla, and J. L. Alba-Castro, “Handgestures to control infotainment equipment in cars,” in IEEE Intelligent
Vehicles Symposium, 2014, pp. 1–6.
[7] P. Molchanov, X. Yang, S. Gupta, K. Kim, S. Tyree, and J. Kautz,“Online detection and classification of dynamic hand gestures withrecurrent 3d convolutional neural network,” in IEEE Conference on
Computer Vision and Pattern Recognition (CVPR), 2016, pp. 4207–
4215.
[8] Z. Liu, X. Chai, Z. Liu, and X. Chen, “Continuous gesture recognition
with hand-oriented spatiotemporal feature,” in IEEE International Conference on Computer Vision Workshop (ICCVW), 2017, pp. 3056–3064.
[9] G. Zhu, L. Zhang, P. Shen, J. Song, S. Shah, and M. Bennamoun,“Continuous gesture segmentation and recognition using 3dcnn andconvolutional lstm,” IEEE Transactions on Multimedia, 2018.
[10] P. Narayana, J. R. Beveridge, and B. Draper, “Continuous gesturerecognition through selective temporal fusion,” in International Joint
Conference on Neural Networks (IJCNN). IEEE, 2019, pp. 1–8.
[11] O. Köpüklü, A. Gunduz, N. Köse, and G. Rigoll, “Real-time hand gesture detection and classification using convolutional neural networks,”
in 14th IEEE International Conference on Automatic Face & Gesture
Recognition (FG), 2019, pp. 1–8.
[12] J. Wan, Y. Zhao, S. Zhou, I. Guyon, S. Escalera, and S. Z. Li, “Chalearn
looking at people rgb-d isolated and continuous datasets for gesturerecognition,” in IEEE Conference on Computer Vision and Pattern
Recognition Workshops (CVPRW), 2016, pp. 761–769.
[13] Y. Zhang, C. Cao, J. Cheng, and H. Lu, “Egogesture: A new dataset andbenchmark for egocentric hand gesture recognition,” IEEE Transactions
on Multimedia, vol. 20, no. 5, pp. 1038–1050, 2018.
[14] G. Benitez-Garcia, M. Haris, Y. Tsuda, and N. Ukita, “Similar finger
gesture recognition using triplet-loss networks,” in Sixteenth IAPR
International Conference on Machine Vision Applications (MVA). IEEE,
2019, pp. 1–6.
[15] ——, “Finger gesture spotting from long sequences based on multistream recurrent neural networks,” Sensors, vol. 20, no. 2, p. 528, 2020.

total
370 MB
375 MB386 MB360 MB365 MB376 MBdetector
2.9 ms11.1 ms
24.8 ms
2.9 ms11.1 ms
24.8 ms

Inference timeclassifier
total
27.7 ms
30.1 ms
28.9 ms
53.7 ms
28.9 ms
39.9 ms
17.8 ms
20.4 ms
18.2 ms
43.1 ms
18.2 ms
29.2 ms[16] K. Simonyan and A. Zisserman, “Two-stream convolutional networks
for action recognition in videos,” in Advances in Neural Information
Processing Systems (NeurIPS), 2014, pp. 568–576.
[17] J. Wan, S. Escalera, G. Anbarjafari, H. J. Escalante, X. Baró, I. Guyon,
M. Madadi, J. Allik, J. Gorbova, C. Lin et al., “Results and analysis ofchalearn lap multi-modal isolated and continuous gesture recognition,and real versus fake expressed emotions challenges,” in IEEE International Conference on Computer Vision Workshop (ICCVW), 2017, pp.
3189–3197.
[18] P. Chao, C.-Y. Kao, Y.-S. Ruan, C.-H. Huang, and Y.-L. Lin, “HarDNet:
A Low Memory Traffic Network,” in The IEEE International Conference
on Computer Vision (ICCV), October 2019.
[19] G. Huang, Z. Liu, L. van der Maaten, and K. Q. Weinberger, “Densely
Connected Convolutional Networks,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
[20] D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri, “Learning
spatiotemporal features with 3d convolutional networks,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015, pp.
4489–4497.
[21] K. Hara, H. Kataoka, and Y. Satoh, “Can spatiotemporal 3d cnns retrace
the history of 2d cnns and imagenet?” in IEEE Conference on Computer
Vision and Pattern Recognition (CVPR), 2018, pp. 6546–6555.
[22] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning forimage recognition,” in IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), 2016, pp. 770–778.
[23] S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He, “Aggregated residualtransformations for deep neural networks,” in IEEE Conference on
Computer Vision and Pattern Recognition (CVPR), 2017, pp. 1492–
1500.
[24] A. Ranjan and M. J. Black, “Optical flow estimation using a spatial
pyramid network,” in IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), 2017, pp. 4161–4170.
[25] C. Zimmermann and T. Brox, “Learning to estimate 3d hand pose fromsingle rgb images,” in IEEE International Conference on Computer
Vision (ICCV), 2017, pp. 4903–4911.
[26] S. Niklaus, “A reimplementation of SPyNet using PyTorch,” https://github.com/sniklaus/pytorch-spynet, 2018.
[27] J. Materzynska, G. Berger, I. Bax, and R. Memisevic, “The jester dataset:
A large-scale video dataset of human gestures,” in IEEE International
Conference on Computer Vision Workshop (ICCVW), 2019, pp. 0–0.
[28] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma,
Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and
L. Fei-Fei, “ImageNet Large Scale Visual Recognition Challenge,”
International Journal of Computer Vision (IJCV), vol. 115, no. 3, pp.
211–252, 2015.
[29] M. Abavisani, H. R. V. Joze, and V. M. Patel, “Improving the performance of unimodal dynamic hand-gesture recognition with multimodal
training,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 1165–1174.
[30] O. Kopuklu, N. Kose, and G. Rigoll, “Motion fused frames: Data level
fusion strategy for hand gesture recognition,” in IEEE Conference on
Computer Vision and Pattern Recognition Workshops (CVPRW), 2018,
pp. 2184–21 848.