아래는 어떤 내용인가?

Quantized CNN-based efficient hardware architecture for real-time hand gesture recognition
Author links open overlay panel
Mohita Jaiswal
, 
Vaidehi Sharma
, 
<PRESIDIO_ANONYMIZED_PERSON>
, 
Sandeep Saini
, 
Raghuvir Tomar

Show more

Add to Mendeley

Share

Cite
https://doi.org/10.1016/j.mejo.2024.106345
Get rights and content
Abstract
Nowadays, Convolutional Neural Networks (CNN) have been widely adopted for vision-based hand gesture recognition. Several existing CNN architectures designed for gesture classification perform well with high accuracy but require a high memory footprint and processing when deployed on low-power embedded devices. To address this issue, we present a quantized CNN-based efficient framework for meeting real-life hand gesture recognition challenges. The proposed quantized CNN is designed and implemented using the FINN-based pipelined streaming architecture on an FPGA. Moreover, hardware-based optimizations are used to minimize the resources needed and to achieve fast memory access. Experimental results demonstrate that the developed recognition system achieves an average accuracy of 92% on the numeral database of Indian Sign Language (ISL). Additionally, our optimized design attains an inference latency of 0.85ms for real-time single gesture prediction on the PYNQ Zynq Ultrascale (ZU) FPGA, consuming only 3.63 W of power. The proposed design achieves a better trade-off between hardware resource utilization and speed performance, over previous designs.
Introduction
Hand gesture recognition has significantly become an engrossing area in the sphere of human–computer or human–machine interaction. [1], [2]. The use of hand gestures and facial expressions serves as a common form of human communication and interaction. Gesture recognition technology implemented on embedded platforms would benefit a wide variety of application areas, including virtual reality systems, robotics control, healthcare technology, surveillance systems, motion gaming, and sign language translation [2].
Sign language recognition is the challenging and most demanding application of hand gesture recognition. This gesture-based language has fast become a vital communication tool for hearing-impaired and speech-impaired persons for social interactions. The task of recognizing hand sign gestures in complex real-time situations poses inherent challenges like inter-hand pose variations and orientations [3], self-similarity among hand fingers, improper illuminance variations, and background clutter. Moreover, adopting hand gestures for spontaneous interaction with mobile devices and consumer electronics is generally restricted by memory requirements, high power consumption, and low computing speed. However, real-time interpretation and assistance have become essential for low-power embedded devices. The implementation of gesture recognition tasks on resource-constrained devices has become a research area of high contemporary interest.
With the rapid advent of Deep Learning (DL) algorithms in the past few years, many researchers have been reporting their achievements in implementing DL algorithms especially, Convolutional Neural Networks (CNN) for gesture classification [4], [5], [6], [7], [8] and thus, outperforms well-known Machine Learning (ML) algorithms. CNN is one of the most efficient DL algorithms and is specifically used to tackle cumbersome image classification tasks [9]. Researchers have also explored various types of data to detect and segment hand traits, including red–green–blue (RGB) [5], skin color [10], and depth [11].
In recent years, a plethora of ML-based and DL-based techniques have been reported using the aforementioned data types on sign language recognition in the published literature. Dudhal et al. [12] presented a CNN and scale-invariant feature transform (SIFT)-based methodology to recognize 50 static Indian Sign Language (ISL) gestures. The SIFT is hybridized with adaptive thresholding and Gaussian smoothing techniques for feature extraction and achieves 91.28% and 92.78% accuracy. Sharma and Anand [13] discussed the performance of two pre-trained deep models with four adaptive gradient-based optimizers on static gestures of ISL. These authors claimed that their model InceptionResNetV2 with Adam optimizer performs better as compared to other recent ML approaches and achieves an accuracy of 94.42% and 85.65% on numerals and alphabets, respectively. Sharma et al. [14] evaluated the 26 ISL gestures on three models i.e. Visual Geometry Group (VGG-16) with transfer learning, a pre-trained VGG-16 with finetuning, and a VGG-16 with a hierarchical neural network. The hierarchical model achieves accuracy i.e. 98.52% for one-hand and 97% for two-hand gestures and outperforms the other two models. Gangrade et al. [15] proposed a novel method to recognize numerals (0–9) static ISL gestures using the RGBD sensor. The variant of Oriented FAST and Rotated BRIEF (ORB), SIFT, and Speed Up Robust Features (SURF) methods are used to extract hand features and ORB performs best among others. The ORB features are tuned with k-Nearest Neighbor (k-NN) classification and produce a recognition accuracy of 93.26% on the ISL dataset.
Sharma and Singh [16] proposed a CNN based approach to evaluate the performance of self-collected ISL and American Sign Language (ASL) datasets and achieved recognition accuracy of 92.43%, 88.01%, and 99.52%. An automated gesture recognition system using a skin segmentation technique and Support Vector Machine (SVM) classifier was proposed by Reshna and Jayaraju [10] to extract hand region and then, recognize ISL sign gestures in real-time. The above-reported methods are only verified using their respective datasets due to the lack of a benchmark dataset of ISL. This issue has hampered the development of hand sign recognition, particularly in ISL. In this work, we prepare a diverse ISL dataset based on numerals which can benefit other researchers working in the same field. Few attempts have been made to recognize hand gestures using RGB images with different lighting conditions [13], [14], [16]. But, due to the illuminance with varying intensity and color balance associated with the quality of RGB images, it is challenging to develop gesture classification tasks in a real-time environment. To address this issue, we propose to use skin color segmentation with an Otsu thresholding-based approach to offer more reliable gesture recognition. All the aforementioned studies involve software-based methodologies, where the feasibility of the system was evaluated on the power-hungry Graphical Processing Unit (GPU) platforms. Implementing computing-intensive CNNs on these platforms requires huge power consumption and memory size. Therefore, such algorithms are not suitable for embedded systems. Hereby, Field-Programmable Gate Arrays (FPGAs) are the better alternative that can tackle both computational expense and power consumption constraints. Recently, many studies have used FPGAs for Deep Neural Network (DNNs) acceleration [17], [18], [19], [20]. The high parallelism and flexible architecture of FPGAs make them a suitable embedded platform to perform hand gesture recognition tasks.
In recent times, some of the research has been conducted on developing SLR systems. Barbhuiya et al. [21] developed an efficient gesture recognition system based on a deep ensemble neural network for ASL classification. The authors employed a pre-trained VGG16 architecture network, embedding a self-attention layer within the architecture to capture distinguishing image features. Their suggested methodology attained 99.76% and 95.10% accuracy on two publicly available hand gesture datasets. In Ref. [22], the authors presented a robust gesture recognition-based methodology by combining CNN with Zernike moment-based features. For gesture classification, they utilize the library for support vector machines (LIBSVM). Their proposed model was evaluated on four different standard datasets. Barbhuiya et al. [23] proposed an architecture based on modified pre-trained AlexNet and VGG16 models and further utilized a multiclass SVM classifier for classification. The authors performed experimental evaluation using two distinct validation approaches: a random 70–30 split and a leave-one-subject-out method. Their proposed SLR system attained an impressive recognition accuracy of 99.82%.
Several researchers have also been focusing on designing hardware accelerators for hand gesture and sign language recognition on FPGAs [24], [25], [26]. Zhang et al. [24] designed a CNN hardware accelerator using the Caffe framework for hand gesture classification. They used a Bilinear interpolation-based algorithm for image resizing. Their system recognizes gestures with 86.3% accuracy, but they have not mentioned the FPGA platform used for work. Moreover, their proposed CNN uses large convolution kernels (5 × 5) which increases the computational cost and power consumption of FPGA. Said et al. [25] proposed a CNN model to perform the classification of static hand gestures. The proposed system achieves 92.8% accuracy and takes around 42 ms total runtime to recognize gestures. They implemented one of the convolutional layers of CNN on the FPGA platform using fixed-point representation. But the weights of the remaining layers are still high to fit on FPGA.
Prieto et al. [26] proposed the ZynqNet-based 8-bit fixed point CNN design to measure the performance of Swedish sign language gestures on the Zynq Ultrasacle platform. A series of optimizations were also applied to achieve high throughput and fast data access. This system offers 23.5 FPS performance with a recognition rate of 80% on embedded FPGA. But the implementation of CNN tends to have large resource utilization when implemented on hardware. Authors in [27] proposed a Binary Neural Network (BNN) classifier having 1-bit weights and 1-bit activations to recognize ISL gestures. In addition, the data augmentation technique is applied to the ISL dataset to enhance the generalization capability of the BNN model. The proposed approach takes raw RGB gestures as input, performs classifications on the PYNQ-Z2 FPGA board, and achieves 85% recognition accuracy. Their BNN model drastically reduces computations but leads to a significant loss in accuracy which degrades the real-time performance of gestures.
Thus, based on the literature review outlined above, significantly less research work has been done on deploying hand gesture recognition tasks using FPGA. Furthermore, some of the existing systems described in the literature pose challenges in implementing CNN on resource-limited FPGAs. Motivated by this research gap, we propose an efficient quantized CNN with minimal power and resource consumption, while maintaining a good classification accuracy on the ISL dataset. Also, we focused our research on developing an FPGA-based system that can be beneficial for the hearing and speaking-impaired community.
Next, we summarize the significant contributions of this article:
•
An efficient quantized CNN (qCNN) based on 1-bit binary weights and 4-bit activations is proposed for static hand gesture recognition. A skin color-based hand segmentation technique is adopted to improve gesture classification accuracy. Furthermore, the performance of the quantized model is evaluated on two different Numerals ISL databases and the results are compared with other models.
•
The proposed qCNN is implemented using Xilinx FINN-based pipelined inter-layer streaming architecture. Further, the qCNN accelerator is designed and optimized by reducing BlockRAMs (BRAM). Also, a Scatter Gather (SG) mode-based Direct Memory Access (DMA) controller is implemented with a qCNN design to achieve fast memory access.
•
The synthesized results of the original and optimized qCNN design has been analyzed. Furthermore, the post-implementation results of different versions of qCNN architectures are compared and discussed.
•
The feasibility of the whole recognition system is evaluated using an optimized qCNN accelerator on the Xilinx PYNQ-Zynq Ultrascale (ZU) FPGA. A real-case demo for hand gesture recognition is also presented.
The rest of the work is structured in the following manner. Section 2 briefly explains the proposed hand gesture recognition system modules. Section 3 describes the hardware architecture of quantized CNN and optimization techniques. Section 4 reports experimental results, design space exploration, and real-time demonstration on FPGA. Finally, a summary of the conclusions is presented.
Section snippets
Proposed hand gesture recognition system description
The workflow of the proposed hand gesture recognition system is illustrated in Fig. 1. The system consists of four main functional blocks highlighted as dataset preparation, data pre-processing includes hand region segmentation using skin color filtering, basic blocks of quantized CNN architecture, and FPGA-based qCNN Accelerator. A detailed explanation of each block will be presented in the respective sections.
Hardware architecture of quantized CNN accelerator
In the preliminary section, we discuss how to quantize the model using binary weights and low-bitwidth activations using the FINN framework and implement them at the software level. Next, we implement the whole gesture recognition system using quantized CNN on embedded SoC platforms. In this section, we elaborate on FINN-based hardware design and some optimization techniques for qCNN Accelerator.
Experimental evaluation and results
The proposed qCNN model is implemented on the PYNQ-ZU FPGA to evaluate the performance of real-world gesture classification. Further, the implementation results of the optimized qCNN design are compared with the previous research works.
Conclusions
This paper presents a quantized CNN-based static gesture recognition and evaluates the feasibility of the system on resource-constrained FPGA. A simple, yet effective skin color hand segmentation technique is used to extract hand region information. The quantized CNN is designed using 1-bit weights and 4-bit activations and is implemented on FPGA with optimal resources. The proposed approach is trained on the Single Hand Indian Sign Language (SHISL) dataset, and its performance was tested by
CRediT authorship contribution statement
Mohita Jaiswal: Conceptualization, Methodology, Software, Investigation, Writing – original draft, Writing – review & editing. Vaidehi Sharma: Validation, Writing – review & editing. Abhishek Sharma: Investigation, Validation, Software. Sandeep Saini: Writing – reviewing, Supervision. Raghuvir Tomar: Supervision.
Declaration of competing interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgments
This research work was supported by the Department of Science and Technology (DST), India under the project name Sign Language to Regional Language Converter (SLRLC) with project number SEED/TIDE/063/2016.
References (40)
KumarP. et al.
A multimodal framework for sensor based sign language recognition
Neurocomputing
(2017)
WangY. et al.
An FPGA-based online reconfigurable CNN edge computing device for object detection
Microelectron. J.
(2023)
LiX. et al.
A hardware-efficient computing engine for FPGA-based deep convolutional neural network accelerator
Microelectron. J.
(2022)
ManiV. et al.
Performance comparison of CNN, QNN and BNN deep neural networks for real-time object detection using ZYNQ FPGA node
Microelectron. J.
(2022)
El-SharkawyM. et al.
Re-configurable parallel feed-forward neural network implementation using FPGA
Integration
(2024)
ShaikK.B. et al.
Comparative study of skin color detection and segmentation in HSV and ycbcr color space
Procedia Comput. Sci.
(2015)
RautarayS.S. et al.
Vision based hand gesture recognition for human computer interaction: A survey
Artif. Intell. Rev.
(2012)
CheokM.J. et al.
A review of hand gesture and sign language recognition techniques
Int. J. Mach. Learn. Cybern.
(2019)
NeethuP.S. et al.
An efficient method for human hand gesture detection and recognition using deep learning convolutional neural networks
Soft Comput.
(2020)
M.Z. Islam, M.S. Hossain, R. ul Islam, K. Andersson, Static Hand Gesture Recognition using Convolutional Neural Network...
ChungH.-Y. et al.
An efficient hand gesture recognition system based on deep CNN
BaoP. et al.
Tiny hand gesture recognition without localization via a deep convolutional network
IEEE Trans. Consum. Electron.
(2017)
AlaniA.A. et al.
Hand gesture recognition using an adapted convolutional neural network with data augmentation
RawatW. et al.
Deep convolutional neural networks for image classification: A comprehensive review
Neural Comput.
(2017)
S. Reshna, M. Jayaraju, Spotting and recognition of hand gesture for Indian sign language recognition system with skin...
SharmaP. et al.
Depth data and fusion of feature descriptors for static gesture recognition
IET Image Process.
(2020)
A. Dudhal, H. Mathkar, A. Jain, O. Kadam, M. Shirole, Hybrid SIFT Feature Extraction Approach for Indian Sign Language...
SharmaP. et al.
Deep models and optimizers for Indian sign language recognition
SharmaA. et al.
Benchmarking deep neural network approaches for Indian sign language recognition
Neural Comput. Appl.
(2021)
GangradeJ. et al.
Recognition of Indian sign language using ORB with bag of visual words by kinect sensor
IETE J. Res.
(2020)
There are more references available in the full text version of this article.