# A High-Security and Smart Interaction System Based on Hand Gesture Recognition for Internet of Things

Authors: Jun Xu; Xiong Zhang; Meng Zhou
Journal: Security and Communication Networks
 (2018)

---

## Abstract

In this work, we propose a vision-based hand gesture recognition system to provide a high-security and smart node in the application layer of Internet of Things. The system can be installed in any terminal device with a monocular camera and interact with users by recognizing pointing gestures in the captured images. The interaction information is determined by a straight line from the user’s eye to the tip of the index finger, which achieves real-time and authentic data communication. The system mainly contains two modules. The first module is an edge repair-based hand subpart segmentation algorithm which combines pictorial structures and edge information to extract hand regions from complex backgrounds. Second, the position which the user focuses on is located by an adaptive method of pointing gesture estimation, which adjusts the offsets between the target position and the calculated position due to lack of depth information.


---

## Body

## 1. Introduction

Internet of Things (IoT) which connects objects around us together to the Internet becomes one of the most important information technologies nowadays. One of the issues of IoT is how to make the object-object interaction and human-object interaction smart and safe. Data collected from users may include sensitive or private information of their daily activities; hence security protection and privacy preserving are vital for the development of IoT. According to the principle of information generation and the technical architecture, IoT consists of three layers, namely, perception [1, 2], network protocol [3, 4], and application layers [5, 6]. The perception layer is composed of various sensors and gateways, such as temperature sensor, two-dimensional barcode, camera, and Global Positioning System (GPS). The perception layer recognizes objects and collects information with Radio Frequency Identification (RFID) and Wireless Sensor Networks (WSNs); therefore, the security issue for the perception layer is essentially on node capture attacks of RFID and WSNs [7, 8]. The network protocol layer is the central pivot of IoT which transmits and processes the information from the perception layer. As one of the key technologies in the network layer, wireless mobile communication networks employ traditional encryption and authentication techniques to improve the security and privacy of IoT [9, 10]. The application layer realizes the intelligent services by connecting IoT and users, such as city management, intelligent transportation, telemedicine, and smart home. The application layer also faces the challenges of security threat. For example, the attackers can capture the unattended equipment and send data to the application system by identity impersonation and data modification. Therefore, this study is aimed at developing a secure and authentic interaction system based on computer vision for smart home in the application layer of IoT. As shown in Figure 1, our interaction system can control the cursor of a computer or a smart TV by recognizing users’ pointing gestures. The cursor is located immediately according to the line from eye to fingertip when the user points to the target. Since the interaction data is produced only when the user’s eye and fingertip are detected and the pointing direction is recognized, the system can effectively avoid unauthorized access of illegal users, which is more secure than the traditional point-touch devices.Figure 1
Applications of our interaction system based on pointing gesture recognition.Many studies have been conducted to recognize pointing gestures based on computer vision. When human interacts with an object by pointing gesture, the pointing direction is estimated by two points on the camera perspective line. According to the different input cameras, pointing gesture recognition technologies can be classified into 3D methods [11–13] and 2D methods [14–16]. 3D methods rely on specific input devices, such as stereo camera or Kinect, while 2D methods use less expensive and more easily available cameras, with reduced computation cost for 2D information. However, pointing gesture recognition based on 2D methods is still challenging. First, since 2D information of the hand provides relatively weak and ambiguous characteristics, the discrimination of bare hand from background is easily influenced by the large variability of hand appearances and the presence of other skin-like objects. Some studies have been conducted to deal with this problem [17–19]. For example, Li and Wachs [17] proposed a weighted elastic graph matching method to detect and recognize ten hand postures under complex backgrounds. The average recognition accuracy reached 97%; however the performance was unclear when hand shape distorts. In [18], hand regions were extracted based on the Bayesian model of visual attention by combining shape, texture, and color cues. The long run time, 2.65 s, of this method clearly made it unsuitable for real-time application. Gonzalez et al. [19] proposed a hand segmentation method based on pixel color and edge orientation to deal with the overlap of hand and face, whereas the segmentation results were affected by thin lines under the chin or over the collar. The second challenge is to estimate the pointing position determined by intersection between the interaction plane and the pointing vector due to lack of depth information. In order to solve this problem, some interaction systems based on 2D pointing gesture methods only utilize information of pointing direction instead of pointing position [14, 15], and some other systems require users to make coordinate calibration before operating [16]. In this paper, we propose an edge repair-based hand subpart segmentation algorithm, which accurately and effectively segments the palm and finger regions from the background by using 2D information. Furthermore, on the basis of the hand segmentation algorithm, an adaptive method of pointing direction estimation is developed which can adjust the eye-fingertip line during operation.


## 2. System Model

We develop a vision-based interaction system using pointing gesture recognition as a node in the application layer of IoT. When the user points to the screen, a straight line from the eye to the fingertip determines the position on the screen where the cursor should locate. Figure2 illustrates the flow chart of our system. First, 2D images are captured by a normal camera. Second, the user’s eye is detected by the AdaBoost classifier based on Haar-like features [20]. Third, hand region is segmented from the complex backgrounds using the edge repair-based hand subpart segmentation algorithm. Fingertip of the index finger is detected by combining convex hull and convexity defect features [21]. If both the eye and the hand are located, an adaptive method of pointing direction estimation is proposed to obtain the pointing position according to the eye-fingertip line. Finally, the cursor is moved to the pointing position. Consequently, the system is secure and authentic because it can only be activated when the eye and hand of the user are both detected.Figure 2
Flow chart of our proposed system.


### 2.1. Methods of Eye Detection and Hand Segmentation

In our system, the face position is first coarsely determined by background subtraction and skin color detection. Then connected regions with large areas are extracted. In order to verify face from these regions, an ellipse model is employed to select the approximately elliptical candidates because of the face shape [22]. For each candidate, the AdaBoost algorithm based on Haar-like features is employed for eye detection. The AdaBoost classifier is trained by the positive samples of eye images and the negative samples of all kinds of background images without human eyes. A result of eye detection is shown in Figure 2.In order to segment hand regions efficiently, we propose an edge repair-based hand subpart segmentation algorithm which includes four procedures as illustrated in Figure3.Figure 3
Architecture of the edge repair-based hand subpart segmentation algorithm.Firstly, a hierarchical chamfer matching algorithm (HCMA) [23] is used to locate the whole hand region in the binary image produced by combining skin color detection and background subtraction. As shown in Figure 4, the chamfer distance image of the hand is searched for the optimal position which matches the hand template from the previous frame. After the distance image is traversed, the optimal position is figured out by calculating the minimum edge distance Ed.(1)
Ed
=
1
3
1
N
∑
i
=
0
N
-
1
v
i
2
,where v
i is the pixel value that the template hits and N is the number of the contour pixels in the template. In order to accelerate the matching, a pyramid structure is built by halving the resolution of the distance image gradually. At the top level of the pyramid structure, a grid of positions is chosen to start the matching. Each position and its neighborhood are computed for Ed. If a smaller edge distance is found, the template is moved to the new position (
X
,
Y
) in the distance image of level n by (2)
X
=
X
r
+
2
n
-
1
2
n
Y
=
Y
r
+
2
n
-
1
2
n
,where(3)
X
r
=
c
x
+
cos
⁡
θ
·
s
x
·
x
-
sin
⁡
θ
·
s
y
·
y
Y
r
=
c
y
+
sin
⁡
θ
·
s
x
·
x
+
cos
⁡
θ
·
s
y
·
ywhere (
x
,
y
) are the coordinates of the points in the template and (
c
x
,
c
y
), (
s
x
,
s
y
), andθ are translation, scaling, and rotation parameters, respectively. For each start position, the position with local minima is obtained and then used as the start position at the next level n
-
1. When all the levels are traversed, the optimal position is finally identified, which is illustrated by the blue rectangle in Figure 4(b).Figure 4
Template matching of the whole hand by HCMA. (a) Hand segmentation by skin color detection and background subtraction. (b) Edge extraction by Canny edge detector. (c) The edge distance is computed by the pixel valuesv
i, which the hand template hits.
(a)
(b)
(c)Secondly, the located hand region is analyzed to detect the palm and fingers separately by combining pictorial structures and Histogram of Oriented Gradient (HOG) features [24]. Figure 5(a) shows the hand model based on pictorial structures, which includes a root part of the palm and five-finger parts. Thus the hand configuration is denoted by L
=
l
0
,
l
1
,
l
2
,
…
,
l
n where the subscript 0 corresponds to the palm part and the subscripts 1
~
n correspond to the finger parts. Given an image I and a set of hand model parameters θ
=
u
i
,
c
i
j, the maximum a posteriori (MAP) probability of L is represented by (4)
P
L
∣
I
,
θ
∝
P
I
∣
l
0
,
u
0
·
∏
i
=
1
n
P
I
∣
l
i
,
u
i
·
P
l
i
,
l
0
∣
c
i
0
,where u
i is the appearance parameter of parti and c
i
j is the connection parameter between part i and part j. Due to the different characteristics of the palm and finger in the model, two support vector machine (SVM) classifiers are employed to detect the subparts of the hand. On the one hand, the classifier for the palm part is trained by HOG features. On the other hand, the input feature vector of the classifier for the finger part considers both HOG features and the spatial relationship between the finger and palm. Let the state of the finger part i be l
i
(
x
i
,
y
i
,
θ
i
). Assuming (
x
j
,
y
j
) and (
x
0
,
y
0
) are the coordinates of the joint and the palm center, the relative position (
x
i
,
y
i
) is computed by(5)
x
i
=
x
j
-
x
0
w
0
y
i
=
y
j
-
y
0
w
0
,where w
0 is the size of the palm part. θ
i is the absolute part orientation as shown in Figure 5(a). Figure 5(b) shows the detection result of the image in Figure 4.Figure 5
Palm and finger detection based on pictorial structure and HOG features. (a) Hand modeling. (b) Detection result.
(a)
(b)Thirdly, since the blurry border of the hand and face probably leads to an incompletely connected hand silhouette, we propose an edge repair method to recover the contour of each subpart. The edge image of each subpart is extracted to detect where the contour breaks. An edge point is determined as a breakpoint if one or two adjacent points among its eight-neighborhood are edge points. For each breakpointP
i, a contour point Q
i in the prestored template is found, which is the closest to P
i based on Euclidean distance. As shown in Figure 6(a), in order to connect the adjacent breakpoints P
1
P
2 depending on the template, Q
1
Q
2 is divided into several subsegments by Q
k
∣
k
=
1
,
…
,
m and a set of P
k
∣
k
=
1
,
…
,
m is generated by P
k
=
P
k
-
1
+
Q
k
-
Q
k
-
1.Then, P
k
-
1 and P
k are connected by the Catmull–Rom interpolation method [25]. Figure 6(b) shows the result of edge repair where the connections of breakpoints are illustrated in red.Figure 6
Edge repair. (a) Breakpoint connection. (b) Edge repair results. (c) Refined segmentation using edge repair results. (d) Combination of all subparts.
(a)
(b)
(c)
(d)Finally, the repaired edge images of all the subparts are used to extract the refined hand pixels from the coarse binary image as shown in Figure6(c). All images of hand subparts are combined to generate the whole hand region in Figure 6(d).


### 2.2. Cursor Positioning

When both the eye and the fingertip are detected, the position on the screen where the user points to can be figured out by the line extending forward from the dominant eye to the fingertip. Since the depth information of the eyes and fingertips is unavailable and the pointing ways of different users are individually different, the offset exists between the target position and the calculated position. Thus, we propose an adaptive method of pointing direction estimation which can adjust the eye-fingertip line through a learning process.As shown in Figure7, a three-dimensional coordinate system with the camera position as the origin is established. The intersection point (
x
i
,
y
i
) of the screen and the eye-fingertip line can be calculated through similar triangle theory as (6)
x
i
=
x
E
_real
-
x
E
_real
-
x
F
_real
z
E
_real
-
z
F
_real
·
z
E
_real
y
i
=
y
E
_real
-
y
E
_real
-
y
F
_real
z
E
_real
-
z
F
_real
·
z
E
_real
,where (
x
E
_real
,
y
E
_real
,
z
E
_real
) and (
x
F
_real
,
y
F
_real
,
z
F
_real
) represent the eye’s coordinates and the fingertip’s coordinates in the 3D coordinate system, respectively. Since the relationship of those coordinates in 3D coordinate system is similar to that in the captured image (Figure 7(c)), (
x
E
_real
,
y
E
_real
,
z
E
_real
) can be computed by(7)
x
E
_real
=
z
E
_real
·
tan
⁡
θ
h
·
x
E
_image
-
a
a
y
E
_real
=
z
E
_real
·
tan
⁡
θ
v
·
y
E
_image
-
b
b
,where (
x
E
_image
,
y
E
_image
) are the coordinates of the human eye in the image, θ
h and θ
v are the camera’s angles of view in the horizontal and vertical direction, and (
a
,
b
) is half of the spatial resolution of the image. (x
F
_real
,
y
F
_
r
e
a
l
,
z
F
_real) can be figured out by a similar equation to (7).Figure 7
Cursor positioning based on the eye-fingertip line. (a) 3D coordinate system. (b)X
-
Z plane of (a). (c) The captured image with the resolution of 2
a
×
2
b.
(a)
(b)
(c)Then the cursor’s coordinates(
x
c
u
r
s
o
r
,
y
c
u
r
s
o
r
) are computed by transforming (
x
i
,
y
i
) into the screen coordinates in pixels as (8)
x
cursor
=
x
i
+
1
2
W
·
x
res
W
y
cursor
=
y
i
·
y
res
H
,where (
x
res
,
y
res
) is the spatial resolution of the screen and W and H are the width and height of the screen.It can be seen that the cursor’s position is closely related to the distance between the fingertip and the screenz
F
_real and the distance between the eye and the screen z
E
_real. However, the two distances cannot be accurately obtained from our monocular vision-based system. The inaccurate distances will lead to an offset between the calculated position and the target position of the cursor. Hence, we propose a method to adjust the cursor’s position through a learning process. Firstly, z
E
_real is estimated according to the face’s area and z
F
_real is initialized to z
E
_real minus 30 based on users’ habits. Secondly, when the cursor is not located at the desired position, the user is allowed to alter the cursor’s position by moving the fingertip slightly. The cursor’s coordinate is adjusted from (
x
cursor
,
y
cursor
) to (
x
cursor
′
,
y
cursor
′
) by (9)
x
cursor
′
=
s
x
·
d
x
+
x
cursor
y
cursor
′
=
s
y
·
d
y
+
y
cursor
,where (
d
x
,
d
y
) represents the moving distance of the fingertip in the successive frames and (s
x,s
y) is the multiple coefficient. Our system monitors the fingertip’s movement and records (
x
cursor
′
,
y
cursor
′
) when the fingertip moves a short distance after a pause. Then z
F
_real can be adjusted by (10) from z
F
_real
(
n
) to z
F
_real
(
n
+
1
).(10)
z
F
_real
n
+
1
=
1
-
α
z
F
_real
n
+
α
·
z
F
_real
′
=
1
-
α
z
F
_real
n
+
α
·
1
2
z
F
_real
x
+
z
F
_real
y
,where α is the update rate. z
F
_real
x and z
F
_real
y are estimated based on x
cursor
′ and y
cursor
′, respectively, according to the derivation of (6)–(9). After the adjusting process works several times, z
F
_real will approach the real value.


### 2.3. Feasibility Verification of Our System

In order to verify whether our cursor positioning system is feasible or not, we perform error analysis to evaluate how much the cursor error depends on the eye-hand position in the direction ofz-axis and the eye-hand position in the image. Because of the lack of depth information, the real locations of the fingertip and the eye in the direction of z-axis cannot be obtained, represented by z
′
F
_real and z
′
E
_real. Hence the estimated values of z
F
_real and z
E
_real are used in our pointing direction estimation method. Generally, the offsets of Δ
z
F
_real and Δ
z
E
_real exist between the real values and the estimated values, which lead to the errors of the cursor position. Similarly, assuming only X
-
Z plane is considered; Δx
F
_image and Δx
E
_image possibly exist due to the detection deviations of the fingertip’s and eye’s x-coordinates in the image. Note that the adjusting process of the cursor position is not activated here in order to analyze the error.As shown in Figure8, it is assumed that Δ
z
F
_real exists and the eye is fixed on z-axis; the cursor’s coordinate x
i is computed by(11)
x
i
=
z
E
_real
·
tan
⁡
θ
h
·
x
F
_image
-
a
a
·
z
F
_real
z
E
_real
-
z
F
_real
.Then the cursor error Δx
i is calculated by subtracting the deviation value x
i from the real value x
i
′ as (12)
Δ
x
i
=
z
E
_real
·
tan
⁡
θ
h
·
x
F
_image
-
a
a
·
z
F
_real
′
z
E
_real
-
z
F
_real
′
-
z
F
_real
′
-
Δ
z
F
_real
z
E
_real
-
z
F
_real
′
+
Δ
z
F
_real
,where Δ
z
F
_real = z
F
_real
′
-
z
F
_real. (
x
F
_image
-
a
)
/
a can be computed by using similar triangle principle in Figure 8. Therefore,(13)
Δ
x
i
=
x
i
′
-
z
E
_real
-
z
F
_real
′
·
x
i
′
z
F
_real
′
·
z
F
_real
′
-
Δ
z
F
_real
z
E
_real
-
z
F
_real
′
+
Δ
z
F
_real
.Figure 8
Error analysis model.Assuming the width of the screen is 31 cm, Figure9 shows the errors of the cursor with different z
F
_real
′ and z
E
_real when the user points to the edge of the screen and a quarter of the screen. It is indicated that the cursor error becomes larger when the user points to the position closer to the screen edge. The cursor error increases as z
E
_real decreases and as the hand is closer to the head. Moreover, the cursor error is acceptable when operating the laptop in a close distance less than 100 cm and the error will be the smallest when the hand locates at the midpoint of the screen and the eye.Figure 9
Cursor errors with differentz
′
F
_real and z
E
_real (a) when x
′
i
=
15.5 cm and (b) when x
′
i
=
8 cm.
(a)
(b)Figure10 illustrates how much the cursor error depends on the four offsets of Δ
z
F
_real, Δ
z
E
_real, Δx
F
_image, and Δx
E
_image using the similar derivation of (13), where the two distances from the screen denoted by z
E
_real and z
F
_real are set according to users’ operation habits. Not that the user is assumed to point to the edge of the screen, which causes the maximum error among the entire screen. As shown in Figure 10, the maximum cursor errors fall below 3 cm and 1.5 cm when the offsets of eye and hand positions on z-axis are less than 3 cm, and they fall below 3 cm when the offsets of eye and hand positions in the image are less than 9 pixels. The precision is acceptable for the block-level positioning and the following experiments will prove the attainability when the adjusting process is activated.Figure 10
Maximum cursor errors effected by the four offsets ofΔ
z
F
_real, Δ
z
E
_real, Δx
F
_image, and Δx
E
_image.
(a)
(b)
(c)
(d)


## 2.1. Methods of Eye Detection and Hand Segmentation

In our system, the face position is first coarsely determined by background subtraction and skin color detection. Then connected regions with large areas are extracted. In order to verify face from these regions, an ellipse model is employed to select the approximately elliptical candidates because of the face shape [22]. For each candidate, the AdaBoost algorithm based on Haar-like features is employed for eye detection. The AdaBoost classifier is trained by the positive samples of eye images and the negative samples of all kinds of background images without human eyes. A result of eye detection is shown in Figure 2.In order to segment hand regions efficiently, we propose an edge repair-based hand subpart segmentation algorithm which includes four procedures as illustrated in Figure3.Figure 3
Architecture of the edge repair-based hand subpart segmentation algorithm.Firstly, a hierarchical chamfer matching algorithm (HCMA) [23] is used to locate the whole hand region in the binary image produced by combining skin color detection and background subtraction. As shown in Figure 4, the chamfer distance image of the hand is searched for the optimal position which matches the hand template from the previous frame. After the distance image is traversed, the optimal position is figured out by calculating the minimum edge distance Ed.(1)
Ed
=
1
3
1
N
∑
i
=
0
N
-
1
v
i
2
,where v
i is the pixel value that the template hits and N is the number of the contour pixels in the template. In order to accelerate the matching, a pyramid structure is built by halving the resolution of the distance image gradually. At the top level of the pyramid structure, a grid of positions is chosen to start the matching. Each position and its neighborhood are computed for Ed. If a smaller edge distance is found, the template is moved to the new position (
X
,
Y
) in the distance image of level n by (2)
X
=
X
r
+
2
n
-
1
2
n
Y
=
Y
r
+
2
n
-
1
2
n
,where(3)
X
r
=
c
x
+
cos
⁡
θ
·
s
x
·
x
-
sin
⁡
θ
·
s
y
·
y
Y
r
=
c
y
+
sin
⁡
θ
·
s
x
·
x
+
cos
⁡
θ
·
s
y
·
ywhere (
x
,
y
) are the coordinates of the points in the template and (
c
x
,
c
y
), (
s
x
,
s
y
), andθ are translation, scaling, and rotation parameters, respectively. For each start position, the position with local minima is obtained and then used as the start position at the next level n
-
1. When all the levels are traversed, the optimal position is finally identified, which is illustrated by the blue rectangle in Figure 4(b).Figure 4
Template matching of the whole hand by HCMA. (a) Hand segmentation by skin color detection and background subtraction. (b) Edge extraction by Canny edge detector. (c) The edge distance is computed by the pixel valuesv
i, which the hand template hits.
(a)
(b)
(c)Secondly, the located hand region is analyzed to detect the palm and fingers separately by combining pictorial structures and Histogram of Oriented Gradient (HOG) features [24]. Figure 5(a) shows the hand model based on pictorial structures, which includes a root part of the palm and five-finger parts. Thus the hand configuration is denoted by L
=
l
0
,
l
1
,
l
2
,
…
,
l
n where the subscript 0 corresponds to the palm part and the subscripts 1
~
n correspond to the finger parts. Given an image I and a set of hand model parameters θ
=
u
i
,
c
i
j, the maximum a posteriori (MAP) probability of L is represented by (4)
P
L
∣
I
,
θ
∝
P
I
∣
l
0
,
u
0
·
∏
i
=
1
n
P
I
∣
l
i
,
u
i
·
P
l
i
,
l
0
∣
c
i
0
,where u
i is the appearance parameter of parti and c
i
j is the connection parameter between part i and part j. Due to the different characteristics of the palm and finger in the model, two support vector machine (SVM) classifiers are employed to detect the subparts of the hand. On the one hand, the classifier for the palm part is trained by HOG features. On the other hand, the input feature vector of the classifier for the finger part considers both HOG features and the spatial relationship between the finger and palm. Let the state of the finger part i be l
i
(
x
i
,
y
i
,
θ
i
). Assuming (
x
j
,
y
j
) and (
x
0
,
y
0
) are the coordinates of the joint and the palm center, the relative position (
x
i
,
y
i
) is computed by(5)
x
i
=
x
j
-
x
0
w
0
y
i
=
y
j
-
y
0
w
0
,where w
0 is the size of the palm part. θ
i is the absolute part orientation as shown in Figure 5(a). Figure 5(b) shows the detection result of the image in Figure 4.Figure 5
Palm and finger detection based on pictorial structure and HOG features. (a) Hand modeling. (b) Detection result.
(a)
(b)Thirdly, since the blurry border of the hand and face probably leads to an incompletely connected hand silhouette, we propose an edge repair method to recover the contour of each subpart. The edge image of each subpart is extracted to detect where the contour breaks. An edge point is determined as a breakpoint if one or two adjacent points among its eight-neighborhood are edge points. For each breakpointP
i, a contour point Q
i in the prestored template is found, which is the closest to P
i based on Euclidean distance. As shown in Figure 6(a), in order to connect the adjacent breakpoints P
1
P
2 depending on the template, Q
1
Q
2 is divided into several subsegments by Q
k
∣
k
=
1
,
…
,
m and a set of P
k
∣
k
=
1
,
…
,
m is generated by P
k
=
P
k
-
1
+
Q
k
-
Q
k
-
1.Then, P
k
-
1 and P
k are connected by the Catmull–Rom interpolation method [25]. Figure 6(b) shows the result of edge repair where the connections of breakpoints are illustrated in red.Figure 6
Edge repair. (a) Breakpoint connection. (b) Edge repair results. (c) Refined segmentation using edge repair results. (d) Combination of all subparts.
(a)
(b)
(c)
(d)Finally, the repaired edge images of all the subparts are used to extract the refined hand pixels from the coarse binary image as shown in Figure6(c). All images of hand subparts are combined to generate the whole hand region in Figure 6(d).


## 2.2. Cursor Positioning

When both the eye and the fingertip are detected, the position on the screen where the user points to can be figured out by the line extending forward from the dominant eye to the fingertip. Since the depth information of the eyes and fingertips is unavailable and the pointing ways of different users are individually different, the offset exists between the target position and the calculated position. Thus, we propose an adaptive method of pointing direction estimation which can adjust the eye-fingertip line through a learning process.As shown in Figure7, a three-dimensional coordinate system with the camera position as the origin is established. The intersection point (
x
i
,
y
i
) of the screen and the eye-fingertip line can be calculated through similar triangle theory as (6)
x
i
=
x
E
_real
-
x
E
_real
-
x
F
_real
z
E
_real
-
z
F
_real
·
z
E
_real
y
i
=
y
E
_real
-
y
E
_real
-
y
F
_real
z
E
_real
-
z
F
_real
·
z
E
_real
,where (
x
E
_real
,
y
E
_real
,
z
E
_real
) and (
x
F
_real
,
y
F
_real
,
z
F
_real
) represent the eye’s coordinates and the fingertip’s coordinates in the 3D coordinate system, respectively. Since the relationship of those coordinates in 3D coordinate system is similar to that in the captured image (Figure 7(c)), (
x
E
_real
,
y
E
_real
,
z
E
_real
) can be computed by(7)
x
E
_real
=
z
E
_real
·
tan
⁡
θ
h
·
x
E
_image
-
a
a
y
E
_real
=
z
E
_real
·
tan
⁡
θ
v
·
y
E
_image
-
b
b
,where (
x
E
_image
,
y
E
_image
) are the coordinates of the human eye in the image, θ
h and θ
v are the camera’s angles of view in the horizontal and vertical direction, and (
a
,
b
) is half of the spatial resolution of the image. (x
F
_real
,
y
F
_
r
e
a
l
,
z
F
_real) can be figured out by a similar equation to (7).Figure 7
Cursor positioning based on the eye-fingertip line. (a) 3D coordinate system. (b)X
-
Z plane of (a). (c) The captured image with the resolution of 2
a
×
2
b.
(a)
(b)
(c)Then the cursor’s coordinates(
x
c
u
r
s
o
r
,
y
c
u
r
s
o
r
) are computed by transforming (
x
i
,
y
i
) into the screen coordinates in pixels as (8)
x
cursor
=
x
i
+
1
2
W
·
x
res
W
y
cursor
=
y
i
·
y
res
H
,where (
x
res
,
y
res
) is the spatial resolution of the screen and W and H are the width and height of the screen.It can be seen that the cursor’s position is closely related to the distance between the fingertip and the screenz
F
_real and the distance between the eye and the screen z
E
_real. However, the two distances cannot be accurately obtained from our monocular vision-based system. The inaccurate distances will lead to an offset between the calculated position and the target position of the cursor. Hence, we propose a method to adjust the cursor’s position through a learning process. Firstly, z
E
_real is estimated according to the face’s area and z
F
_real is initialized to z
E
_real minus 30 based on users’ habits. Secondly, when the cursor is not located at the desired position, the user is allowed to alter the cursor’s position by moving the fingertip slightly. The cursor’s coordinate is adjusted from (
x
cursor
,
y
cursor
) to (
x
cursor
′
,
y
cursor
′
) by (9)
x
cursor
′
=
s
x
·
d
x
+
x
cursor
y
cursor
′
=
s
y
·
d
y
+
y
cursor
,where (
d
x
,
d
y
) represents the moving distance of the fingertip in the successive frames and (s
x,s
y) is the multiple coefficient. Our system monitors the fingertip’s movement and records (
x
cursor
′
,
y
cursor
′
) when the fingertip moves a short distance after a pause. Then z
F
_real can be adjusted by (10) from z
F
_real
(
n
) to z
F
_real
(
n
+
1
).(10)
z
F
_real
n
+
1
=
1
-
α
z
F
_real
n
+
α
·
z
F
_real
′
=
1
-
α
z
F
_real
n
+
α
·
1
2
z
F
_real
x
+
z
F
_real
y
,where α is the update rate. z
F
_real
x and z
F
_real
y are estimated based on x
cursor
′ and y
cursor
′, respectively, according to the derivation of (6)–(9). After the adjusting process works several times, z
F
_real will approach the real value.


## 2.3. Feasibility Verification of Our System

In order to verify whether our cursor positioning system is feasible or not, we perform error analysis to evaluate how much the cursor error depends on the eye-hand position in the direction ofz-axis and the eye-hand position in the image. Because of the lack of depth information, the real locations of the fingertip and the eye in the direction of z-axis cannot be obtained, represented by z
′
F
_real and z
′
E
_real. Hence the estimated values of z
F
_real and z
E
_real are used in our pointing direction estimation method. Generally, the offsets of Δ
z
F
_real and Δ
z
E
_real exist between the real values and the estimated values, which lead to the errors of the cursor position. Similarly, assuming only X
-
Z plane is considered; Δx
F
_image and Δx
E
_image possibly exist due to the detection deviations of the fingertip’s and eye’s x-coordinates in the image. Note that the adjusting process of the cursor position is not activated here in order to analyze the error.As shown in Figure8, it is assumed that Δ
z
F
_real exists and the eye is fixed on z-axis; the cursor’s coordinate x
i is computed by(11)
x
i
=
z
E
_real
·
tan
⁡
θ
h
·
x
F
_image
-
a
a
·
z
F
_real
z
E
_real
-
z
F
_real
.Then the cursor error Δx
i is calculated by subtracting the deviation value x
i from the real value x
i
′ as (12)
Δ
x
i
=
z
E
_real
·
tan
⁡
θ
h
·
x
F
_image
-
a
a
·
z
F
_real
′
z
E
_real
-
z
F
_real
′
-
z
F
_real
′
-
Δ
z
F
_real
z
E
_real
-
z
F
_real
′
+
Δ
z
F
_real
,where Δ
z
F
_real = z
F
_real
′
-
z
F
_real. (
x
F
_image
-
a
)
/
a can be computed by using similar triangle principle in Figure 8. Therefore,(13)
Δ
x
i
=
x
i
′
-
z
E
_real
-
z
F
_real
′
·
x
i
′
z
F
_real
′
·
z
F
_real
′
-
Δ
z
F
_real
z
E
_real
-
z
F
_real
′
+
Δ
z
F
_real
.Figure 8
Error analysis model.Assuming the width of the screen is 31 cm, Figure9 shows the errors of the cursor with different z
F
_real
′ and z
E
_real when the user points to the edge of the screen and a quarter of the screen. It is indicated that the cursor error becomes larger when the user points to the position closer to the screen edge. The cursor error increases as z
E
_real decreases and as the hand is closer to the head. Moreover, the cursor error is acceptable when operating the laptop in a close distance less than 100 cm and the error will be the smallest when the hand locates at the midpoint of the screen and the eye.Figure 9
Cursor errors with differentz
′
F
_real and z
E
_real (a) when x
′
i
=
15.5 cm and (b) when x
′
i
=
8 cm.
(a)
(b)Figure10 illustrates how much the cursor error depends on the four offsets of Δ
z
F
_real, Δ
z
E
_real, Δx
F
_image, and Δx
E
_image using the similar derivation of (13), where the two distances from the screen denoted by z
E
_real and z
F
_real are set according to users’ operation habits. Not that the user is assumed to point to the edge of the screen, which causes the maximum error among the entire screen. As shown in Figure 10, the maximum cursor errors fall below 3 cm and 1.5 cm when the offsets of eye and hand positions on z-axis are less than 3 cm, and they fall below 3 cm when the offsets of eye and hand positions in the image are less than 9 pixels. The precision is acceptable for the block-level positioning and the following experiments will prove the attainability when the adjusting process is activated.Figure 10
Maximum cursor errors effected by the four offsets ofΔ
z
F
_real, Δ
z
E
_real, Δx
F
_image, and Δx
E
_image.
(a)
(b)
(c)
(d)


## 3. Experimental Results

Our proposed system was evaluated on a laptop with a 2D camera and the resolutions of the screen and the captured image were1366
∗
768 and 320
∗
240, respectively. Besides, on the basis of the proposed methods mentioned above, the user was not permitted to keep shaking his/her body back and forth which would affect the adjusting process of cursor positioning. Ten subjects were asked to operate the system by pointing gesture.Firstly, some segmentation results of the edge repair-based hand subpart segmentation algorithm are shown in Figure11, where yellow lines indicate the hand regions. It demonstrates that our method can extract hand pixels accurately under different complex backgrounds, including human faces with similar color. Moreover, the method is independent of users and robust to various hand appearances.Figure 11
Segmentation results of the edge repair-based hand subpart segmentation algorithm under different situations.Then in order to evaluate the accuracy of cursor positioning, the computer screen was divided into several blocks as shown in Figure12 and the subjects pointed to the four blocks marked with “1” to “4” repeatedly. Each block was pointed 15 times as a set of data.Figure 12
Qualitative experimental results of cursor positioning where black circles highlight the cursor’s positions.Figure12 shows the qualitative experimental results of cursor positioning by different subjects under different backgrounds. In each row, the same subject points to the different positions of the screen and the relative positions between the fingertip and the eye appear different. The cursors highlighted by the black circles are successfully located at the corresponding marked blocks. It is demonstrated that our cursor positioning method is robust to diverse individuals and different situations. Besides, it is also implied that our hand segmentation method works well when hand overlaps the face.Let the length of the blocks be 1. When the subject points to a block, the cursor errord between the calculated position and the desired position is computed by Euclidean distance. Figure 13 shows the errors of several sets of data. The horizontal axis represents the repeat times when the users point to the blocks. It is proved that the error decreases significantly after several times owing to the adjusting process in the adaptive pointing direction estimation method. Moreover, Table 1 shows the average errors of cursor positioning for the four marked blocks. The errors d
′ are estimated by d
′
=
d
×
S
1
/
2, where S represents the area of the block. Because the size of block icon in Windows 8/10 Metro can be set as 3 cm or 6 cm, it can be concluded that the positioning errors are small and acceptable for our application requirement.Table 1
Average errors of cursor positioning for the four marked blocks.
Error
d
d’(cm)
Block 1
0.707
2.38
Block 2
0.641
2.16
Block 3
0.680
2.29
Block 4
0.536
1.80
Average
0.641
2.16Figure 13
Cursor errors with the repeat times as the horizontal axis.Besides, our system can work in real time with an average speed of 131 ms per frame.


## 4. Conclusion

A security and smart Internet of Things interaction system based on hand gesture recognition is proposed in this work. When a user points to screen, the target position which the user points to is estimated by a straight line from the user’s eye to the fingertip. Therefore, the interaction between human and computer should be activated by the coexisting of eye and hand. In our system, we employ a novel hand segmentation algorithm which combines the pictorial structure model, hierarchical chamfer matching algorithm, and curve fitting that segments hand regions accurately and efficiently. Furthermore, we propose an adaptive pointing direction estimation method for cursor calibration. An adjusting process is presented to correct the offsets between the target position and the calculated position arising from diverse individuals and lack of depth information. Experimental results show that our system provides a natural and friendly human-computer interaction and possesses satisfactory and accuracy of cursor positioning under complex backgrounds.



---

*Source: 4879496-2018-06-06.xml*