# The Manipulation of Real-Time Kinect-Based Robotic Arm Using Double-Hand Gestures

Authors: Chern-Sheng Lin; Pei-Chi Chen; Yu-Ching Pan; Che-Ming Chang; Kuo-Liang Huang
Journal: Journal of Sensors
 (2020)

---

## Abstract

This study focused on utilizing the Kinect depth sensor to track double-hand gestures and control a real-time robotic arm. The control system is mainly composed of the microprocessor, a color camera, the depth sensor, and the robotic arm. The Kinect depth sensor was used to take photos of the human body to analyze the skeleton of a human body and obtain the relevant information. Such information was used to identify the gestures of the left hand and the left palm of the user. The gesture of left hand was used as an input command device. The gesture of the right hand was used for imitation movement teaching of robotic arm. From the depth sensor, the real-time images of the human body and the deep information of each joint were collected and converted to the relative positions of the robotic arm. Combining forward kinematics and inverse kinematics and D-H link, the gesture information of the right hand was calculated, which was converted via coordinates into each angle of the motor of the robotic arm. From the color camera, when the left palm was not detected, the user could simply use the right hand to control the action and movement of the real-time robotic arm. When the left palm was detected and 5 fingertips were identified, it meant the start of recording the real-time imitation movement of the robotic arm by the right hand. When 0 fingertip was identified, it meant the stoppage of the above recording. When 2 fingertips were identified, the user could not only control the real-time robotic arm but also repeat the recorded actions.


---

## Body

## 1. Introduction

The depth sensing techniques have been widely used in the field of virtual reality. A normal measurement system consists of two cameras can obtain the depth values at two different information must be identical because the two cameras look at the same point on the object being measured [1]. The depth sensor can incorporate image processing technology to achieve user and system positioning. The gesture control function and command system enable users to operate the system without wearing complicated and heavy devices [2, 3]. When multiple sensors and cameras were used for positioning, these cameras were set in a sequential group to reduce the complexity of the adjustments and locate the target object at any time [4–6].The main purpose of this study was to combine a machine vision system, algorithm, and robotic arm to develop an interface system for real-time interactive operation among environment, man, and machine. It also combined image processing technology so as to achieve feasible real-time interactive effects between the system and its user. The machine vision system allows a robotic arm to perceive the external environment. Users can thus employ a robotic arm to perform manual labor in a harsh environment.First of all, through the Kinect sensor, the depth, width, and color of the user in the 3D space were captured. OpenCV and Visual C++ were adopted to establish the human skeleton data and analyze the deep information on the user’s joints. The information on the joints of the right hand was converted into the relative coordinate system of the robotic arm. In combination with inverse kinematics and D-H link, the angle of rotation required for each joint of the robotic arm was calculated, which was transferred to the Arduino controller for the real-time operation of the robotic arm by the user’s right hand. In terms of the recognition of the user’s left-hand gestures, the Kinect sensor was used to track the human skeleton so as to determine the positions of the hand. According to the location of the hand, the adjacent depth was set, and the area of the hand was captured. Through OpenCV, the outline of the hand was identified, followed by the convex and concave points of the hand. Such convex and concave points were then analyzed to determine the left-hand gestures.Recently, there are many related researches on object recognition and Kinect sensing devices of different visual systems. For examples, FEMD (Finger-Earth Mover’s Distance) is an algorithm specially designed for the shape of the hand. This method mainly removes the palm part and processes the end of the finger. At this time, Kinect’s sensor can get better hand shape [7]. Shum et al. focused on image reconstruction when the human body is obscured. They used Kinect to recognize human posture. Based on the reliability assessment, it performed an action database query [8]. Chang et al. performed their experiments on three public Kinect-based entertainment data sets related to fitness and gaming [9]. Lloret et al. present a wireless sensor network where each sensor node takes images from the field and internally uses image processing techniques to detect any unusual status in the leaves [10]. A sequential flow of moving object detection can track and identify the behavior completes the processing framework of video surveillance [11].This paper used the Kinect sensor to extract the gestures and depth information of the current user. Through image processing and tracking algorithm, the coordinate positions of the hands and joints of the current user were calculated. Through gestures with image separation and template matching [12, 13], the left palm was identified. The gesture of left hand was used as an input command device. The gesture of the right hand was used for imitation movement teaching of robotic arm. When the left palm was detected and 5 was identified, it meant the start of recording the real-time control of the robotic arm by the right hand. When 0 was identified, it meant the stoppage of the above recording. The right palm could record the real-time control of the robotic arm by the right hand. The user could not only control the real-time robotic arm but also repeat the recorded actions. The gestures of the right hand were then tracked (Figure 1). Through inverse kinematics, the coordinate positions of the joints of the right hand were estimated [14]. Such positions were converted to the coordinate system relative to the positions of the robotic arm. Inverse kinematics was used to estimate the angle of rotation of each joint of the robotic arm required for the motor. The Arduino controller was then used to control the motor on each joint so as to achieve real-time manipulation of the robotic arm by the user.Figure 1
The control process and the main structure of this study.


## 2. The Multiple Sensing Process

This paper used the Kinect sensor to capture the coordinate position of each joint of the user; the paper converted the coordinate system of the user into that of the AL5D robotic arm through a proportional conversion relationship. The control system is mainly composed of the microprocessor, a color camera, the depth sensor, and the robotic arm.From the depth sensor, the real-time images of the human body and the deep information of each joint were collected and converted to the relative positions of the robotic arm. For example, in consideration of the original point of the robotic arm and the coordinate axis directions, the distance between the original point of the arm and the shoulder wasd1=7cm. The distance between the shoulder and the elbow was a2=14.605cm, that between the elbow and the wrist was a3=18.7325cm, and that between the wrist and the end of the palms was a4=10.0076cm. As the coordinates of the shoulder of AL5D robotic arm were not to be changed, the coordinates x, y, and z of the shoulder were set as 0, 0, and 7, respectively. The coordinates of the elbow and arms would change along with the different gestures of the user (Figure 2).Figure 2
The coordinates of the joints of the upper body detected by the Kinect sensor.Kinect was used to measure the coordinates of the shoulder (x1′,y1′,z1′) and arms (x2′,y2′,z2′). The length of the link between the shoulder and elbow was calculated with the following formula:
(1)D1=x2′−x1′2+y2′−y1′2+z2′−z1′2.The length of the link between the shoulder and elbow was 7 cm. Hence, the proportional relationshipa2/D1 between the lengths of the two links was as follows:
(2)a2D1=7x2′−x1′2+y2′−y1′2+z2′−z1′2.In order to calculate the coordinate changes from the shoulder to elbow of the AL5D robotic arm, this paper sets the coordinate changes of the shoulder and elbow as (x2′−x1′,y2′−y1′,z2′−z1′), which were then multiplied with the proportional relationship between the shoulder and elbow of the AL5D robotic arm. The result was the coordinate changes from the shoulder to elbow of AL5D robotic arm:
(3)a2D1∙x2′−x1′,y2′−y1′,z2′−z1′.The coordinates of the elbow of the AL5D robotic arm were set to (0, 0, 7), while those of the elbow were (x2,y2,z2). The coordinates of the shoulder of the users plus the coordinates changes from the shoulder to elbow of AL5D robotic arm resulted in the coordinates of the elbow of AL5D robotic arm (x2,y2,z2), as follows:
(4)x2,y2,z2=0,0,7+a2D1∙x2′−x1′,y2′−y1′,z2′−z1′.After the coordinates (x2,y2,z2) of the elbow of AL5D robotic arm were calculated, for the purpose of the inverse kinematics calculation later, the coordinates of the wrist should be obtained. Thus, the elbow coordinates (x3,y3,z3) of the AL5D robotic arm should be calculated.Set the wrist coordinates of the user as (x2′,y2′,z2′) and the elbow coordinates as (x3′,y3′,z3′). The length of the link between the elbow and wrist was calculated as follows:
(5)D2=x3′−x2′2+y3′−y2′2+z3′−z2′2.The length of the link between the shoulder and elbow of the robotic arm was 18.7325 cm. Hence, the proportional relationshipa3/D2 between the lengths of the two links was as follows:
(6)a3D2=18.7325x3′−x2′2+y3′−y2′2+z3′−z2′2.In order to calculate the coordinate changes from the elbow to wrist of the AL5D robotic arm, this paper sets the coordinate changes of the elbow and wrist of the user as (x3′−x2′,y3′−y2′,z3′−z2′), which were then multiplied with the proportional relationship between the elbow and wrist of the AL5D robotic arm. The result was the coordinate changes from the elbow to wrist of the AL5D robotic arm:
(7)a3D2∙x3′−x2′,y3′−y2′,z3′−z2′.The coordinates of the elbow of the AL5D robotic arm were set to (x2,y2,z2), while those of the wrist were (x3,y3,z3). The coordinates of the elbow of the users (x2,y2,z2) plus the coordinates changes from the elbow to wrist of the AL5D robotic arm resulted in the coordinates of the wrist of AL5D robotic arm (x3,y3,z3), as follows:
(8)x3,y3,z3=x2,y2,z2+a3D2∙x3′−x2′,y3′−y2′,z3′−z2′.After the coordinates (x3,y3,z3) of the elbow of the AL5D robotic arm were calculated, for the purpose of the inverse kinematics calculation later, the coordinates of the wrist should be obtained. Thus, the coordinates (x4,y4,z4) of the end of the palm of the AL5D robotic arm should be calculated.Set the wrist coordinates of the user as (x3′,y3′,z3′) and the coordinates of the end of palm as (x4′,y4′,z4′). The length D3 of the link between the wrist and end of palm was calculated as follows:
(9)D3=x4′−x3′2+y4′−y3′2+z4′−z3′2.The length of the link between the wrist and the end of the palm of the AL5D robotic arm was 10.0076 cm. Hence, the proportional relationshipa4/D3 between the lengths of the two links was as follows:
(10)a4D3=10.0076x4′−x3′+y4′−y3′+z4′−z3′.In order to calculate the coordinate changes from the wrist to the end of the palm of the AL5D robotic arm, this paper sets the coordinate changes of the wrist and the end of the palm of the user as (x4′−x3′,y4′−y3′,z4′−z3′), which were then multiplied with the proportional relationship between the wrist and the end of the palm of the AL5D robotic arm. The result was the coordinate changes from the wrist to the end of the palm of the AL5D robotic arm:
(11)a4D3∙x4′−x3′,y4′−y3′,z4′−z3′.The coordinates of the wrist of the AL5D robotic arm were set to (x3,y3,z3), while those of the end of the palm were (x4,y4,z4). The coordinates of the wrist of the users (x3,y3,z3) plus the coordinates changes from the wrist to the end of the palm of the AL5D robotic arm resulted in the coordinates of the end of the palm of the AL5D robotic arm (x4,y4,z4), as follows:
(12)x4,y4,z4=x3,y3,z3+a4D3∙x4′−x3′,y4′−y3′,z4′−z3′.In terms of direct kinematics, the Denavit-Hartenberg (D-H) rule is widely used to define a robotic arm (Figure3). It originally took six parameters (horizontal movement along and the rotation around the three axes) to describe the motion or location relationship of a robotic arm. With the D-H coordinate system, only four are required, which simplifies mathematical operations and makes the D-H rule a standard to describe the kinematics of a robot.Figure 3
Denavit-Hartenberg representation.This paper used the Denavit-Hartenberg representation proposed in 1955 to set up the coordinate system. The coordinate system of each axis joint was defined according to the following rules. The parameters of the four links [7, 8] and the definitions and diagrams of each link were also obtained.
(i)
Link lengthai: calculate the distance from Zi−1 to Zi along Xi
(ii)
Link twistαi: calculate the distance from Zi−1 to Zi along Xi
(iii)
Link offsetdi: calculate the distance from Xi−1 to Xi along Zi−1
(iv)
Joint angleθi: calculate the distance from Xi−1 to Xi along Zi−1The final result of the transformation matrixTi(i−1) can be obtained from the transformation matrix:
(13)Tii−1=Transz,diRotz,θiTransx,aRotx,α,Tii−1=cosθi−sinθicosαisinθisinαiαicosθisinθcosθicosαi−cosθisinαiαisinθi0sinαicosαidi0001.Denavit-Hartenberg (D-H) conversion matrix [9].From the color camera, the fingertips of the image of two palms can be detected with convex set method. In a real vector spaceV, for the given set X, all the intersection S containing the convex set X were named the convex hull of X. The set mathematical notation is Equation (14) [15–17].
(14)S≔⋂X⊆K⊆VKisconvexK.The definition of “convex” is the link between any two points in an image does not pass through the external part of the image. “Convex” does not refer to the bulged surface. In fact, convex hull is made up of many flat surfaces [18, 19]. The convex hull algorithm was used for fingertip detection. The concept of the algorithm lies in assuming a point was set in a 2D surface; all the peripheral points were connected to one point to cover the point set. If there were overlapping of several points in the convex hull, in the context of multipoint and commonline, then generally the least points would be used to judge the result, i.e., the minimum area [18, 20].The convex hull ofX could be constructed by the linear combination of all the points (X1,⋯,Xn) in X [21, 22].
(15)S≔∑j=1ntjxj∣xj∈X,∑j=1ntj=1,∈0,1.This study used the convex hull algorithm to identify the left-hand gestures so that Kinect could determine the left-hand gesture mode. First, Kinect was used to track the human skeleton and the left-hand joints and search between the two joints of HandLeft and HandTipLeft. The depth outside the scope was excluded so as to find the contour of the hand. As the convex hull algorithm could effectively find the five points corresponding to fingertips, the fingertips were the edges of all the contour polyline, and the edges were the fingers. The convex hull algorithm could be employed to analyze these points and then determine the left-hand gestures.First up was the identification of the left-hand gestures. The Kinect sensor was adopted to track the human skeleton [23–25]. Second, the left-hand joints were detected. Third, the search area was specified. When the left-hand joints were detected, search was made within a reasonable scope of the HandLeft joints. The reasonable search scope was restricted between the joints of HandLeft and HandTipLeft.Fourth, find the contour of the hands. As Step 3 strictly defined the search area in the 3D space, any depth values outside the scope could be excluded. Thus, all the depth values outside the hands would be rejected. Thus, the shape of a hand could be obtained, and the contour of the shape was that of the hand.Lastly, find the convex hull. The contour of the hand was a big set. However, only five points (or less) corresponded to the tip. The tip was the edge including all the contour polyline. With the above steps, the left hand could be tracked, as shown in Figure4. Next, these points could be analyzed to determine hand gestures.Figure 4
Tracking both hands.The flowchart to track the right-hand gestures was next. First, the Kinect sensor was adopted to obtain color and deep image data. Next, the location of the human body was detected, and the skeleton of the right hand was detected and tracked so as to find the joints of the right hand. Finally, it was the conversion of coordinates. The coordinates of the right hand of the user were converted into those of the robotic arm. The D-H rule was used to deduce direct kinematics and obtain the coordinates of the end effector (px,py,pz), which could be used to deduce inverse kinematics. With the known absolute coordinates of each link, the θ1, θ2, θ3, and θ4 positions of each joint were obtained, which were transmitted to the Arduino controller to drive the robotic arm.


## 3. Experimental Results

This paper identified five gestures of the left hand. The gesture identification included 5, 0, 2, gestures not detected, and unknown gestures, as shown in Figure5.Figure 5
Left hand was identified as 5.Because of a mirror reflection relation, the following figure shows the right hand. The main requirement here is to take the position of the robot arm end point in the Cartesian coordinate system (x,y,z) and the angle (θ1,θ2,θ3) of the joint coordinate system of the robot arm Conversion relationship between. Calculate the position of the end point of the arm in the Cardison coordinate system by using the known movement angles (θ1,θ2,θ3) of each axis joint, and then infer each position from the known position of the end point of the arm in the Cardison coordinate system. It can be calculated how many degrees does the shaft joint need to move to reach this position. Mapping from the end point of the robot arm to the movement angle of each axis is inverse kinematics, while mapping from the movement angle of each axis to the end point of the robot arm is forward kinematics. When the left gesture was detected and 5 was identified, it meant the start of the recording the real-time control of the robotic arm by the right hand. When 0 was identified, it meant the stoppage of the above recording. When 2 was identified, the robotic arm repeated the motion twice from the start to the end of the record. When the left-hand gesture was not detected, the right hand could manipulate the real-time robotic arm (Figure 6). When unknown gestures were detected, the robotic arm had no action.Figure 6
(a) The user used her right hand to manipulate the robotic arm. (b) The user pushed her hand right and forward. (c) The user pushed her right hand upward and bent her wrist downward. (d) The user bent her right hand left slightly and bent her wrist upward.
(a)
(b)
(c)
(d)The tracking of the right-hand gestures was for the real-time manipulation of the robotic arm. Due to the mirror, the following figure shows the left hand. The Kinect sensor was used to obtain the information on the skeleton of the right hand. Through the conversion of coordinates and inverse kinematic operation, the angle of rotation required for each joint of the robotic arm was obtained (Figure7). The angles were then transmitted to Arduino, which drove the robotic arm.Figure 7
The relationship of coordinates and inverse kinematic operationθ1θ2θ3θ4= (a) 0000 and (b) 0−π/2π/2π/2.
(a)
(b)Table1 shows the experimental data on the identification of the left-hand gestures from different distances (100 cm, 150 cm, 200 cm) and different locations and directions so as to reach accuracy.Table 1
Experimental data on the identification of the left-hand gestures.
Quantity of samples
Undetected
Unknown
Count of failures
Accuracy
Gesture was 5
150
0
4
4
97.33%
Gesture was 0
150
0
0
0
100%
Gesture was 2
150
0
3
3
98%Each joint of the user’s right hand stretched to 90 degrees. Each motor measured the actual movement angle of the robotic arm for three times. Here, the forward and reverse kinematics are used to calculate and verify the angle of the joint motor of each position of the robot arm. Let the user’s right hand move 90 degrees in each joint and measure the actual movement angle of the robot arm three times in each motor (Table2).Table 2
The tracking accuracy of the robot.
Joint
Actual angle of robot arm movement (degrees)
1
104
102
99
2
99
103
102
3
98
98
102
4
92
96
91It can be known from Table2 that although the actual movement angles have mutual errors, they can basically operate according to the original instructions.


## 4. Discussion

There were errors in the actual moving angles; basically, the robotic arm could follow the instructions to conduct direct and inverse kinematic calculation. Moreover, the angle of joint motor of each gesture was verified. The gesture information of the right hand was calculated, which was converted via coordinates into each angle of the motor of the robotic arm. Lastly, the program transmitted the information to microprocessor, which further transmitted it to the corresponding motor of robotic arm, so that the user could control the real-time robotic arm.The Kinect sensor was employed to identify and track the gestures of both hands. Five gestures of the left hand were identified, including 5, 0, 2, gestures undetected, and unknown gestures. When the left-hand gesture was not detected, the right hand could manipulate the real-time robotic arm. When the left gesture was detected and 5 was identified, it meant the start of the recording the real-time control of the robotic arm by the right hand. When 0 was identified, it meant the stoppage of the above recording. When 2 was identified, the robotic arm repeated the recording twice.


## 5. Conclusions

The main purpose of this study was to combine a machine vision system, algorithm, and robotic arm to develop an interface system for real-time interactive operation among environment, man, and machine. It also combined image processing technology so as to achieve feasible real-time interactive effects between the system and its user. The machine vision system allows a robotic arm to perceive the external environment. Users can use a robotic arm to perform manual labor in a harsh environment.This study tracked the gestures of both hands to realize real-time manipulation of a robotic arm. The Kinect sensor was used to detect the gesture and keep information of the current user. The user could control the actions of the robotic arm though the current images. When the user moved, the D-H rule and the direct and inverse kinematics allowed the robotic arm to perform the same action as that of the user, resulting in the effect of interaction.In this paper, gesture recognition and tracking of both hands are performed through a Kinect sensor. Left-hand gesture recognition can recognize five gestures, which are 5, 0, 2, no gesture detected, and unknown gesture. Through the recognition of left-hand gestures, when no gesture is detected, the right hand can control the AL5D robotic arm in real time; when a gesture is detected as 5, it starts to record the trajectory of the right-handed real-time control robotic arm; then stop recording; when the gesture detection is 2, the robot will repeat the recorded track twice.The color and depth data of the user are mainly obtained through the Kinect sensor to analyze the posture of the left and right hands. The left hand is mainly used to determine the mode that the robot arm must perform. By operating in different modes, the robot arm functions more effectively. It is versatile. The right hand is mainly used to operate the robotic arm. Kinect obtains the user’s right hand motion trajectory and substitutes inverse kinematics to obtain the angle of rotation of each axis motor, which has reached the task of driving the robotic arm.In terms of future development, more gesture recognition can be performed for the left hand, so that the left hand can use more modes to control the robotic arm. At the same time, a camera can be installed on the robotic arm. With the assistance of images, the fuselage can obtain environmental information, analyze its environment, and make future operations more humane.



---

*Source: 9270829-2020-01-25.xml*