## 2 Experimental Setup
### Gesture Recognition and Object Selection  
Based on previous research [34; 35], the following sections list a set of typical gestures. Figure 4 shows the finger joint changes with time. In an actual process, only gestures G1-G4 are needed. The robot does not need to respond to other gestures, but considering the expansion of the function in the future, we expanded them to G1-G9. The joint angle of each gesture is maintained for 30 s, as shown in Figure 5A.  
The hidden layer from input to output. Row \(H\) is equal to row \(W\) multiplied by column \(x\):  
\[H_{i}=\sum\limits_{i=1}^{d}W_{yi}x_{i}+b_{i} \tag{1}\]  
From the hidden layer to output layer:  
\[O_{i}=\sum\limits_{i=1}^{d}W_{yi}x_{i}+b_{i} \tag{2}\]  
Tansig is used to increase the nonlinearity:  
\[T(x)=\frac{2}{1+\exp^{-2x}}-1 \tag{3}\]  
The normalized probability is obtained by using the softmax function:  
\[P(G|x)=\frac{\exp(W_{G}\cdot x)}{\sum\limits_{c=1}^{C}\exp(W_{c}\cdot x)} \tag{4}\]  
Gesture recognition [36] can be based in neural networks with two hidden layers, a Tansig activation function, and softmax classification. It can be seen that the eigenvalues of these nine kinds of algorithms are very obvious, and can recognize each gesture with 100% accuracy (Figure 5B).  
Figure 4: Bending angles of the finger relative to the palm.  
The user's focus does not always stay on one point but is changing constantly. Only useful information is used for higher-level applications of the gaze point. Five algorithms for measuring fixations and saccades, namely, I-VT, I-HMM, I-DT, I-MST, and I-AOI, were described by [37]. This paper used an algorithm similar to I-DT for object selection. The specific idea is to solve each distance between the object and the ray point. The object is selected when the eye and hand are close to the object and maintained within 200 ms of the boundary, as shown in Figure 6. Knowing the accuracy of eye and hand tracking is helpful for boundary determination. [PERSON] [38] removed individuals that are more than two standard deviations away from the average. For objects 1-9 (O1 to O9), we collected 3000 intersection points for each. The human eyes were 40 cm away from the computer screen. For O1, the average intersection point was 0.27 cm to the right and  
Figure 5: (**A**) Finger bending angles in gestures G1–G9. (**B**) The Recognition results for each gesture.  
0.56 cm below the target. The overall offset between the interaction and target is shown in Table 1 (comparing 40 cm and 60 cm distances to the screen).  
Similarly, we measured the jitter of the hand's indicator points (Table 2). We can see that the stability of the finger indicator points is better than that of the eye tracker. However, in experience, the stability of the hand is required to be higher. The reason is that the eye tracker is calibrated for each person and directly displays the center of the line of sight, which is equivalent to an open loop. The hand indicates points, which is only based on the subjective feeling of people. The sensor recognizes the general direction, and then the  
Figure 6: The eye and finger points used to select a target object. (**A**) Eye–hand trigger selection in the dotted line circles. (**B**) The algorithm flow chart of the judging trigger.  
user needs to fine-tune the finger direction according to the position of the indication point, which is equivalent to brain-hand-eye feedback. So, from experience, we hope that the indication point has less jitter. After adding a filter (Equation (5)), the variance in jitter is reduced, and the average distance from the target point is also lower.  
\[y_{i}=\alpha_{i}y_{i-1}+\alpha_{i}y_{i-2}\dots\alpha_{i}y_{i-9}+\alpha_{i}y_{i- 10} \tag{5}\]