## 2 Proposed System  
### Overview of Proposed System  
**Motivation:** The interactions between humans and robots have become increasingly important. Therefore, in many cases, an intelligent robot needs to understand the commands or wills of humans more intuitively. Figure 3 presents examples in which human-robot interaction is quite an important ability for the robot. Moreover, such interaction should not be unidirectional communication, but rather bidirectional. This means that robots should not only be remote controlled by humans, but also understand the human will and take the corresponding actions autonomously. We proposed a gesture-action joint recognition method for human-robot interaction and tested it on a real UAV platform. Such an ability is also important for a UAV to complete numerous difficult  
Figure 3: Gesture/Action recognition is the intuitive way for human-robot interaction under many tasks. tasks. For example, in Figure 3c,d, the person needs to be rescued or the package receiver does not contain a remote controller, and the gesture/action-based human-UAV interaction will play an important role in the rescue operation (or delivery task). Furthermore, we believe that such an interaction method is not limited to UAVs, but offers potential applications for all types of robots (such as industrial robots and unmanned taxis, as illustrated in Figure 3a,b). **Problem statement of current methods:** As illustrated in Figure 4, even for state-of-the-art methods, neither spatio-temporal action recognition nor gesture recognition algorithms are sufficient to be directly applied to human-robot interaction [26]. This is because the action recognition algorithm may suffer from: (1) its limited classification ability (up to 95 % recall rate, as described in [19]); and (2) a similar action caused by another person (as in Figure 4a). Moreover, the gesture recognition algorithm may incorrectly produce a false alarm in the background (Figure 4b is the result of _YOLO V3_[27]). Therefore, in this paper, we combined the deep learning based action recognition with a multi-feature cascade gesture recognition algorithm for human-UAV interaction, because the competition of two algorithms may greatly reduce the false alarm of whole system. The main contributions of this work include the following: (1) a scene-understanding algorithm is applied to enable the UAV to change its flight strategy automatically, according to its working conditions; and (2) a cross-validation process is applied to combine the results of the spatio-temporal action recognition and static gesture recognition, which can significantly reduce the false alarms of the visual recognition algorithms. Compared with the present state-of-art gesture/action-recognition-based human-UAV interaction systems, the proposed method is superior to them in three ways as: (1) intelligent ability to adjust the flight strategy according to the flight condition; (2) increased correct recognition accuracy up to over 99% (the others vary between 90\(\sim\)92%); (3) working stably when multiple persons doing similar action are captured in the image (the others just assume one person is captured). As illustrated in Figure 5, the proposed system is composed of five modules: a scene-understanding module, pilot detection module, action detection and recognition module, gesture recognition module, and joint reason and control module. After the operator is found through the pilot detection module, his or her action will be detected and recognized by a deep learning-based action classification process. Meanwhile, in order to reduce the false alarms produced by the action recognition, a multi-feature cascade gesture recognition method is applied, based on the fact that both the action recognition and gesture recognition producing false alarms is a low-probability event. The combination of each gesture and action corresponds to a unique flight action.