# Real-Time Arm Gesture Recognition Using 3D Skeleton Joint Data
## 1 Introduction  
Both poses and gestures may be used for communication, since they are able to convey some kind of meaning. Pose and gesture recognition aim to recognizing such meaningful expressions performed by a human. These expressions typically involve the motion of hands, arms, head, facial expressions and in some cases the whole body. Their recognition often aims to facilitate the interfacing between humans and computers. Current applications range from sign language, to gaming, medical applications and even virtual reality. Typically, the user stands in front of a camera and some parts of his/her body are detected and tracked. Features are then extracted and compared to models so as to be translated into some predefined gestures.  
In this work we propose an arm gesture recognition approach which introduces a novel set of features. More specifically we use the Kinect sensor and its software development kit (SDK) to extract and track the skeletal joints. Using a subset of these joints, meaningful for the set of gestures that we aim to recognize, we extract their 3D coordinates. Then we calculate a set of statistical features on the aforementioned set of coordinates and for the whole set of video frames that depict the specific gesture. In the following, the term \"gesture\" shall refer to arm gestures that may involve hands, wrists and elbows. We investigate the use of some well-known machine learning algorithms on these features. Moreover, we adopt two real-lifescenarios. We consider the cases where such an approach may be used by (a) a limited set of known users and (b) an unlimited set of both known and unknown users. The former scenario corresponds to the case where all users of the system are a-priori known; i.e., the system shall be designed to work only on them. The latter scenario is the case where the system is pre-trained with a set of available users but also needs to work with previously unseen ones. To our belief, both cases may be involved in real-life scenarios, since they may fit real-life user requirements, thus both need to be addressed and evaluated.  
We also present a novel dataset of gestures, used for the evaluation of the aforementioned algorithms and scenarios. This set of features has been defined in order to be semantically meaningful, to reflect small gesture changes to small feature changes and to require constant time (in terms of complexity). Moreover, we investigate whether the manual filtering of a dataset from \"mischievous\" users that fail (intentionally or not) to provide reliable gestures leads to significant improvement of the results while is worth the effort spent. Since our ultimate goal is to extract features and classify the corresponding gestures in real-time, using trained models, we perform an evaluation of our approach in common hardware architectures and more specifically in terms of the time required for the recognition of a gesture. We discuss which algorithms are suitable for adoption into real-life, real-time applications and the trade-offs between precision and responsiveness. Finally, we perform extensive comparisons with the state-of-the-art and discuss research works that share similar goals to those of this paper. We should clarify that our approach does not perform any temporal segmentation to detect the beginning and the ending of a possible gesture; instead we consider this problem as solved, i.e., we use pre-segmented videos and only aim to recognize gestures per segment. Note that each segment may contain at most a sole gesture.  
The rest of this paper is organized as follows: Section 2 presents related research works in the area of gesture recognition using skeletal data. Section 3 presents the Kinect SDK and the proposed feature extraction approach. The dataset and the experimental results are presented in Section 4, where we also include discussion and comparisons to the state-of-the-art. Finally, in Section 5, we draw our conclusions and discuss plans for future work.