In this work, we address a challenging problem of fine-grained and coarse-grained recognition of object manipulation actions. Due to the variations in geometrical and motion constraints, there are different manipulations actions possible to perform different sets of actions with an object. Also, there are subtle movements involved to complete most of object manipulation actions. This makes the task of object manipulation action recognition difficult with only just the motion information. We propose to use grasp and motion-constraints information to recognise and understand action intention with different objects. We also provide an extensive experimental evaluation on the recent Yale Human Grasping dataset consisting of large set of 455 manipulation actions. The evaluation involves a) Different contemporary multi-class classifiers, and binary classifiers with one-vs-one multi- class voting scheme, b) Differential comparisons results based on subsets of attributes involving information of grasp and motion-constraints, c) Fine-grained and Coarse-grained object manipulation action recognition based on fine-grained as well as coarse-grained grasp type information, and d) Comparison between Instance level and Sequence level modeling of object manipulation actions. Our results justifies the efficacy of grasp attributes for the task of fine-grained and coarse-grained object manipulation action recognition.
Hand gesture recognition is more closer to the problem of object manipulation action recognition. Hand gesture recognition has also been addressed using depth data generated from Kinect in @cite_18 and @cite_27 . But these techniques mainly target sign language gestures and not the human hand-object interactions. @cite_27 treats an action sequence as a 4D shape and propose random occupancy pattern (ROP) features, extracted from random sampling of 4D subvolumes with different sizes and at different locations. In gesture depth sequences, the semantics of the gestures are mainly understood by the large movement of the hand. These approaches use cropped portion of hand using some hand detection approach, to determine these large hand movements to model different gestures. But, these clear motion information are not easily perceivable in the case object manipulation tasks.
@cite_27 @cite_18
2169251375 1568156700
The recently developed depth sensors, e.g., the Kinect sensor, have provided new opportunities for human-computer interaction (HCI). Although great progress has been made by leveraging the Kinect sensor, e.g., in human body tracking, face recognition and human action recognition, robust hand gesture recognition remains an open problem. Compared to the entire human body, the hand is a smaller object with more complex articulations and more easily affected by segmentation errors. It is thus a very challenging problem to recognize hand gestures. This paper focuses on building a robust part-based hand gesture recognition system using Kinect sensor. To handle the noisy hand shapes obtained from the Kinect sensor, we propose a novel distance metric, Finger-Earth Mover's Distance (FEMD), to measure the dissimilarity between hand shapes. As it only matches the finger parts while not the whole hand, it can better distinguish the hand gestures of slight differences. The extensive experiments demonstrate that our hand gesture recognition system is accurate (a 93.2 mean accuracy on a challenging 10-gesture dataset), efficient (average 0.0750 s per frame), robust to hand articulations, distortions and orientation or scale changes, and can work in uncontrolled environments (cluttered backgrounds and lighting conditions). The superiority of our system is further demonstrated in two real-life HCI applications. In this paper we propose a hand gesture recognition system that relies on color and depth images, and on a small pose sensor on the human palm. Monocular and stereo vision systems have been used for human pose and gesture recognition, but with limited scope due to limitations on texture, illumination, etc. New RGB-Depth sensors, that reply on projected light such as the Microsoft Kinect, have overcome many of those limitations. However, the point clouds for hand gestures are still in many cases noisy and partially occluded, and hand gesture recognition is not trivial. Hand gesture recognition is much more complex than full body motion, since we can have the hands in any orientation and can not assume a standing body on a ground plane. In this work we propose to add a tiny pose sensor to the human palm, with a minute accelerometer and magnetometer that combined provide 3D angular pose, to reduce the search space and have a robust and computationally light recognition method. Starting with the full depth image point cloud, segmentation can be performed by taking into account the relative depth and hand orientation, as well as skin color. Identification is then performed by matching 3D voxel occupancy against a gesture template database. Preliminary results are presented for the recognition of Portuguese Sign Language alphabet, showing the validity of the approach. This paper discusses the development of a natural gesture user interface that tracks and recognizes in real time hand gestures based on depth data collected by a Kinect sensor. The interest space corresponding to the hands is first segmented based on the assumption that the hand of the user is the closest object in the scene to the camera. A novel algorithm is proposed to improve the scanning time in order to identify the first pixel on the hand contour within this space. Starting from this pixel, a directional search algorithm allows for the identification of the entire hand contour. The @math -curvature algorithm is then employed to locate the fingertips over the contour, and dynamic time warping is used to select gesture candidates and also to recognize gestures by comparing an observed gesture with a series of prerecorded reference gestures. The comparison of results with state-of-the-art approaches shows that the proposed system outperforms most of the solutions for the static recognition of sign digits and is similar in terms of performance for the static and dynamic recognition of popular signs and for the sign language alphabet. The solution simultaneously deals with static and dynamic gestures as well as with multiple hands within the interest space. An average recognition rate of 92.4 is achieved over 55 static and dynamic gestures. Two possible applications of this work are discussed and evaluated: one for interpretation of sign digits and gestures for a friendlier human-machine interaction and the other one for the natural control of a software interface. This paper presents a literature review on the use of depth for hand tracking and gesture recognition. The survey examines 37 papers describing depth-based gesture recognition systems in terms of (1) the hand localization and gesture classification methods developed and used, (2) the applications where gesture recognition has been tested, and (3) the effects of the low-cost Kinect and OpenNI software libraries on gesture recognition research. The survey is organized around a novel model of the hand gesture recognition process. In the reviewed literature, 13 methods were found for hand localization and 11 were found for gesture classification. 24 of the papers included real-world applications to test a gesture recognition system, but only 8 application categories were found (and three applications accounted for 18 of the papers). The papers that use the Kinect and the OpenNI libraries for hand tracking tend to focus more on applications than on localization and classification methods, and show that the OpenNI hand tracking method is good enough for the applications tested thus far. However, the limitations of the Kinect and other depth sensors for gesture recognition have yet to be tested in challenging applications and environments.