# A Graph-Based Approach to Recognizing Complex Human Object Interactions in Sequential Data
## 1 Introduction  
Artificial intelligence (AI) has revolutionized the healthcare sector. From rehabilitation systems through assisted living programs to patient activity monitoring solutions, the advances in the field of human interaction recognition (HIR) have made numerous contributions to the field of medical engineering. By actively monitoring a person's actions, theearly symptoms of a serious disease can be detected in a timely fashion. Similarly, by recognizing human actions, their progress can be evaluated in rehabilitation programs. Such systems are usually complimented by the use of wearable and vision sensors. Therefore, the accurate recognition of human-object interactions is crucial for the success of a wide range of systems designed for smart homes [1; 2; 3; 4], sports [5; 6], healthcare [7; 8], surveillance [9], counting and tracking [10; 11; 12; 13], e-learning [14], and monitoring [15; 16]. Despite the recent advances, human-object interaction recognition (HOIR) remains a challenging task because of various reasons, including illumination variation, background clutter, intra-class variation, inter-class similarities, scale variation, multiple viewpoints, and occlusion. Moreover, the quality of input data and its challenges depend on the medium of acquisition. Apart from regular static cameras, drone [17; 18] and depth cameras [19; 20] are also quite popular. Over the past few years, it has also been observed that researchers have employed various sensors, including body-worn [21; 22] and inertial sensors [23], to record human activities and their interactions with the surrounding objects. Moreover, instead of relying only on RGB (red, green, blue) data, the use of depth [24; 25] and RGB-D (red, green, blue, depth) [26] data is also common. Therefore, this article proposes a robust HOIR system which takes sequential data as input and recognizes the HOI interaction being performed in it. In particular, firstly, the system pre-processes the input data by adjusting the contrast and smoothing the incoming image frames. Then, it locates the human and object through image segmentation. Based on this, 12 key body parts are identified from the extracted human silhouette through a graph-based image skeletonization technique called image foresting transform (IFT). After this, three types of features are extracted: full-body feature, point-based features, and scene features. Dense trajectories and local intensity order patterns (LIOP) are obtained from full-body silhouettes. Similarly, kinematic postures and local occupancy patterns (LOP) are obtained from the 12 key body points. Finally, spatial pyramid matching (SPM) and generalized search tree (GIST) descriptors are stored as scene features. The next step involves optimizing the six different features using isometric mapping (ISOMAP). Lastly, the optimized feature vector is fed to a graph convolution network (GCN), which performs the HOI classification. The main contributions of this paper are as follows:  
* We used a graph-based image skeletonization called the image foresting transform (IFT) technique to detect 12 human body parts. * We proposed a multi-feature approach involving three different types of features: full-body features, point-based features, and scene features. We also provided types of feature descriptors for each category. * We optimized the large feature vector obtained through isometric feature mapping (ISOMAP). * We implemented a graph convolution network (GCN) and tuned its parameters for the final classification of human locomotion activities. The rest of the article is arranged as follows: Section 2 investigates the related work in the field of HOI recognition. Section 3 explains the proposed method in great detail. Section 4 describes the different datasets that were used to validate the performance of the proposed method and the results of those experiments. Section 5 presents an analysis of the results achieved by the system and discusses the strengths and weaknesses of the system.