# Context-Aware Complex Human Activity Recognition Using Hybrid Deep Learning Models
###### Abstract
Second, multiple complex activities share one or more common and related simple activities. The _driving_ and _eating_ examples above clearly illustrate this. Another good example is making sandwiches, which is expected to exhibit signal patterns as with other activities, such as _making a coffee_ and _cycling_, in terms of activity patterns. Third, complex activities, compared to simple activities, contain higher level semantics that is often more complicated and is known to be more reflective of our daily lives, e.g., _driving, eating_, and _relaxing_. Thus, complex activities often take longer time to detect compared to simple ones [15; 16]. In the past, as illustrated in Figure 1, researchers have solved several HAR problems using traditional machine learning techniques and handcrafted feature extraction methods [17; 18; 19]. Although this process is known to help reduce the learning models' overfitting problem, reduce training time, and improve accuracy, the downside is that when features are handcrafted, the model becomes susceptible to generalization problems. Handcrafted feature extraction is also limited by human domain expertise, i.e., a deep knowledge of the application domain is required. Lastly, because this approach only helps extract shallow features, the performance of the model is largely compromised, resulting in poor generalization [12]. Figure 1: An overview of traditional activity recognition processes. Recently, Deep Convolutional Neural Networks (DNNs) began to deliver on their promises of automatic feature extraction to achieve state-of-the-art results for HAR. When a DNN is applied to raw time-series data, for example, it often outperforms models trained on heuristic handcrafted features [19] because of its automatic feature extraction capabilities. Deep learning models also have capabilities to extract high-level representation within the deep layer making them more applicable to complex activity recognition problems [12]. The most common DNNs for HAR are the Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), and the Long Short-Term Memory (LSTM) networks [2; 20]. CNNs are specifically designed to effectively process image data, solving computer vision problems, such as image classification, image captioning, object localization, etc. [21]. Compared to CNNs, RNNs can learn and keep memories of temporal dependencies of time-series data by relying on their dynamic temporal behaviors. LSTM networks are an extension of RNNs, possessing complex memory cells, which help to avoid the long-term dependency problem of vanilla RNNs. Since DNNs can adaptively learn spatial hierarchies of features to discover low- and high-level patterns in the dataset, new datasets and new sensor modalities can be quickly adapted with minimal configurations. Several authors have demonstrated the effectiveness of DNNs in HAR systems with state-of-the-art results, and with some recently using hybrid deep learning models [22] as we discuss in Section 2. Considering the identified problems above, the goal of this article is to develop context aware HAR models, capable of discriminating between simple and complex human activities of daily living, and for addressing the associated interclass similarity problems using a combination of traditional motion and environmental sensing data with hybrid deep neural networks. The proposed deep hybrid HAR model combines rich context-aware data from ambient sensors with low-level inertial sensor signals as illustrated in Figure 2. The motion sensors include the accelerometer, gyroscope, and other sensors, such as the magnetometer, while the context signals, which serve as the context data, include signals from sound and light sensors of smartphones. The hybrid model effectively combines diverse and important capabilities of CNNs and LSTM to form a CNN-LSTM hybrid model [21; 22; 23]. Existing works using this type of hybrid model have proven to provide very good accuracy [23; 24].