# Activity Recognition Using Different Sensor Modalities and Deep Learning
## 1 Introduction
Despite its efficiency, CNN can be used for fixed and short sequence classification problems but is not useful for long and complex time-series data problems [19]. LSTM NN is designed for sequential data to extract the hidden pattern from sequential data. In general, such neural networks analyze the input latent sequence pattern and predict future sequences by combining previous information with existing information from spatial and temporal dimensions. However, LSTM NN cannot remember information for a long time [20]. Considering the challenge points of CNN and LSTM NN, a hybrid model, namely, a convolutional long-short term neural network (ConvLSTM NN) is used in this study. There are various studies on deep learning and human activity recognition. [PERSON] and [PERSON] proposed CNN for human activity recognition using smartphone sensors [21]. [PERSON] and [PERSON] made a network performance comparison between a convolutional neural network and a convolutional long-short-term memory neural network (ConvLSTM NN) using the Opportunity dataset [22]. Moreover, many researchers presented a performance comparison of methods based on handcrafted and automated feature extraction [23; 24; 25; 26]. Smart insoles are equipped with sensors inside the insoles or the base of shoes. They can measure force information applied by the foot during movement. FSRs are chosen popularly due to significant advantages such as low cost, simple use, and thin surface for smart insole design. They can measure weight between 0.01 kg to 10 kg. However, applied force during performed activity may be greater than the maximum weight limit and may cause saturation. The limitation of the FSRs can cause applied force differences between user's weight and gait characteristics. These reasons may result in inconsistent data, and the deep learning performance can be affected adversely. To overcome this challenge, the usage of foot contact state is suggested instead of ground reaction force information for activity recognition in this study. In our previous study, we designed a wireless sensor network (WSN) consisting of a master device and IMU slave devices [27]. For this study, we have expanded our WSN, adding smart insoles. These insoles were designed to collect foot contact states from four different points of the sole using force-sensitive resistors. Designed smart insoles and IMU sensors were placed on the soles of the feet and on the thigh part of the legs, respectively. The sensor outputs were collected and recorded by the master device during eight gait-based motions, including ascending stairs and running. The obtained data were fed into ConvLSTM NN by applying pre-processing techniques such as removing noise, data normalization, and data segmentation. The activity recognition performance of the networks was compared for different input sets including only foot contact states, only acceleration and gyroscope data, and a combination of them. In this study, a low-cost smart insole design using force-sensitive resistor sensors is presented for medical and sports applications, as well as research that involves mass participation. FSRs have a limited capability to measure ground reaction force and cause saturation. To overcome this challenge, foot contact states were derived from the ground reaction forces. For this purpose; the HAR approach based on foot contact states and deep learning is proposed. In the literature, many studies are available for activity recognition, and the IMU-based HAR approach is the most popular. The neural network performance of the proposed foot contact state-based HAR approach is compared with the IMU-based HAR approach. Moreover, the performance of the HAR approach, consisting of the smart insole and IMU sensor datasets with deep learning, is examined.