# Activity Recognition Using Different Sensor Modalities and Deep Learning
## 2 Materials and Methods
Total 40 healthy volunteers (30 males and 10 females) with ages ranging from 20 to 32 (mean 26.4), height 163-195 cm (mean 178), and weight 50-103 kg (mean 78). Each user was informed about the aim of the study and was requested to realize eight different gait-based activities; namely, walking on flat ground (WG), walking uphill (WU), walking downhill (WD), ascending stairs (ASC), descending stairs (DES), running (RN), jumping (JP), and sit down/stand up (SS). Figure 2: The designed smart insole and position of components. For each subject, a total of 40 min of data were recorded for eight different activities by collecting 5 min of data per activity. Accelerometers and FSRs are highly sensitive sensors and include noises and artifacts during the movement. Moving average filters were used to cancel them out. The data values of the 3-axis accelerometer and 3-axis gyroscope outputs, and also foot contact states, were not in the same range. for a uniform data range, min-max normalization was separately applied for each feature of the accelerometer and gyroscope. This is important during neural network training and testing to eliminate the dominant characteristics of large values over small values. All the recorded data from all the volunteers were divided into frames using 1.5 s non-overlapping window sizes due to the average cycle duration of activities. 8000 frames for each activity, and a total of 64,000 frames were obtained. Then, the frames were divided into three sections for training, validation, and testing. Specifically, 70% and 30% of the total frames belonging to 28 volunteers were used for training and validation, respectively. Then, all frames of the other 12 volunteers were used to test and verify the results. The processed data were separated into three categories; only foot contact state, the combination of accelerometer and gyroscope outputs, and a set of all the data according to feature type. Each category was fed into the ConvLSTM NN, respectively. A comparison of the network performances was performed in terms of precision, recall, \(F_{score}\), and accuracy for different input sets. Deep learning is a modern machine learning tool that extracts features automatically from raw data and processes them for prediction and classification. The most popular types are convolutional neural networks and long-short-term memory neural networks (LSTM NN). We have used a hybrid model, which is called convolutional long-short-term memory, that presents a superior performance compared to these two neural networks [22; 27]. This model is a combination of the feature extraction layer of CNN and LSTM. CNN extracts the features from the raw data using convolutional and pooling layers. Then, the extracted features are input to the LSTM NN that models the temporal dynamics. The outputs of LSTM NN feed into fully connected layers (FC), consisting of one or more feed-forward hidden layers. Finally, the softmax activation function and classification layer produce a class label using outputs of FC. Figure 3: Sensor position on the human body. Hyperparameters are factors that affect the performance as well as network type. There are various parameters such as the number of layers, learning rate, filter number, filter size, etc. All these parameters are determined using the trial and error method. All data processes and deep learning applications were conducted in MATLAB 2020a. To speed up the process duration, we have used a graphical processor unit (GPU) [33,34]. Its specifications are the NVIDIA RTX 2070 model, 8 GB memory capacity, and 14 Gbps memory speed.