# Context-Aware Complex Human Activity Recognition Using Hybrid Deep Learning Models
## 3 Methodology
### The CNN-LSTM Hybrid Model
#### 3.4.4 Hyperparameter Tuning  
To ensure optimal performance for the model, we evaluated the influence of several hyperparameter settings on the performance of the model. We adopted the grid search methodology, a tested and reliable method for tuning hyperparameters [59]. Grid search methodology proved to be the optimal approach for setting the hyperparameters by creating a grid of a certain range and then looping through the candidate elements to find the best values for each hyperparameter. Several studies provide sufficient guidance on how to approach grid search, especially when a hybrid CNN-LSTM model is involved [22, 67]. Both works analyzed the effect and shared the many benefits of adding the CNN layers before the LSTM to solve speech and activity recognition problems. We approached the hyperparameter tuning by first optimizing specific components of the model, and then we moved on to tuning the model for overfitting. For the subcomponent tuning, a common practice is to have 2 consecutive CNN layers followed by a dropout and a max pooling layer [18]. We adopted this simple structure for the CNN-LSTM model and trained the 2 fully connected convolutional layers with 9 feature maps representing the 3 (x, y, and z) axes of the 3 inertial sensors (accelerometer, gyroscope, and magnetometer). The number of nodes in the hidden layer was replaced by the number of filter maps and kernel size, which we grid searched as well. The number of filters was (16, 32, 64, 128, and 256), while the kernels investigated were between 2 and 7 in the following order (2, 3, 4, 5, and 7). Moreover, for the convolutional layers, the input data must be in the form of (samples, timesteps, and features) where features map onto the channels, and in this case, it was 9 for the 9 variables that we had from the 3D inertial sensors. The approach to implementing this model was to split each window of 64-time steps, for example, into subsequences for the CNN model to process. The 64-time steps in each window could then be split into 4 subsequences of 16-time steps. As a result, we defined a CNN model that read in sequences with a length of 16-time steps and 9 features. The entire CNN model was also wrapped in a _time-distributed_ layer to allow the same CNN model to read in each of the subsequences in the window. The extracted features were then flattened and fed into the LSTM extracting its features before a final mapping to the activity. Another important parameter that requires tuning is the batch size. The training data were fed to the model in one or more batches. A batch represents the collection of samples that the model processes prior to its weights being updated. Depending on the dataset, adequate measures were taken to ensure that the model was fed with the right batch size for it to learn optimally [68]. Using the grid search, we passed batches ranging from 16 to 512 to train the model for each window length since the interaction between the two parameters affects model performance. To prevent overfitting, we optimized several parameters including weight decay, dropout regularization, number of epochs, and learning rate. Weight decay or weight regularization provides an approach to reduce overfitting of the training data and improve generalization capability of new data. We tuned the L2 weight regularization for both the CNN layers and the LSTM cells. We confirmed from the literature that the CNN layers usually require small L2 weight decay to perform optimally [64, 69]. Dropout regularization is another way to prevent the model from overfitting. This is often referred to as the dropout rate, which can be specified for different layers of the model, and it defines the probability of setting each input to the layer to zero [70]. In other words, a dropout rate of 0.3 informs the hidden layer to set 30% of input to zero. For the hybrid model, we grid searched between 0 and 0.9 dropout rates for each of the hidden layers, i.e., the CNN layers and the LSTM cells.