## 4 Comparative Study
### Model Setup  
For the assessment, a variety of models from three deep learning classes were implemented: Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and Convolutional Autoencoders (CAEs) with fully connected networks.  
The CNN model consisted of three instances of _2D convolutional layers_, where each was followed by _a maximum pooling layer_ and a _dropout layer_. Next, the network concluded with a flattening layer for implementing vectorization and connecting to two fully connected layers (see Figure 12). The architecture was implemented based on the _Keras sequential API_ using the _Input_, _Conv2D_, _MaxPooling_, _Dropout_, _Flatten_, and _Dense_ layer functions; the _SGD_ and _Adam_ optimizers; and the _Categorical Cross-Entropy_ loss function from the _keras.layers_, _keras.optimizers_, and _keras.losses_ packages, respectively.  
The increasing number of filters in each convolutional layer helped build hierarchical features and prevent overfitting. The first layers captured low-level information, whereas  
Figure 11: Images of micro-Doppler signatures (spectrograms) for six activities (from [5]).  
Figure 12: Structure of selected CNN.  
the last ones reached higher levels of abstraction with higher complexity and became smaller to enforce generalization. The 3-by-3 kernel, with a stride of 1 and without padding, was required to halve the dimensions of the feature maps until the smallest size was reached, producing good results--28 \(\times\) 28. The values for the penalty function of the regularizers for the kernel, bias, and activity were set to be \(1.0\times 10^{-2}\), \(1.0\times 10^{-4}\), and \(1.0\times 10^{-5}\), which were good empirical values to start with. A _Rectified Linear Unit_ (_ReLU_) activation function was selected for faster learning. The comparably average dropout rate of 0.4 was well-suited for this network since the aforementioned regularizers had to be taken into account. The first downstream FCN, which consisted of 212 nodes and connected the last maximum pooling layer with the output FCN, was used for the classifier. It was required to transform the spatial features of the feature maps into complex relationships. The output FCN had six nodes, each representing one class and using a softmax activation function to determine the probability of class assignment for the input image.  
The RNN models were constructed using simple RNNs, LSTMs, Bidirectional LSTMs, and Gated Recurrent Units (GRUs), which were also implemented based on the _Keras sequential model_ using the _SimpleRNN_, _LSTM_, _GRU_, _Bidirectional_, and _Dense_ layer functions from the _keras.layers_ package.  
The number of nodes in the first part, which was the recurrent network, was uniformly set to 128, which led to good results and prevented overfitting. For activation, the _hyperbolic tangent function_ (_tanh_) was selected, as it is associated with bigger gradients and, in comparison with _the sigmoid function_, faster training. Each network was followed by a fully connected layer to establish the complex nonlinear relationships required to connect the time-specific memory to the respective classes. Using a softmax function for activation, the probabilities were outputted for each class.  
The autoencoder-based model was implemented based on the architecture of the CAE (see Figure 13). In contrast to the aforementioned implementation, it used the _Keras functional API_, as it is more flexible and allows for branching and varying the numbers of inputs and outputs. The branching option was required to independently define the encoder and decoder parts since two consecutive training sessions were required. The first training (_pretraining_) was performed on the complete autoencoder model consisting of the encoder and decoder parts to train the feature-extracting capabilities. Then, the trained weightings and biases were transferred to a separate model consisting of the encoder part and an FCN, which implemented the classifier, to output the class probabilities.