## 4 Non-Vision-Based HAR
### Multimodal HAR
proposed a Distributed Sensors Fusion Network (DSFNet) [62]. The model has two branches: an angular velocity-based branch and an acceleration-based branch. Angular velocity and acceleration are taken from the inertial sensors on different positions on the body and fed into the corresponding branch of the model. The feature maps from both branches are expanded into a vector and fed into a classification network, which then outputs the classification score for the different categories. They evaluated their model on the Comprehensive Multimodal Human Action Dataset (CZU-MHAD) [73], and it showed competitive performance. They performed an ablation study and showed improved performance by the post-fusion model compared to the pre-fusion model [62]. [PERSON] et al. proposed a multimodal deep learning model for recognising Complex Human Activities called DEBONAIR (deep learning-based multimodal complex human activity recognition) [11]. They classified their input data into three based on their properties: fast-changing and simple data, fast-changing and complex data and slow-changing data. They proposed using different sub-networks to process each type of data. Then, a depth concatenation operation was applied to the output of the sub-networks, and a convolutional layer was used to fuse the features. Two LSTM layers were used to learn sequential information from the fused features, and a final fully connected layer was used to generate the probability distribution for the Complex Human Activities (CHA). DEBONAIR was evaluated on the lifelog dataset and the PAMAP2 dataset using weighted F1-score. It was compared to different models (Hierarchy [74], Non-hierarchy [75], SADeepSense [76], DeepSense [77], Channel-based Late Fusion [78], and DeepConvLSTM [79]) and it scored higher. [PERSON] et al. proposed the use of a Deep Multistage LSTM model to integrate features from multiple sensors for HAR [80]. Their proposed model first extracts temporal features from each sensor datum, which is then aggregated and optimised before being fed into the final activity prediction layer. They tested the model on a dataset from Physionet [81] and compared the effect of the different sensors against the performance of integrating multiple sensors. Their result showed that the integration of multimodal features led to an improvement in accuracy. Multimodal HAR has been shown to demonstrate improved performance over single-modality HAR [11; 56; 58; 67]. It allows for combining different data types as each stream of the model can be adapted to a particular data type. Table 4 shows a summary of some of the multimodal HAR models that have been proposed in recent literature. Almost all of them had an accuracy of over 91% in the experiments conducted, and where ablation tests were performed, they showed that the multimodality increased the accuracy of the models. Its application in healthcare would allow for combining inertial data and vital health data for a more robust HAR solution. However, one of the issues faced by this method is the availability of multimodal datasets. This is largely due to the novelty of the method; with an increase in interest in this methodology, there will also be an increase in the availability of datasets. Section 6 covers some of the available multimodal datasets. Quality multimodal datasets have to ensure that the data are not misaligned, which can be caused by the heterogeneity of the different data types. It is also computationally intensive to process long sequences of multimodal data, and it requires complex fusion techniques. Research in this area should focus on finding optimal solutions to these issues.