# Merging-Squeeze-Excitation Feature Fusion for Human Activity Recognition Using Wearable Sensors
###### Abstract
Since the fused feature maps are the ones who enter the classification process, the channel weights could be computed such that the fused feature maps provide very discriminative information. Therefore, unlike the SE mechanism, we design the MSE feature fusion such that, at each branch, it computes the channel weights based on both local feature maps and fused feature maps. As a result, when we consider a set of \(C\) local feature maps, the \(c\)-th local feature map will be emphasized if either it is important to the classification or the corresponding \(c\)-th fused feature map is useful to the classification. The MSE feature fusion consists of three steps: pre-merging step, squeeze-and-excitation step, and post-merging step. In the pre-merging step, the feature maps from all branches are merged together to obtain a set of pre-merged feature maps. Thereafter, during the squeeze-and-excitation step, the feature maps from each branch are recalibrated according to their importance measured from both the channel-wise statistics obtained from themselves and the channel-wise statistics obtained from the pre-merged feature maps. Finally, in the post-merging step, the MSE feature fusion applies the same merging operation used in the first step to combine the output feature maps from all branches and obtain a set of channel-weighted and merged feature maps which will be used to classify the activities of interest. In this work, we have applied three DL models (i.e., LeNet5, AlexNet, and VGG16) as feature extractors and four merging methods (addition, maximum, minimum, and average) as merging operations in the pre-merging step and post-merging step. Furthermore, we also modify the proposed MSE feature fusion by adding local skip connections, adding a global skip connection, using global channel attention, and stacking a series of the MSE feature fusions to create deep MSE feature fusions. Their performances are evaluated on three public HAR datasets: PAMAP2, DaLiAc, and DSAD. The main contributions of this work are summarized as follows:  
1. We propose five MSE feature fusion architectures for wearable-sensor HAR using multi-branch architectures such that the feature maps will be recalibrated according to their importance during the fusion. Three DL models and four merging methods used in the MSE feature fusion are studied and investigated. 2. Extensive experiments are conducted to evaluate and compare the performances of the proposed methods and baseline architectures by using the PAMAP2, DaLiAc, and DSAD datasets. The results show the following findings: * The MSE feature fusion with a global skip connection when using the average merging and AlexNet achieves the highest accuracy score of 99.24% on classifying the PAMAP2 dataset. * The MSE feature fusion with local skip connections when using the minimum merging and AlexNet achieves the highest accuracy score of 98.59% on classifying the DaLiAc dataset. * The original MSE feature fusion using the average merging and AlexNet achieves the highest accuracy score of 98.04% on classifying the DSAD dataset. * Among the merging methods studied in the proposed methods, the addition merging offers the worst accuracy scores. The maximum, minimum, and average merging have similar performances. * All of the highest accuracy scores are from using AlexNet as the feature extractor. The rest of this paper is organized as follows. Section 2 reviews the previous work focusing on using the SE block in HAR, proposing multi-branch DL architectures for HAR, and presenting SE-based feature fusion methods. The data collection and preparation are described in Section 3. Sections 4 and 5 present the proposed MSE feature fusion and its extensions, respectively. Their performances are evaluated and compared in Section 6. Finally, conclusions and future work are provided in Section 7. The main symbols used in this paper are summarized as follows. Lower-case and upper-case bold letters represent vectors and three-dimensional (3D) arrays, respectively.