# Merging-Squeeze-Excitation Feature Fusion for Human Activity Recognition Using Wearable Sensors
###### Abstract  
Human activity recognition (HAR) has been applied to several advanced applications, especially when individuals may need to be monitored closely. This work focuses on HAR using wearable sensors attached to various locations of the user body. The data from each sensor may provide unequally discriminative information and, then, an effective fusion method is needed. In order to address this issue, inspired by the squeeze-and-excitation (SE) mechanism, we propose the merging-squeeze-excitation (MSE) feature fusion which emphasizes informative feature maps and suppresses ambiguous feature maps during fusion. The MSE feature fusion consists of three steps: pre-merging, squeeze-and-excitation, and post-merging. Unlike the SE mechanism, the set of feature maps from each branch will be recalibrated by using the channel weights also computed from the pre-merged feature maps. The calibrated feature maps from all branches are merged to obtain a set of channel-weighted and merged feature maps which will be used in the classification process. Additionally, a set of MSE feature fusion extensions is presented. In these proposed methods, three deep-learning models (LeNet5, AlexNet, and VGG16) are used as feature extractors and four merging methods (addition, maximum, minimum, and average) are applied as merging operations. The performances of the proposed methods are evaluated by classifying popular public datasets. onvolutional neural networks (CNNs); deep learning; feature fusion; human activity recognition (HAR); inertial measurement units (IMUs); squeeze-and-excitation; wearable sensors +
Footnote †: journal: Medical  
from the sensor data and use them in the classification process. As a result, DL models such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs) have been extensively studied in HAR [9; 13; 14; 15]. A CNN model essentially consists of a series of convolutional layers and pooling layers to extract a set of features called feature maps, which will be used later in the classification. However, only some feature maps may be very useful for classifying the activities of interest. Therefore, informative feature maps should be emphasized while ambiguous feature maps should be suppressed. A channel-attention mechanism called the squeeze-and-excitation (SE) block [16] was proposed to solve this issue. The SE block will recalibrate each feature map by a weight value which is proportional to the importance of this feature map in the classification. The SE block has recently been applied to CNN and/or RNN models to better the HAR performances [17; 18]. The performances of wearable-sensor HAR can be improved by implementing multi-branch DL architectures [19; 20]. A multi-branch DL architecture consists of several parallel branches using DL models to extract different sets of feature maps independently. Specifically, each branch provides a set of feature maps denoting local information. Thereafter, these sets of feature maps will be fused by using feature fusion such as concatenation to generate a set of fused feature maps (denoting global information); which will be used later in the classification process. A traditional feature fusion method combines the local feature maps equally without being aware that, in each set, some feature maps may be informative while some feature maps are ambiguous. Motivated by this issue, we need a feature fusion method which is able to emphasize informative feature maps and suppress ambiguous feature maps in each branch during fusion such that we can combine the local feature maps efficiently and obtain useful discriminative fused feature maps. Inspired by the squeeze-and-excitation (SE) mechanism [16], we propose a feature fusion method called the merging-squeeze-excitation (MSE) feature fusion. In each branch, the MSE feature fusion recalibrates the local feature maps by using a set of channel weights.