ReFuSeAct: Representation fusion using self-supervised learning for activity recognition in next generation networks

Authors:
raw_author_name: Sunder Ali Khowaja
raw_affiliation_string: Faculty of Engineering & Technology, University of Sindh, Pakistan
raw_author_name: Parus Khuwaja
raw_affiliation_string: Institute of Business Administration, University of Sindh, Pakistan
raw_author_name: Fayaz Ali Dharejo
raw_affiliation_string: Department of Electrical Engineering and Computer Science, Khalifa University, Abu Dhabi, UAE
raw_author_name: Saleem Raza
raw_affiliation_string: Science and Technology, Quaid-e-Awam University of Engineering, Larkana, Pakistan
raw_author_name: Ik Hyun Lee
raw_affiliation_string: Department of Mechatronics Engineering, Tech University of Korea, South Korea; IKLAB Inc. Siheung-Si, South Korea
raw_author_name: Rizwan Ali Naqvi
raw_affiliation_string: Department of Intelligent Mechatronics Engineering, Sejong University, Seoul 05006, South Korea
raw_author_name: Kapal Dev
raw_affiliation_string: Department of Computer Science and CONNECT Centre, Munster Technological University, Bishopstown, Cork, T12 P928, Ireland; Department of Electrical and Computer Engineering, Lebanese American University, Byblos, Lebanon; Department of Institute of Intelligent Systems, University of Johannesburg, Auckland Park, 2006, South Africa

Abstract: Over the years, wearable sensors have gained a lot of attention from the research community due to their non-invasive nature, adoption of sensors by general public, and their applicability in healthcare services. With the advancements in communication networks, machine learning methods, and wearable sensor deployment, it is essential to design a method that could accurately classify human activities while reducing the dependence on annotated data. Traditional machine learning approaches require large-scale annotated data in order to provide a reasonable recognition performance. Recently, self-supervised learning methods are proposed but they are either limited to single sensor devices or fail to model intra-modal correlations within the self-supervised learning paradigm. In this work, we propose Representation Fusion using Self-supervised learning for Activity Recognition (ReFuSeAct) framework that uses modality-specific encoders, attention encoders, and decision-level fusion strategies to address the aforementioned limitations. The self-supervised learning paradigm ensures that the method achieves better performance even with less amount of annotated data. The architecture proposed for modality-specific encoder ensures that extraction of representative features that could help in improving recognition performance. The feature-level fusion performed using the proposed attention encoders enhances the quality of representative features that could be used in supervised learning phase. Finally, the decision-level fusion strategy enhances the activity recognition accuracy in comparison to the single deep learning classifier. Our experimental analysis shows that the proposed approach records 9.1% improvement over semi-supervised learning baselines and more than 2% improvement in comparison to existing self-supervised learning approaches.