Multimodal Deep Learning for Activity Detection from IoT Sensors

AbstractThe advancement of various computing technologies such as Artificial Intelligence (AI), cloud computing, mobile computing, and Internet of Things (IoT) is widely applied in vast applications. The rapid evolvement of IoT is determined in advance with embedded systems, data analytics, and wireless connectivity. However, the Deep Learning (DL) method is highly applicable to evaluating human activity which is a significant element to determine the surveillance of human behavior. But the common issue obtained in the detection of human activity is the time series problem and failure to detect accurately in heterogeneous sensors. So a novel Multi-Modal Deep-Learning (MMDL) is proposed to detect human activity accurately with reduced time consumption. However, in real-time applications, huge amounts of data are monitored and the activity of every individual is detected independently by integrating the proposed method with 2D Deep Canonical Correlation Analysis (DCCA). Furthermore, the detection of human activity is employed through IoT devices like multi-modal sensors. The multi-head CNN is used to decrease the cross channels and spatial interactions while training with high-dimensional datasets. The MMDL and binary Support Vector Machine (SVM) architectures overcome the drawbacks of the existing techniques such as the need for increased width/depth for improved performance and the rare usage of hierarchical representations. Thus the effectiveness of the proposed method is validated by STISEN and GAIT datasets. The experimentation results revealed that the proposed method attained a better accuracy of 97% with reduced time in various scenarios related to existing methods.KEYWORDS: IoTdeep learningsensorsmultimodelactivity detection Disclosure statementNo potential conflict of interest was reported by the author(s).Supplemental DataSupplemental data for this article can be accessed at https://doi.org/10.1080/03772063.2023.2248955.Availability of Data and MaterialThe data that support the findings of this study are available from the corresponding author upon reasonable request.Additional informationNotes on contributorsT. HemalathaT Hemalatha received bachelor's degree in computer science & engineering from Thiagarajar College of Engineering in 1994 and masters degree in computer science & engineering from College of Engineering - Guindy, Anna University in 2004. Her interests are in network security, grid & cloud computing, machine learning, distributed system and adhoc & computer networks. She is a life member of ISTE and CSI.T. C. KalaiselviT C Kalaiselvi is currently associate professor in the department of ECE, Kongu Engineerring College, Perundurai Erode. She has organized INSA sponsored Seminar on, “Computer Aided Breast cancer Detection and Diagnosis with Deep Learning Algorithms”. She has organized a Workshop on Freescale Semiconductor Processors and Controllers association with KEC, Perundurai. She is reviewer for various national and International Journals. She has published book chapter in Wiley publishers in IoT smart grids for maintenance management. She has obtained International patent in IoT based Agricultural Application in the year 2021. Email: tckalaiselvi@mail.lvC. Gnana KousalyaC Gnana Kousalya received the bachelor degree in electronics and communication engineering from Government College of Technology, Coimbatore in the year 1993. She is a member of IEEE RAS, IEEE WIE & IEEE EMBS. She is also a corporate member of IETE and Life Member of ISTE. Her research interests include security in wireless sensor networks, image processing, signal processing, IoT and embedded systems design. Email: cgnanakousalya@inbox.lvG. RohiniG Rohini received the bachelor degree in electronics and communication engineering from PSG College of Technology, Coimbatore in 1992. She is a corporate member of IETE, Member of IEEE, and Life Member of ISTE. Her research interests include VLSI design and design for testability, High speed low power VLSI architectures, algorithms for VLSI signal processing, image processing, IoT and embedded system. Email: grohini@inbox.lv