Object of Interest and Unsupervised Learning-based Framework for an Effective Video Summarization Using Deep Learning

AbstractDuring this digital era, a large amount of visual data is being generated by various multimedia sources. A technique is urgently required to provide a clear and accurate summary of the original video to highlight the most informative segments of the video content. However, the varying video resolution, multi-dimensional feature representation, and massive storage create difficulties in key frame extraction techniques. As a result, many unnecessary features of the videos must be dropped to count their unique qualities. Therefore, a novel deep learning-based approach is proposed where frames are first retrieved using 25 frames per second; then, objects are detected on extracted frames using YOLOv5, and the frames with the target object only are processed further to overcome time consumption and high-speed computing hardware limitations. Further features are obtained using VGG-16 and object of Interest (OoI) based ResNet-50, respectively, and comparisons are performed to find the best solution. The extracted features are compressed using Principal Component Analysis (PCA) based on unsupervised Learning, which may efficiently minimize information loss, and potentially reduce dimension. By performing a comprehensive evaluation to obtain the best value of K using the Silhouette score, Candidate frames are extracted with the maximum mean and standard deviation from each K-means algorithm-based cluster. Pearson Correlation Coefficient (PCC) is used as a post- processing step to remove the redundant frames from the candidate frames and final keyframes extraction. The experiment was performed on the benchmark office dataset from industrial surveillance, which outperforms the state-of-the-art models in terms of recall score.KEYWORDS: Keyframe extractionK-meansObject of interest (OoI)Pearson correlation coefficient (PCC)Principal component analysis (PCA)ResNet-50Video summarization (VS)VGG-16 ACKNOWLEDGMENTThe authors would like to thank the DST GoI for sponsoring the work under DST/ICPS/General/2018.Disclosure statementNo potential conflict of interest was reported by the author(s).Additional informationNotes on contributorsAlok NegiAlok Negi is currently pursuing PhD (CSE) at the National Institute of Technology, Uttarakhand. He holds his Mtech (CSE) from Uttarakhand Technical University, Dehradun, and Btech (IT) from DIT University, Dehradun. He is working as an assistant professor (senior scale) in business analytics, big data analytics, data science, artificial intelligence, and web analytics areas at UPES Dehradun. His research interests include AI, ML, deep learning, image and video processing, computer vision, evolutionary algorithms, biomedical, cloud computing, high-performance computing, and IBM mainframe technology. He has worked as IBM Program Training Consultant at Edunet Foundation, Gurgaon in alliance with IBM and Directorate General of Training, Ministry of Skills Development and Entrepreneurship at National Skill Training Institute, Dehradun. He worked with NIIT Technologies Ltd as a Software Developer, IMS Unison University, and Quantum University as an assistant professor. He has published book chapters and research papers in reputed National & International Conferences and Journals.Corresponding author. Email: aloknegi.phd2020@nituk.ac.inKrishan KumarKrishan Kumar received his Mtech and PhD degrees from the Department of Computer Science and Engineering (CSE), VNIT Nagpur, Maharashtra, India. Presently, he is working as an assistant professor at the Computer Engineering Department, National Institute of Technology Kurukshetra, Haryana, India. Prior to this assignment, he spent about 8.5 years at NIT Uttarakhand as a faculty member where, he served as a coordinator of Campus Wide Networking from July 2016 to June 2019, as a Head of the department from March 2019 to February 2022 and as a Coordinator of Career Counselling and Placement from March 2022 to November 2022. He was also the chairman, coordinator, or special invitee in the various department and institute-level committees including the DFB, DPGC, DUGC, Purchase, and Library Committee, Senate, ACoFAR, etc.He hasmore than10 years of experience in teaching, administration, research, and Industry. He supervised 08 Mtech scholars and leading 04 PhD scholars. He authored 100+ technical research articles in reputed international conferences and journals, including IEEE and ACM Transactions with a citation of 1250+ with H-index 21. His research article was selected for the best student paper award at the IAPR endorsed International conference PReMI, ISI Kolkata, India, in December 2017. He received the Best Faculty Award 2022 for his significant contributions to teaching, research, and administration during the academic year 2021–22 at NIT Uttarakhand. He is a Senior Member of IEEE USA, a Life member of ISTE, IETE, and IUIAPR, Member of IET and ACM. Dr. Kumar is serving as an Editor at the IETE Journal of Education (UGC), Senior Editor at the Artificial Intelligence Evolution, Guest Editor at a special session of 29th ICONIP'22, Guest Editor at a special session of 04th ICDLAIR'22, Publicity Chair at 9th UPCON'2022, Organizing Secretary at IEEE IC2E3'23. His research interests include machine learning, deep learning, video processing, NLP, Multimedia analysis, and cloud computing.Email: k2b@ieee.orgParul SainiParul Saini received her Btech in information technology from Uttar Pradesh Technical University, India in 2007 and Mtech in computer science and engineering from Dr. B R Ambedkar National Institute of Technology, Jalandhar, Punjab, India in the year 2010. She has worked as an assistant professor in Dev Bhoomi Group of Institutions Dehradun, India. Currently, she is pursuing PhD in computer science and engineering from National Institute of Technology Uttarakhand, India. Her current research areas are image processing, deep learning, video processing.Email: parulsaini.phd2020@nituk.ac.in