Effective key‐frame extraction approach using TSTBTC–BBA

IET Image ProcessingVolume 14, Issue 4 p. 638-647 Research ArticleFree Access Effective key-frame extraction approach using TSTBTC–BBA Yadwinder Singh, Corresponding Author Yadwinder Singh yadwinder.singh.shergill@gmail.com Computer Engineering Department, Punjabi University, Patiala, 147 001 IndiaSearch for more papers by this authorLakhwinder Kaur, Lakhwinder Kaur Computer Engineering Department, Punjabi University, Patiala, 147 001 IndiaSearch for more papers by this author Yadwinder Singh, Corresponding Author Yadwinder Singh yadwinder.singh.shergill@gmail.com Computer Engineering Department, Punjabi University, Patiala, 147 001 IndiaSearch for more papers by this authorLakhwinder Kaur, Lakhwinder Kaur Computer Engineering Department, Punjabi University, Patiala, 147 001 IndiaSearch for more papers by this author First published: 18 February 2020 https://doi.org/10.1049/iet-ipr.2018.6361Citations: 1AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Video summarisation and key-frame extraction play a vital role in video processing and content-based video analysis. Key-frame extraction facilitates rapid browsing and proficient video ordering in numerous applications such as multimedia, gaming etc. The application of 'effective key-frame extraction' can be used in the field of object detection and tracking, and it is found to be helpful in providing some sort of signal to visually impaired persons. In this study, the advantages of Thepade's sorted ternary block truncation coding (TSTBTC) along with binary bat algorithm (BBA) are used to overcome the limitations by extracting an effective key frame from the videos. Static images in the form of frames are extracted from input video database and processed through TSTBTC algorithm for calculating similarity measures amongst two consecutive frames. The input data is processed using four colour spaces, namely red–green–blue colour space, Kekre's LUV colour space, YCbCr colour space, YUV colour space and BBA are used to optimise the threshold value. Furthermore, five similarity measures indices are used to calculate the number of frames, and the result obtained shows that TSTBTC–BBA algorithm deployed in YUV colour space provides better results compared with other existing techniques regarding precision, F-measure and colour spaces. 1 Introduction In recent years, advancement in the field of fast-growing multimedia, World Wide Web data and editing has witnessed tremendous increase in real-life applications such as online streaming and video data [1]. These increases have led to huge demand for computers and digital video recorders for widespread proliferation of multimedia data and also provided more concentration to the field of video processing and summarisation [2]. Key-frame extraction plays an important role to summarise the video in quickest of time reducing the huge demand of resources required. Video is defined as an electronic medium and visual multimedia source displayed as continuous sequence of frames per second. It has become inextricable part of day-to-day life because of its vast applications such as distance learning, information system, data storage in digital libraries and E-commerce [3]. Furthermore, these applications can be extended in the field of object detection and human–computer interaction such as gaming [4]. With the tremendous use of visual data, it has also become an important to maintain confidentiality, integrity and authenticity of the visual data. There are number of techniques present to secure the images and encryption is one of them, which is a process of transforming an image to cryptic image by using a key [5]. Segmentation or summarisation is defined as a process of extracting characteristic features from video frames such as colour, shape and body motion [6]. Summarising the videos is a crucial task due to its complexity. The uncertainties in the format and length of the videos are the challenges faced while accessing them. The main step in the summarisation of the videos is the extraction of key frames. Researchers have developed various techniques for summarisation, which can be used in real-life domains such as video surveillance, consumer videos, sports and news. The output obtained is in the form of static and dynamic by key frames, or video skims [7]. The sequence of stationary images with constant specific angle is defined as key frames and sequence of dynamic images as moving skims. Static summarisation is defined as representative frames of the still image abstracts. Dynamic summarisation is a process of developing a short video, which comprises with information regarding scenes of the original video [8]. Static summarisation comprises of various classification steps such as sampling, shot segmentation and scene segmentation [9]. The key-frame extraction technique is found to be effective for object detection and extracting the effective features. The key frames can be extracted from the original video through pre-sampling or random selection. Extraction of key frames in video content involves selection of one or more frames, which represents the content of video and further processed for generating summaries. Furthermore, the video key frames extracted can be used to build the object models and recognise the type of the object. Several optimisation techniques are considered to detect the primary objects through image segmentation and key-frame extraction. Motivated by such applications, we propose a new fusional framework for extracting key frames. In this proposed framework, a key frame is extracted in the quickest of time by optimising number of frames in videos and improving precision rate of the system. This paper will be extended further to detect obstacles from the key frame selected and visually impaired person will be intimated. The main contributions of this paper are as follows: Proposal of a framework for key-frame extraction that combines Thepade's sorted ternary block truncation coding (TSTBTC) and binary bat algorithm (BBA). The fusional framework takes advantage of both block encoding method and heuristic algorithm to perform global optimisation for feature selection. Overcomes the problems of information redundancy and computational complexity in the existing techniques. Use of BBA will provide an optimal solution in the quickest of time. The organisation of the rest of this paper is as follows: Section 2 presents the related work. Section 3 describes the detailed research methodology, experimental analysis and results and discussion. Section 4 concludes this paper and also presents future work. 2 Related works A wide range of review is available in the area of 'key-frame extraction' for video summarisation (VSUMM) and browsing. Various authors have developed different algorithms for key-frame extraction and techniques deployed will be explained as follows. Pan et al. [9] proposed an improved algorithm based on fuzzy C-means clustering for the extraction of key frames. From each class, the frame with the highest entropy is selected as the key frame by the algorithm based on the property that in each class the clustering content is nearly consistent. It was observed that the proposed algorithm could determine the key-frame number when compared with the traditional method, where it is fixed. It could also extract the key frames depending on the colour and subshot distribution of the video. Furthermore, a new approach comprising Kanade–Lucas–Tomasi (KLT) algorithm for moving object tracking has been proposed by Anvaripour et al. [10]. Anvaripour et al. have considered spectral residual technique to track multiple objects and KLT algorithm to discuss the optical flow. The results obtained from this paper show that the proposed algorithm is effective in detection and tracking the moving objects and acceptable accuracy rate of 94% is obtained for Olympic sport and 93% for aircraft database. TSTBTC-based key-frame extraction techniques with the help of seven assorted similarity measures were developed by Thepade and Patil [11]. The main idea of this technique is the extraction of key frames with the help of block encoding. It was observed that the experimental results gave the best completeness in the case of Euclidean distance when compared with the other distances. The author has considered a three-phase approach for analysis of video, crime prediction and mapping. K-means clustering algorithm is used to generate a codebook for chain features; decision function for test sample is developed through support vector machine classifier and crime hot spot is modelled using fuzzy logic and artificial neural networks. It has been observed that 95% confidence level has been used for experimental analysis and result obtained in terms of z-score of 1.96 and P-value of 0.05 shows that developed framework is successful in monitoring crime hot spot patterns. An automatic technique comprising structural similarity (SSIM) index and visual features for VSUMM has been proposed by Sandhu and Agarwal [12]. The key frames such as red–green–blue (RGB) channel, colour histograms, moment of inertia and texture has been extracted from the video through visual features and by measuring similarity amongst two frames can be used for dividing video into frames and eliminate redundant frames. It has been observed from this paper that a high-accuracy rate of 0.79 is obtained with less error rate of 0.35. In future, the above technique can be improvised by incorporating semantic features and additional feature sets. A new framework comprising cloud-assisted transmission of video and sharing has been developed by Hossain et al. [13] and Yaseen et al. [14]. Various scenarios including surveillance and recorded videos have been considered as input; genetic algorithm is used to extract the key frames from video and watermarking. From this paper, the results obtained in terms of SSIM >0.97 shows that the video is effectively transmitted through high level of security. Furthermore, this application is extended to support real-time environment in obstacle detection. The evolution of image data processing has led to replacing text-based retrieval systems to video retrieval systems. A new scheme comprising ordered-Dither block truncation coding (ODBTC) technique has been used to enhance the quality of real-time video (RTV) retrieval and segment video into frames [15]. The image frames captured are processed through ODBTC feature extractor to create colour co-occurrence and bit pattern features of video image frames. The study on experimental analysis shows that approximate preciseness of 0.7 is obtained with 0.9 recall factor. The advantages of block truncation coding along with TST has been used to develop a content-based model for video retrieval [16]. The problems arising in traditional methods such as indexing and retrieval of video will be addressed and considered by author while developing TSTBTC-2 algorithm. From this paper, it has been observed that 17 similarity measures divided across five families and further processed to evaluate the performance of the design and results shows that percentage accuracy rate of 64.5% is observed in Euclidean distance. The growth in multimedia technology and big data has led to store and extract huge amount of video data throughout the world. Key frame demonstrates main context in VSUMM, and a novel-based algorithm has been proposed by Zhang et al. [17] and Badre and Thepade [18] to effectively extract key frames using Thepade's sorted quaternary block truncation coding (TSQBTC), Thepade's sorted septnary block truncation coding (TSSBTC), Thepade's sorted pentnary block truncation coding (TSPBTC), Thepade's sorted octanary block truncation coding (TSOBTC) and Thepade's sorted hexnary block truncation coding (TSHBTC) algorithms. Each frame extracted from input video database is processed through individual algorithm, and vector similarity measure such as square–chord (SC) distance, mean square error (MSE), Canberra distance and fidelity distance are used to calculate the distance amongst two frames. From this paper, it has been observed that TSPBTC provides better accuracy rate of 73.26% compared with other existing techniques. Table 1 compares existing studies and discussed their merits and demerits. Table 1. Merits and demerits of the existing key-frame extraction techniques Author/year Technique Merits Demerits Pan et al. [9] improved fuzzy C-means clustering 1. better robustness 1. accuracy only possible when clustering centroid is close to the sample 2. can get better results in case of the presence of outliers 2. sensitive in the iteration process Anvaripour et al. [10] KLT algorithm 1. novel method for moving object tracking in different scales 1. time-consuming process Thepade and Patil [11] TSTBTC 1. less number of resources are required in processing video 1. information redundancy 2. computational complexity Sandhu and Agarwal [12] SSIM index 1. works well for videos of varying illumination 1. difficult to capture all visual features of a video frame 2. low redundancy 2. low-quality summaries Hossain et al. [13] cloud-assisted transmission and sharing 1. secure 1. does not work on different video formats 2. reliable 2. no support for RTVs 3. noise effect ignored Yaseen et al. [14] cloud-based scalable object detection and classification 1. minimum human interaction required 1. system is less generic 2. do not recognise objects of different class 2. coping with an increased volume of data 3. less speed and scalability Borkar and Katariya [15] halftone block truncation coding 1. better video image quality 1. high complexity Thepade and Yadav [16] similarity measurement criteria 1. efficient storage aspect 1. fidelity family's similarity measures are not suitable for applications such as colour content-based video retrieval 2. more accurate 3. fast Zhang et al. [17] layered directed acyclic graph 1. accurate results 1. high complexity 2. time-consuming Badre and Thepade [18] Thepade's sorted n-ary block truncation coding 1. good accuracy 1. accurate only on limited datasets 2. less complexity 3 Research methodology In this paper, TSTBTC technique along with BBA has been proposed for optimising the number of frames in videos and improving the precision rate of the system. The database comprises of two datasets: the YouTube dataset and the Open Video dataset. One RTV is also taken for testing our approach. The video sequence captured is decomposed into several shots and further processed to frame conversion block for key-frame extraction. The dominant colours of the shot with similarity indices are considered for key-frame extraction. Five colour spaces are used to represent digital images on the screen. Furthermore, threshold value is calculated and set to certain limit for selecting effective key frames and processed through TSTBTC algorithm for predicting the number of frames. Fig. 1 shows block diagram along with detailed explanation for the proposed system. The proposed methodology for key-frame extraction is divided into five stages. First stage deals with the input database considered for key-frame extraction. Second stage explains about the process of key-frame extraction and implementation of BBA for key feature optimisation. Third stage states about calculation of threshold voltage. Fourth stage states implementation of TSTBTC algorithm using different similarity measure techniques followed by calculation of output. The detailed explanation of the following blocks will be explained as follows. Fig. 1Open in figure viewerPowerPoint Block diagram of the proposed system 3.1 Dataset The input database comprises of two videos V23, V66 from the Open Video dataset and two videos, namely V85, V101 from the YouTube dataset. Another input video comprising real-time visuals is used to show the real-time performance of the system. These videos are captured in structural format in a hierarchical process of video clips, shots, scenes and frames. A shot is defined as a set of consecutive frames and number of frames are visual input data captured through camera for certain time interval. This input database is further processed through MATLAB software for key-frame extraction. 3.2 Key-frame extraction In this step, key frame is extracted from input video for calculating feature information. The colour is the most widely accepted and key visual feature in analysing video. In this paper, it is assumed that input video comprises of 'V' number of frames and each frame is converted into key-frame extraction using four colour spaces. The consecutive frames are given by Fn and Fn + 1. The differentiation among individual frame is obtained through TSTBTC algorithm along with similarity indices. The four colour spaces considered for this paper are RGB, Kekre's LUV, YCbCr and LUV colour spaces [19, 20]. 3.3 Threshold value The threshold value is defined as the maximum acceptance value, in which the key frame is accepted from the image. In this paper, two conditions are considered to calculate the threshold value. 3.3.1 Condition 1: without BBA The threshold value is calculated using mean value and multiplication of constant G. The threshold value is calculated and compared amongst consecutive video frames for key-frame acceptance and the formula used for calculating the same is considered from [21], and it is given by (1) (2) (3)where T is the threshold value, X is number of frames, Diff (x) is the difference amongst two frames, M is the mean, G is the constant, in this paper, we have considered value as 4 and σ is the standard deviation. 3.3.2 Condition 2: including BBA for optimisation BBA is an optimisation algorithm developed on the basis of echolocation behaviours of the bats. The main theme considered during the development of this algorithm is population of similar individual bats that fly randomly in air in search of their target using echolocation principle to detect the prey by considering surrounding environment. Each individual frame 'i' comprises of an encoded frequency (f), velocity (v) and threshold value (ti). The internal variables will be updated at each individual iteration on the basis of personal, group history and frequency. The equation for individual iteration is considered from [22] and it is given by (4) (5) (6)where β is the random vector ranged from 0 to 1; t_best is the best threshold fitness value obtained from the algorithm; fi is the frequency of i iteration, vi is the velocity, fmin is the fixed frequency with which bat flies and fmax is the path cost function. Furthermore, as the bat approaches the target, the loudness value (A) will be decreased and the pulse rate (r) will be increased, as shown in the equations below: (7) (8)where α and γ are constants which are adjusted to increase the accuracy of selecting threshold value. The α value will be ranged from 0 to 1 and γ value is selected such that it is always >0. Also, uniform random walks equation developed by Yang is considered to improve the exploratory component in pulse rate. It is given by the equation below: (9)where x is the random number that varies from −1 to 1 and A×x is the average loudness value of the bat at iteration x. The pseudocode for the proposed bat algorithm is given below (see Fig. 2). Fig. 2Open in figure viewerPowerPoint P seudocode for the proposed bat algorithm The threshold equation by considering the BBA algorithm is given by the equation below: (10)where M is the mean, G is the constant and is considered as 4, σ is the standard deviation and global best is the location of the solution provided by bats. First, we will calculate the number of frames though normal threshold equation, and in another step, we will calculate threshold value through optimised BBA threshold equation. 3.4 TSTBTC algorithm In TSTBTC algorithm, the intensity values of the coloured component are considered and extracted from individual frame of the video. Then, these features are arranged on the basis of intensity values and sorted in ascending order. Feature set obtained is further divided into three parts on the basis of individual colour component. Suppose we consider 'B' as video having red, blue and green, and consider size of frame is x×y. The total intensity x×y is represented in a single-dimensional array with indices ranged from 1 to x×y. This Single Dimensional Array for Intensity of R Component (SDR) can be rearranged in ascending order as 'sorted SD' and red feature is calculated using the following equation [12]: (11) (12) (13)Similarly, these feature equations are applied to green and blue components (Single Dimensional Array for Intensity of G Component (SDG) and Single Dimensional Array for Intensity of B Component (SDB)) to calculate the feature set, and they are as follows: (14) (15) (16) (17)Thus, the end output video frame feature obtained from TSTBTC algorithm will be in the forms of IR, mR, uR, IG, mG, uG, IB, mB and uB. These features are further processed through similarity measure techniques to calculate the difference amongst two frames. Five similarity measures are considered to calculate, compare and predict the best effective similarity technique for key-frame optimisation. The five techniques considered for analysis are captured from [20] and are as follows. 3.4.1 Mean square error MSE is defined as the statistical method or technique used for estimating the unknown quantity, which mathematically is represented as (18)where Bi and Ci are the consecutive frames. The difference amongst reference image and target image will be in the range of fraction and number of frames. 3.4.2 Euclidean distance Euclidean distance is the ordinary straight line used to measure distance amongst two points in Euclidean space. The equation for Euclidean distance is given by the equation below: (19) 3.4.3 SC distance It is a popular distance technique used by palaeontologists and in studies related to calculating the similarity amongst two objects. The equation for SC is given below: (20) 3.4.4 Canberra (C) In Canberra distance measurement, the distance amongst a pair of values is measured on the basis of numerical values in vector database. The Canberra is calculated through the equation below: (21) 3.4.5 Sorensen distance Sorensen method is the statistical technique used to differentiate amongst two similaritysamples. The equation used in calculating Sorensen distance (SD) is given by theequation below: (22) These five similarity measure techniques are applied alongwith TSTBTC algorithm to compare the similarity amongst two consecutiveframes and optimise the number of frames. The algorithm for the proposed system comprising key feature extraction and TSTBTC algorithm isshown in Algorithm 1 (see Fig. 3). Fig. 3Open in figure viewerPowerPoint Algorithm 1 for the proposed system 3.5 Experimental analysis In this paper, we have considered TSTBTC algorithm with BBA for optimised key feature extraction. Implementation is carried out using MATLAB R2017b software and processed through number of stages to evaluate the performance using with and without BBA algorithms. The videos used in the experiment are of MP4 extension formats, which are processed to calculate the number of frames. Furthermore, the performance metrics such as precision, recall and F-measure are used to calculate the effectiveness and quality of the key-frame extraction. 3.6 Results and discussion To gauge the performance of the proposed optimisation method in key-frame extraction, we compare with and without BBA for optimisation, and it is evaluated using five similarity metric algorithms. MATLAB 2017b framework is used for software analysis and the result obtained through individual colour space will be tabulated in Excel sheet and further used to calculate precision, recall and F-measure. Video V23 of Open Video dataset and V85 of YouTube dataset are taken as inputs for comparing the results of the various key-frame extraction techniques and then the proposed method is applied also on the RTV to check the performance of the proposed system. For image comparative analysis, some images are extracted from the user captured from input online database, some images from 'mutual information-based agglomerative hierarchical clustering' (MIAHC) technique proposed by Fei et al. [23] and some from the results obtained through the proposed method. It is observed that the images extracted from the proposed method are effective and efficient compared with other existing techniques. Furthermore, the key frames extracted from the proposed method are compared with the existing technique and user by using same input dataset, as shown in Figs. 4 and 5. Fig. 6 shows the key frame extracted from the RTV. It is observed that the key frames extracted from the proposed system are better as compared with other existing techniques. Fig. 4Open in figure viewerPowerPoint Key-frame results obtained for video 23 (Open Video dataset) Fig. 5Open in figure viewerPowerPoint Key-frame results obtained for video 85 Fig. 6Open in figure viewerPowerPoint Key-frame extraction from RTV The key frames are extracted using different colour spaces with and without TSTBTC algorithm, and the same are shown in Tables 2–5. Total number of frames are the key frames extracted without TSTBTC algorithm. Actual number of key frames is calculated using TSTBTC algorithm and predicted key frames are obtained by applying BBA optimisation algorithm for extracting key frames. The key frames extracted will be compared with different colour spaces and similarity measurement techniques. Furthermore, a metric equation is used to calculate the accuracy and effectiveness of the system. The results obtained will be shown in further stages as shown in Tables 6–9. The metric values in terms of precision, recall and F-measure for three sets of input videos along with RTV captured through the camera are shown in Tables 6–9. The graph is plotted by considering average best so far values in the x-axis and number of iterations in the Y-axis. Table 2. Key-frame prediction in RGB colour space Sl. no. Total number of frames TSTBTC with ED TSTBTC with MSE TSTBTC with SC TSTBTC with CD TSTBTC with SS Actual key frames (without BBA) Predicted key frames (with BBA) Actual key frames (without BBA) Predicted key frames (with BBA) Actual key frames (without BBA) Predicted key frames (with BBA) Actual key frames (without BBA) Predicted key frames (with BBA) Actual key frames (without BBA) Predicted key frames (with BBA) V23 3454 457 371 494 401 185 170 292 264 526 441 V85 3152 479 464 474 440 224 201 483 435 470 415 V101 4612 402 384 411 407 32 31 559 498 450 417 ED – Euclidean Distance, CD – Canberra Distance, SS – Sorensen Distance Table 3. Key-frame prediction in YCbCr colour space Sl. no. Total number of frames TSTBTC with ED TSTBTC with MSE TSTBTC with SC TSTBTC with CD TSTBTC with SS Actual key frames (without BBA) Predicted key frames (with BBA) Actual key frames (without BBA) Predicted key frames (with BBA) Actual key frames (without BBA) Predicted key frames (with BBA) Actual key frames (without BBA) Predicted key frames (with BBA) Actual key frames (without BBA) Predicted key frames (with BBA) V23 3454 448 372 425 372 178 159 450 373 443 341 V85 3152 480 430 482 447 224 216 494 411 481 426 V101 4612 399 395 429 419 33 28 444 442 430 421 Table 4. Key-frame prediction in YUV colour space Sl. no. Total number of frames TSTBTC with ED TSTBTC with MSE TSTBTC with SC TSTBTC with CD TSTBTC with SS Actual key frames (without BBA) Predicted key frames (with BBA) Actual key frames (without BBA) Predicted key frames (with BBA) Actual key frames (without BBA) Predicted key frames (with BBA) Actual key frames (without BBA) Predicted key frames (with BBA) Actual key frames (without BBA) Predicted key frames (with BBA) V23 3454 457 401 494 435 185 167 292 256 526 447 V85 3152 479 438 474 444 224 198 483 439 470 390 V101 4612 402 394 411 397 32 31 559 516 450 424 Table 5. Key-frame prediction in K-LUV colour space Sl. no. Total number of frames TSTBTC with ED TSTBTC with MSE TSTBTC with SC TSTBTC with CD TSTBTC with SS Actual key frames (without BBA) Predicted key frames (with BBA) Actual key frames (without BBA) Predicted key frames (with BBA) Actual key frames (without BBA) Predicted key frames (with BBA) Actual key frames (without BBA) Predicted key frames (with BBA) Actual key frames (without BBA) Predicted key frames (with BBA) V23 3454 457 423 494 451 185 167 292 264 526 447 V85 3152 479 427 474 451 224 195 483 439 470 402 V101 4612 402 391 411 393 32 30 559 498 450 424 Table 6. Performance evaluation in RGB colour space Sl. no. TSTBTC with ED TSTBTC with MSE TSTBTC with SC TSTBTC with CD TSTBTC with SS P, % R, % F, % P, % R, % F, % P, % R, % F, % P, % R, % F, % P, % R, % F, % V23 81.18 68.