Multimodal framework based on audio‐visual features for summarisation of cricket videos

IET Image ProcessingVolume 13, Issue 4 p. 615-622 Research ArticleFree Access Multimodal framework based on audio-visual features for summarisation of cricket videos Ali Javed, Corresponding Author Ali Javed ali.javed@uettaxila.edu.pk Department of Software Engineering, University of Engineering and Technology, Taxila, PakistanSearch for more papers by this authorAun Irtaza, Aun Irtaza Department of Computer Science, University of Engineering and Technology, Taxila, PakistanSearch for more papers by this authorHafiz Malik, Hafiz Malik Department of Electrical and Computer Engineering, University of Michigan, Dearborn, USASearch for more papers by this authorMuhammad Tariq Mahmood, Muhammad Tariq Mahmood orcid.org/0000-0001-6814-3137 School of Computer Science and Information Engineering, Korea University of Technology and Education, 1600 Chungjeolno, Byeogchunmyun, Cheonan, Republic of KoreaSearch for more papers by this authorSyed Adnan, Syed Adnan Department of Computer Science, University of Engineering and Technology, Taxila, PakistanSearch for more papers by this author Ali Javed, Corresponding Author Ali Javed ali.javed@uettaxila.edu.pk Department of Software Engineering, University of Engineering and Technology, Taxila, PakistanSearch for more papers by this authorAun Irtaza, Aun Irtaza Department of Computer Science, University of Engineering and Technology, Taxila, PakistanSearch for more papers by this authorHafiz Malik, Hafiz Malik Department of Electrical and Computer Engineering, University of Michigan, Dearborn, USASearch for more papers by this authorMuhammad Tariq Mahmood, Muhammad Tariq Mahmood orcid.org/0000-0001-6814-3137 School of Computer Science and Information Engineering, Korea University of Technology and Education, 1600 Chungjeolno, Byeogchunmyun, Cheonan, Republic of KoreaSearch for more papers by this authorSyed Adnan, Syed Adnan Department of Computer Science, University of Engineering and Technology, Taxila, PakistanSearch for more papers by this author First published: 06 March 2019 https://doi.org/10.1049/iet-ipr.2018.5589Citations: 8AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Sports broadcasters generate an enormous amount of video content on the cyberspace due to massive viewership all over the world. Analysis and consumption of this huge repository urges the broadcasters to apply video summarisation to extract the exciting segments from the entire video to capture user's interest and reap the storage and transmission benefits. Therefore, in this study an automatic method for key-events detection and summarisation based on audio-visual features is presented for cricket videos. Acoustic local binary pattern features are used to capture excitement level in the audio stream, which is used to train a binary support vector machine (SVM) classifier. Trained SVM classifier is used to label audio frame as an excited or non-excited frame. Excited audio frames are used to select candidate key-video frames. A decision tree-based classifier is trained to detect key-events in the input cricket videos that are then used for video summarisation. Performance of the proposed framework has been evaluated on a diverse dataset of cricket videos belonging to different tournaments and broadcasters. Experimental results indicate that the proposed method achieves an average accuracy of 95.5%, which signifies its effectiveness. 1 Introduction Exponential growth of sports videos production, sharing, and its availability in the cyberspace have sparked research activities to develop efficient video analysis and content management techniques. Analysis and consumption of available sports videos in the cyberspace is a challenging task. Video summarisation techniques [1, 2] are used to address the aforementioned challenges by providing a short synopsis video consisting of key-events. Motivation behind summarising sports videos are: the transmission requirements over low-bandwidth networks, storage cost, time constraints, and capturing viewer interest through exciting segments in the full-length video. Video summarisation techniques have been proposed for various sports such as soccer [3], tennis [4], baseball [5], cricket [6, 7], basketball [8], rugby [9], and so on. The cricket videos are more difficult to address from video summarisation perspective than any other sports as the cricket matches are of the longest duration with high-frequency key-events. This might be the reason behind limited focus on cricket video analysis and summarisation. Moreover, cricket has unique field rules (i.e. no whistles etc.), therefore, we need to observe the external factors [e.g. excitement in commentary, score-captions (SCs), logos etc.] for video summarisation. Existing state-of-the-art methods [2-5, 9-13] for sports video summarisation can be categorised into two broad categories, i.e. learning-based [2-5, 9, 10] and non-learning-based methods [11-13]. Learning-based methods [2-5, 9, 10] employ classification techniques to detect significant events for sports video summarisation. Learning-based techniques offer better results at the cost of increased computational complexity. Zawbaa et al. [3] proposed a learning-based video summarisation framework for soccer. Neural networks and SVM were trained to identify the logo frames for key-event detection. Javed et al. [10] applied the rule-based induction to identify the excited audio frames followed by using a decision tree to summarise cricket videos. Similarly, Midhu and Padmanabhan [6] detected various key-events to generate the summary of cricket videos. Besides learning-based methods, non-learning-based methods [11-13] have also been proposed for video summarisation. Non-learning-based methods exploit the game observations, e.g. logo placement, game transitions, whistle detection, and so on to find the clear patterns for precise video summarisation [3, 14]. Meanwhile, non-learning-based methods are slightly static in nature, therefore, are unable to detect key-events if the observational information is improperly detected [15]. Chen and Chen [11] used statistical features to propose a non-learning technique for replay detection in basketball videos. Nguyen and Yoshitaka [12] proposed a non-learning technique using histogram difference and contrast to identify the logo frames for replay detection. Existing sports video summarisation methods extensively use visual [16, 17] and audio features [18, 19] to detect the key-events. Visual features are usually computed through the estimation of colour [20], edges [21], texture [22], and salient-points [23]. Tavassolipour et al. [17] proposed a Bayesian network-based method for key-events detection and summarisation of soccer videos. Namuduri [24] applied a hidden Markov model (HMM) to generate the highlights for cricket videos. Histogram difference comparison was employed for shot boundary detection followed by performing the shot classification through MPEG-7 visual descriptors. At each frame, two histograms comparisons were performed that increases the computational cost of this method [24]. Wang et al. [25] proposed a framework for key-events annotation, event boundaries detection, soccer field classification, and whistle detection in soccer videos. Godi et al. [26] applied a deep convolutional neural network (CNN) to generate the highlights for ice-hockey. Similarly, Jiang et al. [27] applied a deep neural network based on CNN and recurrent neural network to detect four key-events in soccer videos. A lot of research works [9, 14, 28, 29] have used audio features for sports video analysis. Baijal et al. [9] proposed a Gaussian mixture model (GMM) based method using audio cues to summarise the sports videos. Mel-frequency cepstral coefficients (MFCC) features were used to train a GMM classifier into two stages. In the first stage, input audio signal was classified into speech and non-speech components. Whereas, in the second stage, speech components were processed to classify between the excited and non-excited segments. Rui et al. [14] used energy features, MFCC, and audio pitch to develop an efficient summarisation method for baseball videos. However, the performance of this method [14] degrades significantly due to the deficiencies recorded in pitch detection method. Xu et al. [28] applied the rule-based heuristics using energy features to train the SVM classifier to detect different events in soccer videos. The performance of events detection in [28] is largely dependent on audio keywords recognition accuracy. Existing techniques [30-32] have also combined the audio and visual features to summarise videos of various sports as fusion of the features result in improved accuracy. Kolekar and Sengupta [30, 31] have used audio-visual features to propose a hierarchical framework for sports video summarisation. Short-term energy features were used to analyse the audio stream of the input video. Whereas, colour and motion features were used to train a HMM to detect the replay segments. Raventos et al. [32] presented an automated framework for soccer video summarisation. For this, shot boundary detection was applied to segment the video into shots. Afterwards, low- and mid-level audio-visual descriptors were computed against each shot to acquire different relevance measures based on specific rules. Finally, these relevance measures were used in combination of user preferences to summarise the soccer videos. The existing video summarisation methods show the computational inefficiency in external factor identification, i.e. logo detection, placement of SCs, illumination changes, replay speeds, camera variations, and so on. However, the proposed method is robust against the aforementioned limitations. To reap the benefits of both learning- and non-learning-based methods, a hybrid approach is proposed in this paper to effectively deal with the summarisation of cricket videos. It has been observed that key-events result in a significant change in the excitement level of the audio stream. Likewise, key-events also result in change in the SCs for visual stream. A two-stage framework is proposed for key-event detection. The first stage uses audio stream to compute excitement score for each audio frame which is used to select the candidate key-event video frames. The second stage analyses the SCs of the candidate video frames to detect key-events in the cricket video. The proposed framework computes the length of each video skim against each key-frame to generate the summary of user-specified length. Moreover, SC and gradual transitions (GTs) are used to detect replay events that are also included in the summarised videos. A diverse dataset consisting of videos of different cricket tournaments and broadcasters are used for performance evaluation. Performance metrics such as precision, recall, accuracy, error rates, and F-1 score are used to evaluate the effectiveness of the proposed method. Effect of various recording parameters such as camera angle, SC design and placement, and replay speed variation on the performance of the proposed method has also been evaluated. Experimental results illustrate that the proposed method achieves an average accuracy of 95.5% which signifies its effectiveness for video summarisation. 2 Proposed framework The proposed video summarisation method exploits audio-visual cues for key-event detection. Specifically, audio features are used to detect excitement through commentary voices and crowd cheers that is then used for candidate key-video frame selection. Selected candidate frames are analysed further to detect the key-events. The proposed system, therefore, can be divided into two stages: first stage uses audio stream corresponding to a video frame to compute excitement through SVM classifier by representing audio frames in the form of feature vectors. The key-audio frames are used to select associated video frames. The second stage uses selected video frames to detect the key-events. The architecture of the proposed video summarisation system is presented in Fig. 1 that is described in detail in further subsections. Fig. 1Open in figure viewerPowerPoint Block diagram of the proposed framework 2.1 Excitement-based key-audio frame detection 2.1.1 Problem formulation Let be the audio signal having samples associated with a cricket video containing video frames represented as , and K video frames are associated with N excited audio frames/windows where and . For key-event detection, we analyse audio frames on the basis of audio recording sampling rate and representing them in the form of feature vectors. Once the excitement detection occurs through classification, we analyse the corresponding video frame for key-event detection. 2.1.2 Feature extraction Effective feature extraction is an indispensable requirement to achieve higher classification accuracy. Acoustic features provide numerical representation of the information present in the form of sound waves. For implementation of this work, we extracted the acoustic-local binary pattern (acoustic-LBP) features from the audio stream of the input cricket videos. As described in [33], acoustic-LBP is a fast and computationally inexpensive mechanism for signal representation that distinctively marks certain signal features. Another reason to utilise the acoustic-LBP is that the acoustic-LBPs are never employed in excitement detection based video summarisation research [9, 10]. The signal features in the form of linear LBP codes can be adopted for signal segmentation and thumb-impression generation. The LBP examines the neighbourhood of data samples from a signal and assigns an LBP code to each centre sample after thresholding them against the neighbouring samples [33]. Let be the central sample in the samples window with elements in audio signal , where The acoustic-LBP can be defined as (1)where the sign function S[·] is given by (2)In acoustic-LBP, the sample serves as a threshold for the neighbouring samples, and the sign function S[·] transforms the difference between and the neighbourhood as a P-bit binary code where P = 8 in our implementation. The binomial weights are then multiplied to the acoustic-LBP code and summed to generate the acoustic-LBP value for the sample (as shown in Fig. 2). The acoustic-LBP locally describes a sample using neighbourhood differences; for a constant signal these differences cluster near zero; whereas, at peaks and plateaus the difference is large. The acoustic-LBP codes are used to describe the local patterns as (3)where m = 1…z, and z describes the histogram bins corresponding to each acoustic-LBP code and is the Kronecker delta function. The parameter Q specifies the audio samples that correspond to the feature vector representing an audio frame that is defined as (4)where is the audio sampling rate (e.g. 48 kbps) and is the time interval that is set to 5 s in our implementation. Corresponding to the feature vector there are G video frames that are defined as follows: (5)where is the video frame rate (e.g. 25 fps). Fig. 2Open in figure viewerPowerPoint Acoustic-LBP calculation 2.1.3 Excited audio-frame detection We classify excited and non-excited audio clips through SVM classifier [34]. The training data consisting of M excited and non-excited audio features is prepared as: , where specifies the excited and non-excited audio classes. Hyperplanes linearly separating the two classes are given as (6)where is the weighting vector and b is the bias. The objective is to maximise the separation between two planes by minimising the norm which can be formulated as a quadratic optimisation problem (7)The two events (excited and non-excited) can then be detected by using the discriminating function such as (8)The excited audio-frames are used to select the corresponding video frames which are candidates for key-events. The second stage analyses only the candidate video frames for key-event detection. The benefit of the proposed two-stage approach is that it significantly reduces the processing time required to detect key-events in the input video. 2.2 Excitement-driven key-event detection and video summarisation At the second stage of video summarisation, video frames corresponding to the excited audio frames are processed further for key-event detection. Each step of the proposed framework is discussed in the subsequent sections. 2.2.1 Video-frame enhancement The excited colour video frames are transformed into greyscale and every tenth frame is processed. Top-hat filtering [35] is applied for illumination adjustment that performs morphological opening with a structuring element SE of size α followed by the difference operation expressed as follows: (9) (10)where and represent the morphed, and illumination adjusted frames, respectively. Whereas, is the opening operator. The size α of the structuring element SE is set to 3 × 3 in order to preserve the effectiveness of frame enhancement through illumination adjustment. 2.2.2 SC detection As the SCs appear at fixed location in the frames of input cricket video, therefore, temporal image averaging is used to filter-out the SC region from the video frames. A sliding overlapped window of length L frames and step size W is used to compute temporal running average sequence expressed as follows: (11)In (11), represents average at ith frame, and L is the length of sliding window. Morphological opening is performed on the extracted SC region to further enhance the SC contents followed by computation of the first- and second-order statistics that generates the binary image as follows: (12) (13) (14) (15)where and represent the mean and standard deviation for morphed frame , and represents the binary image for ith frame, whereas β is a positive real constant that is set to 2.5 after detailed experimentation. The parameter β is used to select the range of intensity to classify each pixel into either foreground or background region. To remove the outliers, two passes of morphological thinning are applied on the binary image to generate thinned image as (16)To bridge the gaps and preserving fine details in characters, dilation is applied on as follows: (17)where is the dilated image for ith frame, and ⊕ is the dilation operator. The transformed image finally contains the SC contents that are more suitable to be processed by the optical character recognition (OCR). The processed SC region obtained after the dilation operation is passed to the OCR algorithm [15] to recognise the characters. For OCR, connected component analysis is performed initially to store the outlines of the components together into blobs. These blobs are organised into text lines that are further partitioned into characters based on identifying the connected components. Finally, each character is passed to an adaptive classifier for recognition. 2.2.3 Decision-tree-based key-event detection In cricket, SCs display score and wicket information by using one of two separators '/' or '-', i.e. score/wicket or score-wicket. Therefore, in this work, SCs are processed for key-event detection by designing a five-layer decision tree to detect various key-events, i.e. boundary, six, wicket, and replay. The layout of the proposed decision tree is presented in Fig. 3. Fig. 3Open in figure viewerPowerPoint Decision tree for key-events classification As the broadcasters usually omit SCs during replays due to the disagreement of game stats in live and replay frames, therefore, in the present work we used this observation to classify the frames as either active/live or replay frames. Another observation about the replay segments is that the replay frames are sandwiched between GT frames. Therefore, the detection of GT in the absence of SC is the representation of replay frame. The frames marked as replay frames are also used for video summarisation, i.e. to capture any interesting event (e.g. misfields, overthrows, player collisions etc.) other than the key-events in the form of boundary, six, or wicket. For GT detection, we used a dual-threshold-based method that is proposed in our earlier work [15]. Detected GT frames are used to extract the candidate replay segments. Shown in Fig. 4 are the GT frames of the input cricket video. Fig. 4Open in figure viewerPowerPoint GT frames of crick1 video For replay event detection, a rule R2 is defined to classify between the replay and closed frames (CFs) as follows: (18)Absence of the GT and SCs marks the frames as CF, i.e. the frames that cannot be used for the key-events detection. If the SCs are found in active frames, then score separator (SS) is used for separating score value (SV) and wicket value (WV), respectively. The W counter contains the difference between WVs from the current and the previous frames, whereas S counter stores the difference between the SVs from the current and the previous frames. For wicket event, a rule R3 is defined as follows: (19)where W is expressed as (20)More specifically for wicket event, W counter must contain a value other than zero otherwise no key-event in the form of wicket is detected. Similarly, for boundary event a rule R4 is defined as follows: (21)where S counter represents the difference in the SV between the current and previous frame and expressed as follows: (22)More specifically, if S counter is greater than or equal to 4 and less than 6 then 'boundary' event is detected. The reason to use two thresholds, e.g. 4 and 6, is that: the boundary event can also occur on a no/wide ball and also in the form of misfield/overthrows and increments the score counter S by 5 instead of 4. Score counter is further analysed to detect the six-event by defining a rule R5 as follows: (23)More specifically, if S is greater than or equal to 6 then a six-event is detected. The six-event increments the score counter by a factor of either 6 or 7 in a cricket video. In case of 7, six is scored on a no-ball in cricket. For each detected event, the corresponding video frame is marked as a key-frame. A video skim is created for each key-event followed by generating the summarised video according to the user-specified length as proposed in our earlier work [10]. 3 Experiments and results This section presents a detailed discussion of the results and various experiments that are designed to evaluate the performance of the proposed video summarisation system. 3.1 Dataset Twenty real-world you-tube cricket videos of total duration of 10 h are used for performance evaluation. The dataset consists of videos from different sports broadcasters and tournaments following the similar approach adopted in sports video summarisation research [6, 7, 10, 30, 31]. Each video in the dataset has a frame resolution of 640 × 480 pixels and a frame rate of 25 fps. The cricket videos contain samples from 2006 One Day International (ODI) series between Australia and South Africa, 2014 test series between Australia and Pakistan, 2014 ODI series between South Africa and New Zealand, 2014 (T20) cricket world-cup tournament, and 2015 ODI cricket world-cup tournament. Some snapshots of our dataset are shown in Fig. 5. Fig. 5Open in figure viewerPowerPoint Snapshots of dataset As the proposed video summarisation method also performs the acoustic analysis of the audio stream associated with the input cricket videos, an audio dataset was created that consisted of excited and non-excited audio clips. For this, the annotation of the audio stream was performed, and the audio frames with loud commentator voices or crowd cheers were considered as the excited audio frames. Whereas, the other audio stream portions were considered as non-excited audio frames. The dataset consists of 100 audio clips with 50 audio clips in each class. Thirty clips from both excited and non-excited classes were used for classifier training, whereas, the remaining 40 clips were used to evaluate the classification. The dataset we developed is publically available at [36]. 3.2 Performance evaluation For performance evaluation of the proposed method, precision, recall, F-1 score, accuracy, and error rates are computed. In order to justify the effectiveness of the proposed scheme, we also compared our method against state-of-the-art techniques for summarisation of the cricket videos. 3.2.1 Performance evaluation of excitement-based key-audio frame detection Performance of the proposed video summarisation framework depends on accuracy of the underlying excitement-based key-audio frame detection. The goal of this experiment is to evaluate the performance of the proposed key-audio frame detection method. The results presented here are averaged over all the audio streams of the input cricket videos. The proposed excitement detection method can reliably detect the key-audio frames and achieved 97.76, 98.87, 97.60, and 98.7%, accuracy, precision, recall, and F-1 score, respectively. The higher evaluation scores are attributed to the robustness of acoustic-LBP features that enable SVM classifier to correctly classify excited and non-excited classes. 3.2.2 Performance comparison of acoustic-LBP features against spectro-temporal features We designed an experiment to compare the performance of acoustic-LBP features against the spectro-temporal features [e.g. MFCC, linear predictive coding (LPC)] by training the SVM classifier. Impact of acoustic-LBP features selection on classification performance is highlighted in Fig. 6. Fig. 6Open in figure viewerPowerPoint Performance comparison of excitement detection with LPC, MFCC, and acoustic-LBP features It can be observed from Fig. 6 that acoustic-LBP features significantly improve the classification performance of the excited audio frame selection method as compared to MFCC [9] and LPC [37] features. 3.2.3 Performance comparison of excitement detection against state-of-the-art methods In this experiment, we evaluated the performance of the proposed excitement detection method in terms of key-events detection against state-of-the-art methods [9, 18, 32]. The brief description of the comparative methods can be found in Section 1. The reason for selecting these methods for comparison is that these methods also perform the excitement detection in audio streams for summarising the sports videos. From the results presented in Table 1, we can observe that the proposed method has the highest precision, recall, and accuracy rates. The performance difference against [18] is even significant, where, our method achieved ∼18% higher accuracy rates that clearly elaborate the effectiveness of the proposed scheme. Table 1. Performance comparison of excitement detection methods against the proposed method Excitement detection methods Precision, % Recall, % Accuracy, % Baijal et al. [9] 97.37 96.77 — Merler et al. [18] — — 80 Raventos et al. [32] 88 93 — proposed method 98.87 97.60 97.76 3.2.4 Key-event detection evaluation In our second experiment, the proposed framework is evaluated in terms of key-events detection for cricket videos. The results are obtained by processing the candidate video frames selected by the excitement-based key-audio frame detection stage. The results presented here are averaged over all the 20 input cricket videos of our dataset. The objective evaluation statistics of the proposed system for each of the four key-events, i.e. boundary, six, wicket, and replay are shown in Table 2. Whereas, the graphical illustration of same experiment is presented in Fig. 7. The average precision, recall, accuracy, and error rate of 92.94, 91.04, 95.50, and 4.5% signifies the effectiveness of the proposed key-event detection method. Table 2. Key-event detection results for cricket videos Key-events Precision rate, % Recall rate, % F-1 score, % Accuracy rate, % Error rate, % boundary 95.33 90.64 92.92 92.17 7.83 six 91.78 89.33 90.53 96.08 3.92 wicket 90.47 92.68 91.56 98.04 1.96 replay 94.19 91.53 92.84 95.72 4.28 average 92.94 91.04 91.96 95.50 4.50 Fig. 7Open in figure viewerPowerPoint Objective evaluation of boundary, six, wicket, and replay events Confusion matrix (Table 3) is also provided to measure the classification performance of the proposed method for key-events detection. From confusion matrix it can be observed that the classification accuracy of the proposed system for key-events detection is remarkably well for all key-events. The slight degradation in the classification accuracy of boundary event is attributed to the fact that the appearance of advertisement bars in front of SCs. High accuracy of the replay detection, however, compensates this degradation. Table 3. Confusion matrix analysis Actual class Predicted class Boundary Six Wicket Replay Classes Positive Negative Positive Negative Positive Negative Positive Negative positive 182 21 66 09 37 04 292 27 negative 10 145 07 276 05 312 18 715 3.2.5 Performan