No‐reference quality assessment for live broadcasting videos in temporal and spatial domains

IET Image ProcessingVolume 14, Issue 4 p. 774-781 Research ArticleFree Access No-reference quality assessment for live broadcasting videos in temporal and spatial domains Yipo Huang, Yipo Huang School of Information and Control Engineering, China University of Mining and Technology, Xuzhou, 221116 People's Republic of ChinaSearch for more papers by this authorLeida Li, Corresponding Author Leida Li reader1104@hotmail.com School of Artificial Intelligence, Xidian University, Xi'an, 710071 People's Republic of ChinaSearch for more papers by this authorYu Zhou, Yu Zhou School of Information and Control Engineering, China University of Mining and Technology, Xuzhou, 221116 People's Republic of ChinaSearch for more papers by this authorBo Hu, Bo Hu School of Information and Control Engineering, China University of Mining and Technology, Xuzhou, 221116 People's Republic of ChinaSearch for more papers by this author Yipo Huang, Yipo Huang School of Information and Control Engineering, China University of Mining and Technology, Xuzhou, 221116 People's Republic of ChinaSearch for more papers by this authorLeida Li, Corresponding Author Leida Li reader1104@hotmail.com School of Artificial Intelligence, Xidian University, Xi'an, 710071 People's Republic of ChinaSearch for more papers by this authorYu Zhou, Yu Zhou School of Information and Control Engineering, China University of Mining and Technology, Xuzhou, 221116 People's Republic of ChinaSearch for more papers by this authorBo Hu, Bo Hu School of Information and Control Engineering, China University of Mining and Technology, Xuzhou, 221116 People's Republic of ChinaSearch for more papers by this author First published: 19 February 2020 https://doi.org/10.1049/iet-ipr.2019.1195Citations: 1AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract Nowadays, live broadcasting video has become increasingly popular and high-quality live broadcasting video is highly needed. In practice, live broadcasting videos usually undergo several processing stages, which inevitably introduce multiple distortions, e. g. frame freezing and intensity mutation, causing the degraded quality of experience. However, little work has been done to the quality evaluation of live broadcasting videos, which may hinder the further development of more advanced live broadcasting video delivery systems. Motivated by this, this study presents a no-reference quality evaluation model for live broadcasting videos (LBVQA) in temporal and spatial domains. In the temporal domain, statistic features are extracted to measure the frame freezing and intensity mutation, and the entropy-based feature is extracted to describe the global jitter. In the spatial domain, blurring is measured based on phase coherence, and abnormal exposure ratio is calculated based on an adaptive threshold. Finally, all features are fed into a backpropagation neural network to train the quality prediction model. Experimental results on the Live Broadcasting Video Database demonstrate the advantages of the proposed metric over the state-of-the-art image and video quality metrics. 1 Introduction With the widespread of mobile internet, a variety of mobile streaming media such as live broadcasting videos, video calls and video conferences, are recorded and shared in daily life [1-3]. Among them, live broadcasting videos are the most common type filling the lives of billions of people. Hence, the demand for high-quality live broadcasting video is becoming more and more urgent. In practice, live broadcasting videos usually undergo several stages before reaching the end users, including acquisition, pre-processing, encoding, transmission, decoding and post-processing [4]. Each stage may introduce certain types of distortions. For example, during the transmission, frame freezing (FF) is caused by network delays, and blurring may occur due to the limited resolution of the lens. Therefore, multiple types of distortions may be introduced to the live broadcasting videos, which in turn degrade the quality of experience (QoE) of the end users. An accurate quality evaluation model can be used to monitor the delivery of live broadcasting videos, based on which real-time optimisation of the broadcasting system can be achieved. However, the existing quality metrics are limited in evaluating the quality of the live broadcasting video, which is usually distorted by multiple types of real-world realistic distortions. Therefore, it is highly desired to design accurate metrics to evaluate the quality of live broadcasting videos. Motivated by the above facts, this paper presents a no-reference quality assessment metric for live broadcasting videos, which mainly measures the dynamic and static distortions in the temporal and spatial domains (S-D). In the temporal domain, considering that the continuity and coherence of the information conversion between time and space are usually destroyed by FF and global jitter (GJ), an effective measure is defined based on inter-frame difference (FD), and entropy-based feature is extracted to portray the global motion information based on optical flow. Then intensity mutation (IM) is quantified by calculating the standard deviation of the average intensity of each frame based on the human visual perception of intensity changes. In the S-D, considering the human visual characteristics of frequency, intensity and colour, blurring is first measured based on phase coherence, and abnormal exposure ratio is calculated based on an adaptive threshold. Then the colour richness is measured based on RGYB colour space. In addition, the information entropy and resolution of each frame are also used as quality features based on visual integration effect. Finally, all extracted features are utilised to train a backpropagation (BP) neural network regression model for quality prediction. The experimental results on the Live Broadcasting Video Database (LBVD) [5] show the superiority of the proposed metric over the existing metrics. 2 Related work In the past decade, a number of video quality assessment (VQA) metrics have been reported. According to the availability of a reference video, the current VQA metrics can be classified into full-reference (FR), reduced-reference (RR), and no-reference (NR) metrics [6]. In [7], Sheikh et al. proposed an FR-VQA metric based on the natural image statistical model and the human visual system (HVS) model. In [8], Wu et al. proposed an objective FR-VQA metric based on the degradation along attended motion trajectories. One of the pioneering work on RR-VQA was proposed in [9], which extracts extensive features related to the luminance and colour video components from both reference and distorted videos, such as Sobel filtering, Laplace filtering, first-order difference, colour distortion measurements, moment calculations and fast Fourier transformations. In [10], Gunawan et al. proposed an RR-VQA metric based on a local harmonic strength method. In [11], Rohani et al. applied a weighting map based on different motion speeds of objects in each frame and combined it with a divisive normalisation transform to generate the video quality score. In [12], a spatial–temporal RR-VQA metric was proposed based on motion activity density. In [13], Ma et al. calculated an energy variation descriptor in the space domain and a histogram distribution of the difference between frames in the temporal domain. In [14], a hybrid quality assessment metric was proposed by combining bitstream and pixel analysis. Kim et al. [15] proposed an NR-VQA metric by measuring brightness, motion, contrast, edge, and chrominance. Saad et al. [16] proposed the NR Video BLind Image Integrity Notator using DCT Statistics (BLIINDS) index by extracting the spatio-temporal statistic features in the DCT domain. In [17], Mittal et al. proposed the video intrinsic integrity and distortion evaluation oracle (VIIDEO) metric by measuring the statistical naturalness of the distorted video. The above VQA metrics have achieved notable success in revealing the influence of the synthetic distortion on video quality. However, these metrics are limited in evaluating the quality of the live broadcasting video, which is usually distorted by multiple types of real-world realistic distortions. First, the existing VQA metrics are mainly designed and tested on synthetic distortions. For example, the synthetic blur is generated by applying the Gaussian low-pass filtering on the frames of high-quality pristine videos [18], so it is simple and distributed uniformly in the frames. However, in the live broadcasting video, the blur is much more complex. Second, the existing VQA metrics are usually designed for a specific type of distortion, such as motion blur or FF [19]. However, live broadcasting videos are typically distorted by multiple distortions simultaneously. Finally, in addition to the common distortions such as FF, GJ and real blurring, IM is usually a key distortion for the live broadcasting video [20]. Therefore, the distortion of the live broadcasting video is much more complex, which poses a great challenge to the design of live broadcasting video quality evaluation metrics. 3 Proposed NR quality metric for live broadcasting videos The HVS is a multichannel model and usually perceives the quality of video from both dynamic and static information simultaneously [21]. From the dynamic perspective, the HVS can capture the information conversion between time and space, and content information can be captured from static aspect based on the human visual characteristics of intensity, frequency and colour [8]. Inspired by this, this paper presents a NR quality assessment metric for live broadcasting videos, which mainly measures the dynamic and static distortions in the temporal and S-Ds. Fig. 1 shows the framework of the proposed method. Specifically, considering that the continuity and coherence of the information conversion between time and space are usually destroyed by FF and GJ in the temporal domain, an effective measure is defined based on inter-FD and entropy-based feature is extracted to portray the global motion information based on optical flow. Then IM is quantified by calculating the standard deviation of the average intensity of each frame based on the human visual perception of intensity changes. In the S-D, considering the human visual characteristics of frequency, intensity and colour, blurring is first measured based on phase coherence, and abnormal exposure ratio is calculated based on an adaptive threshold. Then the colour richness is measured based on RGYB colour space. In addition, the information entropy and resolution of all frames are also used as quality features based on visual integration effect. Finally, a BP neural network is trained to predict the quality of the live broadcasting video. Fig. 1Open in figure viewerPowerPoint Framework of the proposed LBVQA metric 3.1 Temporal domain feature extraction Ideally, high-quality live broadcasting videos are usually fluent and stable. In practice, due to the network delay and unstable live broadcasting environment, the most common distortions consist of FF, IM and GJ. In the following subsections, we detail the feature extraction for each of the distortions. 3.1.1 Frame freezing The visual staying phenomenon is the basis for video play. When consecutive frames are switched at a very short time interval (denoted by t), a fluent video will be perceived by the HVS. In other words, when the time interval is longer than t, the freezing phenomenon is perceived by the HVS. Inspired by this, FF is measured based on the inter-FD. First, video frames are obtained with a sampling rate as (1)Then the FD is defined as follows: (2)where is the pixel value at (i, j) of the kth frame. When the two frames are exactly the same, the value of is 0. FF is defined as follows (3)where represents the kth frame is freezing. Fig. 2 shows the FD and FF of four videos with different freezing levels. We can observe from Fig. 2 that the degree of FF can be measured using the values of FF. Intuitively, as the degree of freezing increases, the number of gradually increases. Therefore, according to the number of , the ratio of freezing duration (RFD) can be easily calculated as (4) Fig. 2Open in figure viewerPowerPoint FD and FF of four videos with different freezing levels (a) Normal video, (b) Slight freezing, (c) Moderate freezing, (d) Heavy freezing In order to comprehensively describe the FF distortion, another two features based on the are further defined, which are the number of freezing (NOF) and the longest duration of freezing (LDF). In this work, the sampling frequency is 15, which is based on a large number of experiments by Ou et al. [22]. 3.1.2 Global jitter In the literature, optical flow has been widely used in capturing motion information [23, 24]. In this work, optical flow is used to capture object movement between two consecutive frames. With the Horn Schunck (HS) optical flow method [25], we assume that the pixel will move to the position of the next frame. The optical map is defined as [25]: (5)where is the kth optical map, is the HS optical flow operation symbol and the size of the optical flow map is . Fig. 3 visually illustrates the proposed feature for measuring GJ. For privacy reason, we did photographic mosaic. From Figs. 3b and e, it can be observed that the optical flow maps of the video with GJ have more information than the optical flow maps of the normal video. Next, the information entropy of each optical flow map is calculated as [18] (6)where is the information entropy of , L is grey level on the , and denotes the probability of occurrence of the ith grey level. Fig. 3Open in figure viewerPowerPoint Illustration of the proposed feature for measuring GJ (a) A normal video, (b) Corresponding optical flow maps of (a), (c) Corresponding feature values distribution of (a) and the GJ value is 0.21, (d) A distorted video with GJ, (e) Corresponding optical flow maps of (d), (f) Corresponding feature values distribution of (d) and the GJ value is 1.52, (g) A video with large displacement motion, (h) Corresponding optical flow maps of (g), (i) Corresponding feature values distribution of (g) and the GJ value is 0.38 In practice, in addition to videos with GJ, high-quality videos with large displacement motion such as sports video will also get high information entropy as shown in Fig. 3h. In order to address this problem, the difference of entropy (DE) between adjacent optical maps is further calculated as follows: (7)where , and K is the total number of frames in the video. From Figs. 3e and h, we can see that although the video with GJ and the video with large displacement motion both have high entropy of optical flow maps, the entropy difference of video with GJ is expected to be big, while the entropy difference of video with large displacement motion is usually small as shown in Figs. 3f and i. This is because that jittering video is characterised by irregular motion between adjacent frames, but the natural video is characterised by regular motion. Finally, the DE of all optical maps is averaged for evaluating the GJ as follows: (8) From Figs. 3c, f and i, it can be clearly observed that the GJ values of the normal video and the video with large displacement motion both are smaller than the video with GJ, which also confirms the effectiveness of the entropy difference feature for portraying the GJ of videos. 3.1.3 Intensity mutation In HVS, the rods of retina is sensitive to changes in light intensity. In the live broadcasting video, IM may be caused by unstable live broadcasting environment, which heavily degrades the QoE of the end users. In this work, the standard deviation of intensity is taken as a feature to measure IM. Specifically, the intensity of each frame is first determined, and then the standard deviation of the intensity of the entire video is obtained by (9)where K is the total number of frames in the video, is the intensity of kth frame and is the mean value of total frames. Fig. 4 shows the intensity distribution of two sequences with different mean opinion score (MOS) values. The MOS denotes the average value of human rating scores collected based on a subjective experiment. Generally, MOS is used as the ground truth of image quality in the IQA community. In Fig. 4, it can be easily found that the MOS of the video with uniform intensity is higher compared with video with IM. Fig. 4Open in figure viewerPowerPoint Intensity distribution of two sequences with different MOS values 3.2 S-D feature extraction In the S-D, the factors affecting the quality of live broadcasting videos mainly include devices and environment. Specifically, blurring and colour distortion may be introduced due to the limited performance of live broadcasting devices and abnormal exposure may be introduced by the over-bright or over-dark environment. In addition, the information entropy and resolution of all frames can describe the information richness and the delicateness of live broadcasting videos. The resolution (R) can be obtained directly from the parameters, and the feature extraction for other distortions is detailed in the following subsections. 3.2.1 Blurring Blurring may occur due to the limited resolution of the lens during the acquisition, which is a key determinant in the perception of video quality [26]. It has been demonstrated that the phase coherence metric can be employed in revealing the influence of blurring on image quality in the literature. In this part, a local phase coherence (LPC) metric [27] is used to evaluate the sharpness of the frames in the live broadcasting video. For a frame , the LPC strength can be computed based on a series of N-scale and M-orientation log-Gabor filter as follows [27]: (10)where denotes the real part of a complex number and the is a complex coefficient. Then the orientations that contain more energy are given by [27] (11)where c is added to avoid instability when the magnitudes of the coefficients are close to zero. The collection of at all location constitutes a spatial LPC map. Finally, a weighted averaging method is used based on the ranked LPC value. The overall LPC-based sharpness method is defined as follows [27] (12)where is the weight assigned to the kth ranked spatial LPC value and is computed as an exponentially decaying function given by (13)which gives a weight 1 to the highest LPC value and the decaying speed of the weights is controlled by the parameter , and the other parameters are set as . 3.2.2 Abnormal exposure In live broadcasting videos, abnormal exposure is caused by the over-bright or over-dark environment. According to the human visual characteristic of intensity, the HVS can only perceive a certain range of intensity [28]. In this work, two adaptive thresholds are calculated based on the maximum inter-class variance to divide the over-bright, normal and over-dark regions. Then, a window with a size of is used to generate the over- and under-exposure distortion masks according to the adaptive thresholds. Fig. 5 shows the flowchart of the proposed method for measuring over- and under-exposure rate. It can be observed that the coarse overexposure mask is generated according to the adaptive threshold. Fig. 5Open in figure viewerPowerPoint Flowchart of the proposed method for measuring over- and under-exposure rate. The overexposure rate of the coarse overexposure mask is 0.23. The overexposure rate of the fine overexposure mask is 0.18 Then the sensitivity of the HVS to different regions is considered, and the different distortion regions are weighted by the saliency map to obtain the fine overexposure mask. For saliency map computing, we employ the saliency detection by simple priors (SDSPs) model [29], which was proposed based on three priors. First, that the HVS detects salient object can be modelled by band-pass filters. Second, human eyes tend to pay more attention to the centre of an image. Third, people more likely to pay their attention to warm colours. Moreover, the SDSP model is very fast and has been shown effective for IQA [30]. Finally, the overexposure rate is obtained according to the area ratio of the fine overexposure mask. Obviously, the overexposure rate of the fine overexposure mask is more consistent with the judgment of HVS on the overexposed area. Finally, abnormal exposure is defined as: (14)where K is the frame number of the video, is the pixel number in the overexposure mask of kth frame, is the pixel number in the underexposure mask of kth frame and is the number of pixel of kth frame. 3.2.3 Colour and information richness Colour richness is an important factor affecting the quality of live broadcasting videos, which is usually dependent on the performance of image sensors and live broadcasting environment [31]. High-quality video is usually richer in colour on the basis of the human visual characteristics of colour. In this work, the colour colourfulness index (CCI) is used to evaluate colour distortion. To calculate CCI, we first need to convert the input RGB image into the RGYB colour space, which is defined as (15) (16) Then, the CCI is calculated as [31] (17) (18) (19)where and represent the variances in the RG and YB colour space, respectively, and represent the corresponding mean values. On the other hand, information entropy is a measure of the richness of image information. In general, a larger information entropy represents a richer amount of information and higher quality of image [32]. In this work, the information entropy is calculated as [18] (20)where is the information entropy of the kth frame, L is the grey level on the kth frame, and denotes the probability of occurrence of the ith grey level. Finally, the information entropy of all frames is averaged for evaluating the information richness as follows: (21)where K is the total number of frames in the video. To summarise, a sum of ten features is extracted elaborately tomeasure seven distortions in the temporal and S-Ds. For clarity, all theabove-described features are listed in Table 1. Table 1. Summary of extracted features of the proposed LBVQA metric Component Distortions Feature symbol temporaldomain FF RFD, NOF, LDF GJ GJ IM IM S-D blurring abnormal exposure AE resolution R colour and informationrichness CCI, IR 3.3 BP-based quality model In this work, a total of ten quality-aware features are extracted for each video, including five features in the temporal domain and five features in the S-D. In order to combine them to generate an overall quality score, we employ the AdaBoosting BP neural network [33] to learn the quality model. Then the trained BP model is utilised to predict the quality score of the distorted video. In the implementation, the BP model has an input layer, two hidden layers and an output layer. Further details of the BP model can be found in [33]. 4 Experimental results and analysis 4.1 Experiment settings In this section, we conduct experiments on the LBVD database [5] to verify the performance of the proposed LBVQA metric. All the videos in the database were collected from public Chinese live broadcasting websites. Thousands of videos were gathered and cropped to ten-second short videos. LBVD database contains 1013 videos with diversified visual contents and the predominant resolutions are , and . The content of the video is mostly human-centred, the scenes are captured under different lighting conditions, and both indoor and outdoor scenes are included. In this paper, four commonly used criteria are adopted for performance evaluation, including Spearman rank-order correlation coefficient (SRCC), KLCC, Pearson linear correlation coefficient (PLCC) and root mean squared error (RMSE) [34]. PLCC and RMSE are used for measuring prediction accuracy, while SRCC and KRCC are used for measuring monotonicity. To compute PLCC and RMSE, the following five-parameter logistic function is used to map the predicted scores and subjective scores as follows [35]: (22)where , , are the fitting parameters. Generally, a good metric produces high SRCC, KRCC and PLCC values, as well as a low RMSE value. 4.2 Performance evaluation on LBVD database In this section, the performance of the proposed LBVQA metric is thoroughly investigated and compared with the existing quality metrics in terms of the prediction monotonicity and accuracy. (1) First, we compare the performance of the proposed LBVQA with two state-of-the-art NR video quality metrics, including Video BLIINDS index [16] and Video Integrity and Distortion Evaluation Oracle (VIIDEO) metric [17]. (2) General-purpose NR-IQA methods have been presented for evaluating image quality without knowing specific distortion types. In principle, all image quality metrics can be used for VQA in a frame-by-frame manner. As a quality metric specifically designed for real multiply-distorted live broadcasting videos, the proposed LBVQA is expected to outperform these general-purpose quality metrics. Therefore, we compare the proposed LBVQA with eight state-of-the-art general-purpose NR-IQA metrics, including Blind Image Quality Index (BIQI) [36], Blind Image Quality Assessment (BIQA) [37], dubbed spatial-spectral entropy-based quality (SSEQ) [38], DErivative Statistics-based QUality Evaluator (DESIQUE) [39], Distortion Identification-base Image Verity and INtegrity Evaluation (DIIVINE) Index [40], BLIINDS2 [41], Blind/Referenceless Image Spatial QUality Evaluator (BRISQUE) [35] and Natural Image Quality Evaluator (NIQE) [42]. (3) The live broadcasting video is destroyed by multiple distortions, so it is necessary to investigate the performances of quality metrics for multiple distortions. To this end, we further compare the proposed LBVQA with three state-of-the-art image quality metrics for multiply-distorted images, including gradient-weighted histogram of local binary pattern calculated on the gradient map (GWH-GLBP) [43], FIve-Step BLInd Metric (FISBLIM) [44], and SIx-Step BLInd Metric (SISBLIM) [45]. It is worth to emphasise that all aforementioned learning-based metrics are re-trained on LBVD database for a fair comparison. On the other hand, for all aforementioned IQA metrics, the video quality is measured with the frame-by-frame IQA procedure on the LBVD database [46]. Since the LBVD database contains 1013 distorted videos, 810 distorted video is used for model training, and the remaining 203 distorted videos are used for the test. In order to ensure that the reported results are reliable, we repeat the train-test process 1000 times. Then the median values, as well as, the corresponding standard deviations are reported. Table 2 summarises the experimental results on the LBVD database, where the best performance values are marked in boldface. In this table, the type 'GNR' represents the general-purpose NR-IQA metric, the 'MDI' represents the IQA metric for multiple distortions, and the 'NRV' represents the NR-VQA metric. Table 2. Performances of the proposed LBVQA metric and three types of metricson LBVD database Category Metrics Type PLCC SRCC KRCC RMSE IQA BIQI [36] GNR BIQA [37] GNR DIIVINE [40] GNR BLIINDS2 [41] GNR BRISQUE [35] GNR NIQE [42] GNR 0.3443 0.2410 0.3696 0.7806 SSEQ [38] GNR DESIQUE [39] GNR GWH-GLBP [43] MDI FISBLIM [44] MDI 0.1652 0.0118 0.0088 0.8413 SISBLIM [45] MDI 0.3288 0.2808 0.1937 0.8120 VQA VIIDEO [17] NRV 0.2392 0.2206 0.1989 0.8036 V-BLIINDS [16] NRV proposed S-D NRV LBVQA (proposed) NRV It is known from Table 2 that the proposed LBVQA model achieves the best performance on the LBVD database, regardless of prediction accuracy and monotonicity. The feature learning-based Video BLIINDS index performs the second best. Compared with the general-purpose NR-IQA metrics, the performance of the proposed LBVQA metric is significantly better than the state-of-the-art. The MDI metrics do not perform very well either. For the live broadcasting videos in the LBVD database, most of the existing NR-IQA and NR-VQA metrics can only produce quality scores that are moderately correlated with the subjective scores, and most of the correlation values are 0.5. In addition, the proposed LBVQA metric obtains the lowest standard deviation, which demonstrates the performance of the proposed LBVQA metric is very stable in 1000 times experiments. Furthermore, to more fairly compare with the existing IQA metrics, the features extracted in the S-D are trained and tested with the same frame-by-frame IQA procedure on the LBVD database, and the performance is also listed in Table 2. It is worthy to emphasise that the S-D part of the proposed LBVQA metric outperforms all the existing NR-IQA metrics, including general-purpose NR-IQA metrics and multiply-distorted NR-IQA metrics. These results demonstrate that the qu